Information processing system, information processing method, and program

Abstract

An information processing system according to one embodiment of the present technology is provided with a generation unit and a calculation unit. The generation unit generates a virtual model of a detection target in a virtual space on the basis of data of the detection target, which is detected by a sensor system in real space, so as to be in synchronism with the detection target in the real space. The calculation unit calculates a recommended detection point for data detection with respect to the detection target, on the basis of the data of the detection target. This makes it possible to generate a highly accurate virtual model.

Description

Information processing system, information processing method, and program

[0001] The present technology relates to an information processing system, an information processing method, and a program that can be applied to generating virtual models such as digital twins.

[0002] Patent Document 1 discloses a digital twin management system that aims to improve usability for users.

[0003] Japanese Patent Application Laid-Open No. 2022-161342

[0004] There is a demand for technology that can generate highly accurate virtual models such as digital twins.

[0005] In view of the above circumstances, an object of the present technology is to provide an information processing system, an information processing method, and a program that enable generation of a highly accurate virtual model.

[0006] In order to achieve the above object, an information processing system according to one aspect of the present technology includes a generation unit and a calculation unit. The generation unit generates a virtual model of the detection target in a virtual space based on data of the detection target detected by a sensor system in a real space so as to be synchronized with the detection target in the real space. The calculation unit calculates recommended detection points for detecting data for the detection target based on the data of the detection target.

[0007] In this information processing system, a virtual model of the detection target is generated in virtual space based on data of the detection target obtained by the sensor system so as to be synchronized with the detection target in real space. Furthermore, recommended detection points for data detection of the detection target are calculated based on the data of the detection target. This makes it possible to generate a highly accurate virtual model.

[0008] The generation unit may generate the virtual model of the detection target based on data of the detection target at the recommended detection point.

[0009] The detection target may be configured in a predetermined three-dimensional region in the real space. In this case, the data of the detection target may include detection data for each detection point detected by the sensor system in the predetermined three-dimensional region. The generation unit may generate the virtual model by predicting data for each point in the predetermined three-dimensional region through probabilistic modeling based on the detection data for each detection point.

[0010] The generation unit may calculate the likelihood of predicted data, which is data for each of the points predicted by the probabilistic modeling.

[0011] The calculation unit may generate an acquisition function for each of the points based on predicted data, which is data for each of the points predicted by the probabilistic modeling, and calculate the recommended detection points based on the acquisition function.

[0012] The sensor system may include one or more sensors, and in this case, the calculation unit may generate the acquisition function based on at least one of a distance of each of the points from the one or more sensors and a time elapsed since the most recent detection by the sensor system at each of the points.

[0013] The calculation unit may generate the acquisition function based on information of data desired by a user regarding the virtual model.

[0014] The calculation unit may calculate, as the recommended detection point, a point at which the value of the acquisition function is maximum.

[0015] The information processing system may further include a GUI presentation unit that presents a GUI (Graphical User Interface) for managing data related to the virtual model. In this case, the GUI for managing data related to the virtual model may be configured to enable confirmation or adjustment of at least one of the virtual model, the detection data, the prediction data, the likelihood, and the acquisition function.

[0016] The information processing system may further include a GUI presentation unit that presents a GUI for collecting data of the detection target, and a plan information generation unit that generates plan information for executing detection of the data of the detection target by the sensor system. In this case, the GUI for collecting data of the detection target may be configured to allow confirmation or correction of the plan information.

[0017] The plan information generating unit may generate the plan information based on the recommended detection points.

[0018] The information processing system may further include a GUI presentation unit that presents a GUI for utilizing data related to the virtual model. In this case, the GUI for utilizing data related to the virtual model may be configured to enable at least one of the detection data, the prediction data, and the likelihood to be acquired by specifying the data.

[0019] The GUI for utilizing data related to the virtual model may be configured to allow input of desired data information related to the virtual model.

[0020] The virtual model may be a digital twin of the detection object.

[0021] According to one aspect of the present technology, there is provided an information processing method executed by a computer system, the information processing method including: generating a virtual model of a detection target in a virtual space based on data of the detection target detected by a sensor system in a real space, the virtual model being synchronized with the detection target in the real space; and calculating a recommended detection point for detecting data for the detection target based on the data of the detection target.

[0022] A program according to an embodiment of the present technology causes a computer system to execute the information processing method.

[0023] 1 is a schematic diagram showing an example of the configuration of a digital twin management system according to the present technology. FIG. 1 is a diagram showing an example of the overall operation flow of a digital twin management system. FIG. 2 is a schematic graph for explaining the generation of a digital twin by probabilistic modeling. FIG. 3 is a block diagram showing an example of the functional configuration of a data collector terminal, a server device, a data manager terminal, and a data user terminal. FIG. 1 is a schematic diagram showing an example of the operation flow of a data collector terminal. FIG. 2 is a schematic diagram showing an example of the operation flow of a server device and a data manager terminal. FIG. 3 is a schematic diagram showing an example of the operation flow of a data user terminal. FIG. 4 is a schematic diagram showing an example of a data management GUI displayed on a data manager terminal. FIG. 5 is a diagram showing an example of the display content of the operation and confirmation information display section of the data management GUI ("Graph" tab / "Adjustment" tab). FIG. 6 is a schematic diagram showing an example of a data collection GUI displayed on a data collector terminal. FIG. 7 is a schematic diagram showing an example of a data collection GUI (two data collection robots). FIG. 8 is a schematic diagram showing an example of a data collection GUI ("Plan Correction" tab). FIG. 9 is a diagram showing an example of the display content of the operation and confirmation information display section of the data collection GUI ("Upload" tab). FIG. 10 is a schematic diagram showing an example of a data utilization GUI displayed on a data user terminal. 1 is a diagram showing an example of the display content of the operation / confirmation information display section of the data utilization GUI ("Data Registration"); and FIG. 2 is a block diagram showing an example of the hardware configuration of a computer 60 that can be used as a server device, a data administrator terminal, a data collector terminal, a data user terminal, etc.

[0024] Hereinafter, embodiments of the present technology will be described with reference to the drawings.

[0025] [Digital Twin Management System] Fig. 1 is a schematic diagram showing an example configuration of a digital twin management system according to the present technology. The digital twin management system 1 corresponds to an embodiment of an information processing system according to the present technology.

[0026] The digital twin management system 1 is a system that can provide a wide range of services to various users, from the generation of digital twins to the utilization of digital twins.

[0027] A digital twin is a twin-like reproduction of a physical object in the real world or a real system (real environment) constructed in the real world, based on sensing data detected by a sensor system. Virtual space can also be considered digital space.

[0028] The virtual model created as a digital twin can be used to monitor the real world, perform various simulations, predict and forecast events in the near future, etc. Furthermore, the use of digital twins is expected to bring about various benefits, such as preventive maintenance, quality assurance, cost and risk reduction, shortened development and production times, and proper production and inventory management.

[0029] Furthermore, the virtual model generated as a digital twin can be constructed in synchronization with physical objects and real systems in the real world based on data detected in real time by a sensor system. This makes it possible to perform highly real-time simulations and predictions that are in line with the situation in the real world. It is also possible to feed back simulation and prediction results to the real world at the optimal timing.

[0030] As shown in FIG. 1 , the digital twin management system 1 includes a sensor system 2 , a server device 3 , a data manager terminal 4 , a data collector terminal 5 , and a data user terminal 6 .

[0031] The sensor system 2, server device 3, data manager terminal 4, data collector terminal 5, and data user terminal 6 are communicably connected to one another via a network 7. The network 7 is constructed using, for example, the Internet or a wide area communication network. Alternatively, any WAN (Wide Area Network) or LAN (Local Area Network) may be used, and the protocol for constructing the network 7 is not limited.

[0032] The sensor system 2 is configured to be able to sense data of a detection target 8, which is a target for generating a digital twin, in real space. In the example shown in Fig. 1, the sensor system 2 is configured with a picking system in a logistics warehouse as the detection target 8.

[0033] The sensor system 2 has one or more sensors (not shown). For example, an image sensor (photographing device) or a depth sensor (distance measuring device) placed in the logistics warehouse, or an image sensor or a depth sensor mounted on a data collection robot that can move within the logistics warehouse, is used as the one or more sensors that make up the sensor system 2. For example, these sensors can detect photographed images and distance information (depth information) at various points (positions) within the logistics warehouse.

[0034] In this disclosure, the detection data (sensing data) detected by the sensor system 2 includes not only the output of the various sensors included in the sensor system 2 (captured images, distance information, etc.), but also various information and data generated from the sensor output.

[0035] For example, when map information of the inside of a logistics warehouse is created based on photographed images, distance information, etc., the map information is also included in the detection data detected by the sensor system 2. Furthermore, based on photographed images, distance information, etc., it is also possible to generate various information and data regarding shelves, products, workers, forklifts, picking support robots, etc. in the logistics warehouse, such as whether or not there is movement, movement route, movement speed, density, picking operation, etc. This information and data is also included in the detection data detected by the sensor system 2.

[0036] The function of generating detection data such as map information and travel routes based on the sensor output (captured images, distance information, etc.) may be installed in the sensor system 2, or may be installed in a device other than the sensor system 2, such as a server device 3.

[0037] The specific configuration of the sensor system 2 is not limited, and the sensor system 2 can be configured using various sensors. For example, the sensor system 2 may be configured using any image sensor or any depth sensor. Note that in the present disclosure, images include both still images and moving images (video).

[0038] Examples of image sensors capable of acquiring two-dimensional images include an RGB camera (visible light camera) and an infrared camera. Examples of depth sensors capable of acquiring three-dimensional information include a stereo camera, a ToF (Time of Flight) sensor, and a structured light depth sensor. Furthermore, an ultrasonic sensor or the like may be used as a sensor for acquiring the state of an object. Other examples of sensor devices that may be used include any distance measuring device such as a laser distance measuring sensor, 2D / 3D LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), a contact sensor, a sonar, an illuminometer, a biosensor, and the like.

[0039] Of course, various sensors may be mounted on workers or picking support robots in the logistics warehouse. For example, a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), or an inertial measurement unit (IMU) that integrates these may be mounted on a worker or the like and function as a sensor included in the sensor system 2. Other environmental sensors such as a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, and an illuminance sensor may also be used as sensors included in the sensor system 2.

[0040] The sensor system 2 may be configured in any manner so as to be able to detect data of a detection target 8, i.e., a physical object or real system that is the target of generating a digital twin.

[0041] The server device 3, the data manager terminal 4, the data collector terminal 5, and the data user terminal 6 each have hardware necessary for a computer, such as a processor such as a CPU, GPU, or DSP, memory such as a ROM or RAM, and a storage device such as an HDD (see FIG. 16 ). The processor loads the program related to the present technology stored in the storage unit or memory into RAM and executes it, thereby executing the information processing method related to the present technology (digital twin management method / digital twin generation method / data acquisition method / data management method / data utilization method).

[0042] Any computer such as various PCs or tablet terminals may be used as the server device 3, the data manager terminal 4, the data collector terminal 5, and the data user terminal 6. Note that any computer equipped with a display may be used as the data manager terminal 4, the data collector terminal 5, and the data user terminal 6.

[0043] Based on data of the detection target 8 detected by the sensor system 2 in real space, the server device 3 generates a digital twin (virtual model) 9 of the detection target 8 in virtual space so as to be synchronized with the detection target 8 in real space.

[0044] In the example shown in Figure 1, a digital twin 9 of a picking system in a logistics warehouse is generated in synchronization with the picking system in real space. Note that the example shown in Figure 1 illustrates visualized data related to the digital twin 9. The generation of the digital twin 9 and the visualization of the data will be explained in detail later.

[0045] In addition, in this digital twin management system 1, various GUIs (Graphical User Interfaces) are generated by the server device 3 and output to the data manager terminal 4, data collector terminal 5, and data user terminal 6.

[0046] The data manager terminal 4 is a terminal used by the data manager 10. The data manager 10 performs various data management tasks, including management of the generation of the digital twin 9, via, for example, a GUI for managing data related to the digital twin 9 (hereinafter referred to as a data management GUI).

[0047] In this embodiment, the data manager terminal 4 executes various processes related to data management mainly in cooperation with the server device 3. It is also possible to incorporate the functions of the server device 3 into the data manager terminal 4, and to configure the data manager terminal 4 and the server device 3 as an integrated unit.

[0048] The data collector terminal 5 is a terminal used by a data collector 11. The data collector 11 controls sensing of the detection target 8 by the sensor system 2 via, for example, a GUI for collecting data of the detection target 8 (hereinafter referred to as a data collection GUI), and collects detection data for generating a digital twin 9.

[0049] The data user terminal 6 is a terminal used by the data user 12. The data user 12 can download the data they want to utilize and input information about the desired data, for example, via a GUI for utilizing data related to the digital twin 9 (hereinafter referred to as a data utilization GUI).

[0050] In applying this technology, the relationships between the data manager 10, the data collector 11, and the data user 12 are not limited. For example, the data manager 10, the data collector 11, and the data user 12 may belong to the same organization such as a company. Alternatively, the data collector 11 and the data user may be business partners such as customers of the data manager 10.

[0051] Furthermore, in the present disclosure, each of the data manager 10, the data collector 11, and the data user 12 is considered to be included in the users who use the digital twin management system 1. In other words, each of the data manager 10, the data collector 11, and the data user 12 can be an embodiment of a user related to the present technology.

[0052] The DB 13 stores any information related to the generation of the digital twin 9 and the utilization of the digital twin 9. For example, various information is stored, such as information related to the detection target 8 in real space, information related to the sensor system 2, information related to the generation of the digital twin 9, various data related to the digital twin 9, analytical information and prediction information for the digital twin 9, information related to the data manager 10, information related to the data collector 11, and information related to the data user 12.

[0053] 1, DB 13 is configured by a storage device or the like separate from server device 3 and is connected to server device 3. However, the configuration is not limited to this, and DB 13 may be configured by a storage unit (see FIG. 16 ) of server device 3. Alternatively, DB 13 may be constructed on network 7, and a configuration may be adopted in which DB 13 is accessible via network 7.

[0054] [Overall Operational Flow of Digital Twin Management System] The overall operational flow of the digital twin management system 1 will be described with reference to Fig. 2. Fig. 2A is a diagram showing an outline of the overall operational flow. Fig. 2B is a diagram showing the overall operational flow in the case of managing the digital twin 9 of the picking system in the logistics warehouse illustrated in Fig. 1 as an example.

[0055] As shown in Figure 2A, in this digital twin management system 1, the generation of a digital twin 9 is performed (hereinafter referred to as digital twin generation phase P1).

[0056] 2B , a sensor system 2 including a data collection robot 15 and the like performs sensing on a picking system, which is a detection target 8 in the real space, and detects data (step 101). For example, data on a warehouse, shelves, products, workers, forklifts, picking support robots, etc. is detected.

[0057] The server device 3 generates a digital twin 9 of the picking system in the logistics warehouse. In this embodiment, the digital twin 9 is generated by probabilistically modeling the detection data detected by the sensor system 2 (step 102).

[0058] FIG. 3 is a schematic graph for explaining the generation of a digital twin 9 through probabilistic modeling.

[0059] In real space, the detection target 8, which is the target for generating the digital twin 9, is configured in a predetermined three-dimensional area. For example, in the example of a picking system in a logistics warehouse shown in Figure 1, the internal space of the logistics warehouse is one embodiment of the predetermined three-dimensional area in real space in which the detection target 8 is configured.

[0060] The horizontal axis of each graph shown in Fig. 3 schematically represents each point (position) in a three-dimensional area (the internal space of the logistics warehouse). Each point in the three-dimensional area is represented by, for example, (x, y, z) coordinate values.

[0061] As a three-dimensional coordinate system for defining each point in a three-dimensional region, for example, an absolute coordinate system (world coordinate system) may be used. Alternatively, a relative coordinate system with a predetermined point as a reference (origin) may be used. When a relative coordinate system is used, the reference origin may be set arbitrarily.

[0062] 3, in order to facilitate understanding of the generation of the digital twin 9 through probabilistic modeling, three-dimensional position information (x, y, z) is illustrated schematically on only one axis, the horizontal axis. The generation of the digital twin 9 through probabilistic modeling described below is also applicable when the position of each point is expanded into a three-dimensional coordinate system (x, y, z).

[0063] The vertical axis of each graph shown in Figure 3 represents data at each point in the three-dimensional region. For example, it is possible to detect data such as the presence or absence of an object at each point and the type of object present based on distance information and images captured at a predetermined frame rate. Of course, various other data may also be detected.

[0064] 3A, the dotted line graph corresponds to actual data in real space. The black dots on the dotted line graph represent detection data actually detected by the sensor system 2.

[0065] Hereinafter, a point where data is actually detected by the sensor system 2 will be referred to as a detection point. In the graph shown in Fig. 3A, detection data for each of three detection points in a three-dimensional area is plotted. No data is detected for other points in the three-dimensional area.

[0066] 3B , the server device 3 generates a digital twin (virtual model) 9 by predicting data at each point in a three-dimensional area through probabilistic modeling based on the detection data for each detection point. That is, the digital twin 9 is generated from the actually detected detection data and prediction data, which is data predicted through probabilistic modeling.

[0067] In this example, probabilistic modeling is performed using Bayesian optimization with Gaussian process regression. The solid line graph in Figure 3B is a graph of the mean value (mean value of the probability distribution) calculated by the probabilistic modeling, and this mean value becomes the predicted data for each point in the three-dimensional region.

[0068] 3B, the standard deviation (variance) of the probability distribution is calculated for the predicted data of each point in the three-dimensional region by the probabilistic modeling. The standard deviation value can be used as a parameter representing the likelihood of the predicted data of each point predicted by the probabilistic modeling.

[0069] 3B , the standard deviation is zero and the likelihood is high (maximum) for detection data at a detection point actually detected by the sensor system 2. On the other hand, predicted data with a large standard deviation is data with low likelihood and is likely to deviate significantly from the actual data in the real space.

[0070] In this way, the digital twin management system 1 can generate a digital twin 9 along with probability parameters through probabilistic modeling. Generating the data that makes up the digital twin 9 according to a probability distribution is a novel technical matter not found in the past.

[0071] As shown in FIG. 2B, in this embodiment, in the digital twin generation phase P1, the server device 3 generates an acquisition function (step 103).

[0072] As illustrated in Figure 3C, an acquisition function is generated for each point in the three-dimensional domain, i.e., for each point position, the value of the acquisition function can be calculated.

[0073] As the acquisition function, for example, EL (Expected Improvement), PI (Probability of Improvement), LCB (Lower Confidence Bound), etc. can be used. Furthermore, in this digital twin management system 1, it is also possible to generate an acquisition function specific to this technology based on parameters related to the detection of data of the detection target 8 by the sensor system 2, requests from the data user 12, etc. This point will be explained later.

[0074] As shown in Figure 2A, in this digital twin management system 1, an update of the digital twin is performed (hereinafter referred to as digital twin update phase P2).

[0075] 2B, recommended detection points for detecting data for the detection target 8 are calculated based on the acquisition function generated in step 102. Then, based on the calculated recommended detection points, the sensor system 2 senses the real space (step 104).

[0076] As shown in Fig. 3C, the point at which the value of the acquisition function is maximum can be calculated as the recommended detection point. Note that as shown in Fig. 3C, the recommended detection point can also be considered as a point requiring exploration.

[0077] As shown in Figure 3D, the sensor system 2 detects data at the recommended detection points. The detected data becomes additional detection data at the additional detection points. Using the additional detection data, a digital twin 9 is generated again through probabilistic modeling. This updates the digital twin 9 (step 105).

[0078] As shown in Figure 3D, data is detected by the sensor system 2 at recommended detection points where the probability is low and the deviation from the actual data in the real space is large. This makes it possible to bring the predicted data (average) predicted by probabilistic modeling closer to the actual data in the real space. As a result, it becomes possible to generate a highly accurate digital twin 9.

[0079] When the digital twin 9 is updated, the acquisition function is generated again. That is, in this digital twin management system 1, the acquisition function is updated in response to the update of the digital twin 9 (step 106).

[0080] As shown in Figure 3D, recommended detection points are further calculated based on the updated acquisition function. Data is detected at the calculated recommended detection points, and the digital twin is updated through probabilistic modeling. By repeating this cycle of updating the digital twin 9 and the acquisition function, it is possible to generate a highly accurate digital twin 9.

[0081] As shown in Figure 2A, in this digital twin management system 1, data utilization is carried out (hereinafter referred to as data utilization phase P3).

[0082] In the example shown in Figure 2B, the data user terminal 6 performs learning and analysis on the data (mean / standard deviation) generated as the digital twin 9, thereby utilizing the data (step 107). For example, it is possible to perform a simulation to confirm the safety or work efficiency of a picking support robot using data on the number of workers (mean / standard deviation). Of course, it is also possible to utilize various types of data, such as for other simulations, monitoring, and predictions.

[0083] Furthermore, in this digital twin management system 1, the data user 12 can register information about missing data (step 108). For example, it is possible to register that the number of data points and the likelihood of data about workers in a specific area in a three-dimensional space are insufficient. It is also possible to register that information about the movement path of a pickup support robot and its likelihood are insufficient.

[0084] Information about the missing data registered by the data user 12 is transmitted to the server device 3. Then, the acquisition function is updated based on the information about the missing data. For example, the acquisition function is updated so that a recommended detection point for detecting the missing data registered by the data user 12 is calculated.

[0085] For example, if the data user 12 registers detection data of a predetermined point as missing data, the acquisition function is updated so that the predetermined point is calculated as a recommended detection point. Alternatively, if probabilistic modeling is performed using detection data from the recommended detection point, the acquisition function is updated so that the likelihood of the data being registered as missing data is increased to a desired level.

[0086] In this way, the digital twin management system 1 makes it possible to update the acquisition function in response to the requests of the data users 12. This makes it possible to efficiently generate digital twins 9 that are useful to the data users 12.

[0087] 3, Bayesian optimization using Gaussian process regression is used as an example of probabilistic modeling. However, the present invention is not limited to this, and any algorithm may be adopted to realize probabilistic modeling.

[0088] [Example of functional configuration of each device] Fig. 4 is a block diagram showing an example of functional configuration of the data collector terminal 5, the server device 3, the data manager terminal 4, and the data user terminal 6. In each device, a processor such as a CPU executes a program related to the present technology to configure the functional blocks shown in Fig. 4. Of course, dedicated hardware such as an IC (integrated circuit) may be used to realize each functional block.

[0089] The program is installed in each device via, for example, various recording media. Alternatively, the program may be installed via the Internet or the like. The type of recording medium on which the program is recorded is not limited, and any computer-readable recording medium may be used. For example, any computer-readable non-transitory storage medium may be used.

[0090] The data collector terminal 5 includes a display control unit 17 , an input receiving unit 18 , a sensing planning unit 19 , a sensing control unit 20 , and a detection data acquisition unit 21 .

[0091] The display control unit 17 controls the image display on the display of the data collector terminal 5. In this embodiment, the data collection GUI generated by the server device 3 is displayed on the display. Of course, any other image can be displayed.

[0092] The input receiving unit 18 receives various inputs from the data collector 11 using the data collector terminal 5. In this embodiment, various inputs such as instructions and selections from the data collector 11 via a data collection GUI displayed on the display are received. Of course, inputs are not limited to those via a GUI.

[0093] The sensing plan unit 19 generates plan information for executing detection of data of the detection target 8 in real space by the sensor system 2. For example, the plan information generated includes the position of the detection point where sensing is performed, the detection timing (detection time), the detection frequency (time interval), a path plan for the data collection robot 15, and a layout plan for multiple data collection robots 15.

[0094] In this embodiment, the sensing planning unit 19 calculates recommended detection points based on an acquisition function such as that shown in Fig. 3. Then, sensing planning information is generated based on the calculated recommended detection points. For example, the planning information is generated so that data of the detection target 8 is detected at the recommended detection points.

[0095] The sensing control unit 20 controls the operation of the sensor system 2 based on the sensing plan information. For example, the sensing control unit 20 controls the operation of the sensor system 2 so that data on the detection target 8 can be detected at the detection points generated as the plan information. It is also possible to control the driving of the data collection robot 15 according to a movement plan for the data collection robot.

[0096] The detection data acquisition unit 21 acquires detection data (sensing data) detected by the sensor system 2. In the present disclosure, acquisition of data (information) includes not only receiving the data (information) from another device, but also generating the data (information) by executing a predetermined algorithm on its own.

[0097] 4 includes not only receiving detection data from the sensor system 2 but also generating various data related to the detection target 8 based on the data from the sensor system 2. In this case, both the data from the sensor system 2 and the data generated by the detection data acquisition unit 21 are included in the detection data of the detection target 8.

[0098] The server device 3 includes a data processing unit 22 , a data management unit 23 , a data visualization unit 24 , and a data output unit 25 .

[0099] The data processing unit 22 executes various data processing related to the digital twin management system 1. In this embodiment, the data processing unit 22 executes generation of a digital twin 9 through probabilistic modeling, as exemplified in FIG. 3 . The data processing unit 22 also executes generation of an acquisition function. Of course, other data processing is also executed.

[0100] The data management unit 23 performs various data management related to the digital twin management system 1. In this embodiment, the data management unit 23 performs data management mainly in response to various inputs by the data manager 10 via the data manager terminal 4. Of course, this is not limited to this.

[0101] The data visualization unit 24 generates visualization information. The visualization information includes any image information that can be viewed by the data collector 11, the data manager 10, and the data user 12, and includes various image information such as text, icons, GUI, etc.

[0102] In this embodiment, the data visualization unit 24 generates a data collection GUI, a data management GUI, and a data utilization GUI.

[0103] The data output unit 25 outputs data related to the digital twin 9. In this embodiment, the data output unit 25 outputs detection data from detection points, prediction data (average) generated by probabilistic modeling, and the likelihood (standard deviation) of the prediction data, as shown in Fig. 3. Of course, other data can also be output.

[0104] The data manager terminal 4 includes a display control unit 26 and an input receiving unit 27 .

[0105] The display control unit 26 controls the image display on the display of the data manager terminal 4. In this embodiment, a data management GUI generated by the server device 3 is displayed on the display. Of course, any other image can be displayed.

[0106] The input reception unit 27 receives various inputs from the data manager 10 using the data manager terminal 4. In this embodiment, various inputs such as instructions and selections from the data manager 10 via a data management GUI displayed on the display are received. Of course, inputs are not limited to those via a GUI.

[0107] The data user terminal 6 includes a display control unit 28 , an input receiving unit 29 , and a data utilization unit 30 .

[0108] The display control unit 28 controls the image display on the display of the data user terminal 6. In this embodiment, the data utilization GUI generated by the server device 3 is displayed on the display. Of course, any other image can be displayed.

[0109] The input receiving unit 29 receives various inputs from the data user 12 using the data user terminal 6. In this embodiment, various inputs such as instructions and selections from the data user 12 via the data utilization GUI displayed on the display are received. Of course, the inputs are not limited to those via the GUI.

[0110] The data utilization unit 30 performs learning and analysis on the data generated as the digital twin 9. In this embodiment, learning and analysis can be performed on the predicted data (average) generated by probabilistic modeling and the likelihood (standard deviation) of the predicted data. There are no specific limitations on the algorithms used for learning and analysis, and a wide range of processes can be performed, such as monitoring, simulation, and prediction.

[0111] 4 , the data processing unit 22 of the server device 3 realizes an embodiment of a “generation unit” according to the present technology that generates a virtual model of a detection target in a virtual space based on data of the detection target detected by a sensor system in real space so as to be synchronized with the detection target in real space. Furthermore, the digital twin 9 corresponds to an embodiment of a “virtual model” according to the present technology.

[0112] In addition, the data processing unit 22 of the server device 3 and the sensing planning unit 19 of the data collector terminal 5 realize one embodiment of a ``calculation unit'' related to the present technology that calculates recommended detection points for detecting data for a detection target based on data of the detection target.

[0113] Furthermore, the DB 13 connected to the server device 3 implements an embodiment of a "storage unit" that stores the likelihood of predicted data, which is data for each point predicted by probabilistic modeling.

[0114] Furthermore, the data visualization unit 24 of the server device 3 and the display control units 17, 26, and 28 of the devices implement an embodiment of a "GUI presentation unit" according to the present technology.

[0115] Furthermore, the sensing planning unit 19 of the data collector terminal 5 realizes one embodiment of a “planning information generating unit” that generates planning information for executing detection of data of detection targets by the sensor system 2 .

[0116] Of course, the configuration for realizing the "generation unit," "calculation unit," "storage unit," "GUI presentation unit," and "plan information generation unit" according to the present technology is not limited to the example shown in Fig. 4. These components may be realized by a single device included in the digital twin management system 1, or may be realized by multiple devices working together.

[0117] [Example of Operational Flow of Each Device] Fig. 5 is a schematic diagram showing an example of the operational flow of the data collector terminal 5. In the digital twin generation phase P1, sensing of the real space is executed (step 201).

[0118] Specifically, sensing plan information is generated by the sensing plan unit 19 of the data collector terminal 5. As the initial plan information, for example, default plan information may be generated, or the initial plan information may be created by the data collector 11 via the data collection GUI.

[0119] The sensing control unit 20 controls the operation of the sensor system 2 based on the sensing plan information. Then, the sensor system 2 detects detection data at predetermined detection points. The detected detection data is acquired by the detection data acquisition unit 21 of the data collector terminal 5.

[0120] The acquired detection data is shared with the digital twin management system 1 (step 202). In this embodiment, the detection data is transmitted from the data collector terminal 5 to the server device 3.

[0121] The data collector terminal 5 acquires the acquisition function transmitted from the digital twin management system 1 (step 203). In this embodiment, the acquisition function is transmitted from the server device 3 to the data collector terminal 5. The acquired acquisition function is output to the sensing planning unit 19.

[0122] In the digital twin update phase P2, sensing of the real space is performed based on the acquisition function obtained in the digital twin generation phase P1 (step 204).

[0123] Based on the acquisition function, the sensing planning unit 19 of the data collector terminal 5 calculates recommended detection points for detecting data for the detection target. Then, based on the calculated recommended detection points, sensing plan information is generated.

[0124] The sensing control unit 20 controls the operation of the sensor system 2 based on the sensing plan information. For example, the sensor system 2 detects detection data at, for example, recommended detection points.

[0125] The acquired detection data is again shared with the digital twin management system 1 (step 202). Then, the acquisition function sent from the digital twin management system 1 is again acquired (step 206). The acquired acquisition function is output to the sensing planning unit 19.

[0126] In this way, the digital twin management system 1 can perform sensing based on recommended detection points calculated based on the acquisition function. Furthermore, by repeating the cycle of transmitting detection data at the recommended detection points and obtaining the acquisition function, a highly accurate digital twin 9 can be generated.

[0127] It is also possible for the data collector 11 to slightly adjust the recommended detection points via the data collection GUI and execute data detection. It is also possible to input that data detection will not be performed at the recommended detection points.

[0128] In this embodiment, the data collector terminal 5 does not operate in the data utilization phase P3.

[0129] 6 is a schematic diagram showing an example of the operation flow of the server device 3 and the data manager terminal 4. In the digital twin generation phase P1, detection data of the real space transmitted from the data collector terminal 5 is acquired (step 301).

[0130] The data processing unit 22 of the server device 3 probabilistically models the detection data to calculate predicted data (average) and probability (standard deviation), and generates a digital twin 9 (step 302).

[0131] The data processing unit 22 of the server device 3 generates an acquisition function (step 303). As described above, the acquisition function can be, for example, EL, PI, LCB, etc. That is, if a point in a predetermined three-dimensional region where the detection target 8 is configured is Px (=xyz), it is possible to calculate a recommended detection point based on the values ​​of EL(Px), PI(Px), and LCP(Px).

[0132] In addition, in this digital twin management system 1, it is also possible to generate an acquisition function specific to this technology based on parameters related to the detection of data of the detection object 8 by the sensor system 2, etc.

[0133] For example, it is possible to generate an acquisition function based on the characteristics of the sensor used in the sensor system 2. For example, the greater the distance from the sensor that detects data, the lower the accuracy of the data detected by the sensor. Focusing on this point, it is also possible to generate an acquisition function so that the value of the acquisition function becomes larger (= the position to be sensed) for points that are farther away from the sensor.

[0134] For example, if a point in a predetermined three-dimensional region where the detection target 8 is configured is Px (=xyz), it is possible to generate an acquisition function using parameters defined by the following equation: f(Px)=a (distance from the sensor to Px), where a is a predetermined coefficient.

[0135] Furthermore, when multiple sensors (n sensors) are used in the sensor system 2, it is also possible to generate an acquisition function using parameters defined by the following multiple equations: f1(Px) = a1 (distance from sensor 1 to Px) f2(Px) = a2 (distance from sensor 2 to Px) fn(Px) = an (distance from sensor n to Px) a1 to an are coefficients

[0136] In this way, it is possible to generate an acquisition function specific to the present technology based on the separation distance of each point relative to one or more sensors included in the sensor system 2.

[0137] It is also possible to configure the acquisition function so that the longer the time that has elapsed since the most recent detection by the sensor system 2, the larger the value of the acquisition function (=the position to be sensed).

[0138] For example, if the current time is t and the most recent detection time (sensing time) of each point Px is ts, it is possible to generate an acquisition function using parameters defined by the following equation: gPx(t)=b(t-ts), where b is a coefficient

[0139] In this way, it is possible to generate an acquisition function specific to the present technology based on the time elapsed since the most recent detection by the sensor system 2.

[0140] For example, using the above (equation), it is possible to define the acquisition function at(Px) for each point Px at the current time t using the following (equation 1) or (equation 2).

[0141] (Formula 1) at(Px)=f(Px)+gPx(t) (Formula 2) at(Px)=f1(Px)+...+fn(Px)+gPx(t)

[0142] Of course, it is also possible to generate an acquisition function according to the present technology by appropriately combining the above-mentioned EL(Px), PI(Px), and LCP(Px). In addition, the formulation of the acquisition function and the parameters to be used can be set arbitrarily. Of course, the data manager 10 can also input instructions regarding the formulation of the acquisition function and the parameters via the data management GUI.

[0143] The calculated acquisition function is transmitted to the data collector terminal 5 .

[0144] In the digital twin update phase P2, the data collector terminal 5 acquires sensed detection data based on the acquisition function (step 304). By probabilistically modeling the acquired detection data, predicted data (average) and likelihood (standard deviation) are calculated, and the digital twin is updated (step 305).

[0145] The data processing unit 22 updates the acquisition function and transmits it to the data collector terminal 5 (step 306).

[0146] In the data utilization phase P3, information on the data to be used by the data user 12 is acquired (step 307). The information on the data to be used by the data user 12 is input by the data user 12 via a data utilization GUI, for example.

[0147] The data used by the data user 12 is extracted by the data management unit 23 and data visualization unit 24 of the server device 3, and is visualized on the data manager terminal 4 if necessary (step 308).

[0148] The data to be used by the data user 12 is output (step 309). In this embodiment, the predicted data (average) and the likelihood (standard deviation) are output.

[0149] The information on the missing data registered by the data user 12 via the data utilization GUI is acquired (step 310). The information on the missing data is output to the data processing unit 22.

[0150] The data processing unit 22 updates the acquisition function based on the information on the missing data (step 311). For example, the acquisition function is updated so that a recommended detection point for detecting the missing data registered by the data user 12 is calculated. Alternatively, the acquisition function is updated so that the likelihood of the data being registered as missing data is increased to a desired level.

[0151] In this way, in this digital twin management system 1, it is also possible to generate an acquisition function specific to this technology based on a request from the data user 12. The updated acquisition function is sent to the data collector terminal 5.

[0152] 7 is a schematic diagram showing an example of the operation flow of the data user terminal 6. In this embodiment, the data user terminal 6 does not operate in the digital twin generation phase P1 and the digital twin update phase P2.

[0153] In the data utilization phase P3, the data user 12 selects the data to be used via the data utilization GUI (step 401). The data management unit 23 and the data visualization unit 24 of the server device 3 extract the data to be used by the data user 12 and visualize it on the data user terminal 6.

[0154] Specifically, the data to be used is visualized in the data utilization GUI and is confirmed by the data user 12 (step 402).

[0155] The data to be used is transmitted from the server device 3 to the data user terminal 6. The data utilization unit 30 of the data user terminal 6 performs learning and analysis using the data (step 403). In this embodiment, learning and analysis can be performed using predicted data (average) generated by probabilistic modeling and the likelihood of the predicted data (standard deviation).

[0156] The specific content of data utilization is not limited, and it is possible to monitor real-world space, perform various simulations, and predict and forecast events in the near future.

[0157] The data user 12 registers the missing data via the data utilization GUI (step 404). Information on the registered missing data is transmitted to the server device 3. The information on the missing data corresponds to one embodiment of information on data desired by the user (data user 12) regarding the virtual model (digital twin 9) according to the present technology.

[0158] [Data Format Example] An example of a data format used in the digital twin management system 1 will be described. Data consisting of the following parameters can be used as data constituting the digital twin 9. Px = (xyz): Position information of each point μ: Prediction data (mean of probability distribution) (detection data is used at detection points) σ: Probability (standard deviation of probability distribution) (standard deviation is zero at detection points) at(Px): Acquisition function at current time t

[0159] Among these parameters, (μ, σ) is information indicating the probability distribution at each point Px = (x, y, z), and at(Px) is an index indicating whether or not each point Px = (x, y, z) is a position where sensing should be performed.

[0160] In this way, by separating the information (μ, σ) required for data utilization from the index at (Px) required for data collection, it is possible to improve the ease of use for users (data users 12 and data collectors 11) and also improve the accuracy of both tasks. Of course, this is also advantageous for improving the ease of use and accuracy of data management by the data manager 10.

[0161] Of course, the data format used is not limited, and other data formats may be adopted.

[0162] [Configuration Example of GUI Displayed on Each Device] FIG. 8 is a schematic diagram showing an example of a data management GUI displayed on the data manager terminal 4. As shown in FIG.

[0163] The data management GUI 32 has a digital twin display unit 33 , an operation / confirmation information display unit 34 , and a data selection operation unit 35 .

[0164] The digital twin display unit 33 displays the digital twin 9 generated by the data processing unit 22. The digital twin display unit 22 also displays various data related to the digital twin 9. For example, (μ, σ) and at(Px)) for each Px = (xyz) exemplified above are appropriately visualized and displayed.

[0165] In the example shown in FIG. 8 , circle marks 36 with different interiors are displayed as an example of visualization of (μ, σ) for each Px = (xyz). For example, the size of the data (size of the average μ) is displayed so that it can be confirmed depending on the color inside the circle marks 36. For example, the size of the data is expressed by a color change (gradation) from purple to blue to red. Furthermore, the certainty of the data (standard deviation σ) is expressed by the transparency of the circle marks 36.

[0166] Of course, there are no restrictions on what information is displayed on the digital twin display unit 33, and the format in which each piece of data is displayed, and these may be set arbitrarily. For example, major data may be visualized automatically, and the data manager 10 may manually select whether or not to visualize other data. Such settings are also possible.

[0167] In this embodiment, the data selection operation unit 35 of the data management GUI 32 allows the data manager 10 to select the "type of data to display," "data display method," and "range of data to display."

[0168] 8, "number of people" and "people movement speed" are shown as "types of data to display." Of course, this is not limited to this, and various data can be displayed, such as whether or not there is movement, movement trajectory, movement speed, density, picking operation, etc., regarding shelves, products, workers, picking support robots, etc.

[0169] As for "data display method", it is possible to input specifications such as how to display "data magnitude (mean μ magnitude)" and "probability (standard deviation)".

[0170] 8 illustrates a display method in which "data magnitude" is represented by "color" and "probability" is represented by "transparency." Of course, the present invention is not limited to these display methods, and any display method may be adopted.

[0171] As a method for visualizing (μ, σ) and at(Px) for each Px = (xyz), any display method capable of expressing numerical values ​​or distributions in two or three dimensions can be employed. For example, any display method such as a three-dimensional scatter plot, line plot, contour plot, vector diagram, surface plot, heat map, ribbon graph, wall graph, water graph, wire frame, color map, bar graph, density dot graph, etc. may be employed.

[0172] For example, in a three-dimensional space, it is also effective to display the probability distribution using the color, density, and transparency of dots, or to display multiple data using different colors.In a three-dimensional area, it is also possible to represent the acquisition function value of each point using a three-dimensional contour map, heat map, surface diagram, etc.

[0173] It is also possible to set the data manager 10 so that when he clicks on data using a cursor or the like, detailed information about the data is displayed.

[0174] As the "range of data to display" of the data selection operation unit 35, it is possible to specify the range of data to be displayed on the digital twin display unit 33 regarding the "average value" indicating predicted data, the "standard deviation" indicating the probability, and the "degree of data deficiency."

[0175] The "degree of data insufficiency" is a parameter corresponding to the value of the acquisition function at(Px). The higher the value of the acquisition function, the higher the "degree of data insufficiency," and the lower the value of the acquisition function, the lower the "degree of data insufficiency."

[0176] For each of the "average value," "standard deviation," and "level of data deficiency," the range of data displayed is the range between two inverted triangle marks 37 arranged side by side as shown in Fig. 8. The data manager 10 can set the display range for each data item by manually sliding the inverted triangle marks 37 left or right from the default range setting, for example.

[0177] The operation / confirmation information display section 34 of the data management GUI 32 displays switchable tabs for "Enlarged View," "Graph," and "Adjustment." In the example shown in Fig. 8, the display content when the "Enlarged View" tab is selected is illustrated, and a portion of the area displayed on the digital twin display section 33 is enlarged and displayed.

[0178] The data manager 10 can select a desired area in the digital twin display unit 33 to display an enlarged view of that area on the operation / confirmation information display unit 34. The method and operation for selecting an area are not limited and may be set arbitrarily.

[0179] FIG. 9A is a schematic diagram showing an example of the display content when the "Graph" tab is selected in the operation / confirmation information display section 34.

[0180] When the "Graph" tab is selected, the contents of the "data to display" selected in the data selection operation unit 35 and a graph showing the probabilistic modeling of the data are displayed. In the example shown in Fig. 9, the predicted data (average) likelihood (standard deviation) and acquisition function for the "number of people" are displayed. Note that text explaining each element of the graph may also be displayed.

[0181] FIG. 9B is a schematic diagram showing an example of the display content when the "Adjustment" tab is selected in the operation / confirmation information display section 34.

[0182] When the "Adjustment" tab is selected, a GUI for selecting data to be adjusted is displayed as "data to be adjusted." An input section 38 for inputting specific parameters to be adjusted is also provided. Furthermore, a graph reflecting the results of parameter adjustment is also displayed.

[0183] The data manager 10 can adjust the digital twin 9 and the acquisition function by adjusting the parameters.

[0184] In this example, the data management GUI 32 provides functions such as visualization of main data (automatic), selection of data to be visualized (manual), display of graphs of probability models and acquisition functions, and adjustment of probability models and acquisition functions, etc. Of course, the functions are not limited to these, and various other functions may also be provided.

[0185] For example, any configuration may be adopted as the data management GUI 32 that allows confirmation or adjustment of at least one of the digital twin 9 (virtual model), detected data, predicted data (average), likelihood (standard deviation), and acquisition function.

[0186] FIG. 10 is a schematic diagram showing an example of a data collection GUI displayed on the data collector terminal 5. As shown in FIG.

[0187] In this embodiment, the data collection GUI 40, like the data management GUI 32, has a digital twin display section 33, an operation / confirmation information display section 34, and a data selection operation section 35. The operation / confirmation information display section 34 displays switchable tabs for "Enlarged View," "Sensing," "Plan Modification," and "Upload."

[0188] When the "Enlarged View" tab is selected, a portion of the area displayed on the digital twin display unit 33 is enlarged and displayed, similar to the example shown in Figure 8.

[0189] 10 shows an example of the display content when the "Sensing" tab is selected. The "Sensing" tab is selected when checking sensing plan information, and displays information on the "Type of data to be sensed" and "Sensing plan to be displayed."

[0190] The "type of sensing data" selected is "number of people" and "speed of people's movement." Of course, it is not limited to these items.

[0191] The "Sensing plan to display" displays the "area to be sensed," "movement route," and "multiple unit placement." When the "area to be sensed" is selected, the area to be sensed related to the "number of people" and "people movement speed" selected in the "type of data to be sensed" is displayed.

[0192] 10, the area displayed as a heat map in shades of gray and with a diagonal line drawn from the upper left to the lower right is area 41 where sensing should be performed for the "number of people." Additionally, the area displayed as a heat map in shades of gray and with a diagonal line drawn from the upper right to the lower left is area 42 where sensing should be performed for the "movement speed of people."

[0193] The heat map display corresponds to the "degree of data deficiency," with the darker the gray, the higher the "degree of data deficiency," i.e., the area with a high acquisition function value. In other words, the display of the area to be sensed corresponds to a visualization of the distribution of the acquisition function, with high and low values ​​being expressed by shades of gray.

[0194] When "Movement Route" is selected, the movement route of the data collection robot 15 for the number of "multiple robots deployed" (a single robot can also be entered) is displayed. The movement route is generated based on the distribution of a probability function so that the "area to be sensed" of "number of people" and "people movement speed" can be sensed. In the example shown in FIG. 10, the movement route of one data collection robot 15 is displayed.

[0195] 11 , for example, if two robots are entered in "Multiple robot placement," the movement paths of each of the two data collection robots 15 are illustrated on the digital twin display unit 33. In this way, in this example, planning information such as the next sensing position, placement of multiple robots, and movement paths of the robots is generated based on the distribution of the acquisition function.

[0196] The data collector 11 can select the "Sensing" tab and input the data selection, the number of data collection robots, etc., to check what kind of sensing plan will be created.

[0197] FIG. 12 is a schematic diagram showing an example of the display content when the "Plan Correction" tab is selected in the operation / confirmation information display unit 34. The "Plan Correction" tab is selected when it is desired to correct sensing plan information. In the example shown in FIG. 12, "Add / move waypoint" and "Move entire route" can be selected as "Modify travel route." Of course, the content that can be corrected is not limited and may be set arbitrarily.

[0198] In the example shown in Figure 12, "Add / move waypoint" is selected. When the data collector 11 selects a specific point on the digital twin display unit 33, a correction point 43 is displayed. The operation / confirmation information display unit 34 also displays the XYZ coordinates of the selected correction point 43. The method and operation for selecting the correction point 43 are not limited and may be set arbitrarily.

[0199] The selected correction point 43 is added as a waypoint, and the travel route is corrected. Alternatively, the waypoint on the original travel route may be slid by a drag operation or the like, and the travel route may be corrected. Other operation methods for correcting the travel route are not limited and may be set arbitrarily.

[0200] For example, if there is a stationary object on the movement path set as sensing plan information, the data collection robot 15 may collide with the stationary object. In such a case, the "Plan Modification" tab is selected and the movement path is modified by the data collector 11. This makes it possible to construct an appropriate sensing plan. Of course, there may be other reasons for modifying the sensing plan.

[0201] 13 is a schematic diagram showing an example of the display content when the "Upload" tab is selected on the operation / confirmation information display unit 34. The "Upload" tab is selected when uploading detection data. In this embodiment, uploading detection data corresponds to transmitting the detection data to the server device 3.

[0202] When the "Upload" tab is selected, a GUI is displayed for selecting the type of data to upload and the file of the data to upload as "Data to Upload." The data collector 11 can select, for example, "Number of people," and then select the file to upload from a pull-down menu. The selected file is sent to the server device 3, and the upload is completed.

[0203] In this example, the data collection GUI 40 provides functions such as visualization of main data (automatic), selection of data to be visualized (manual), proposal and display of data to be sensed (automatic), selection of data to be sensed (manual), confirmation of sensing locations and plans, correction of sensing plans, adjustment of movement routes and placement of multiple units, uploading of sensed data, etc. Of course, the functions are not limited to these, and various other functions may be installed.

[0204] For example, any configuration that allows confirmation or adjustment of plan information may be adopted as the data collection GUI 40.

[0205] FIG. 14 is a schematic diagram showing an example of a data utilization GUI displayed on the data user terminal 6. As shown in FIG.

[0206] In this embodiment, the data utilization GUI 45, like the data management GUI 32 and the data collection GUI 40, has a digital twin display unit 33, an operation / confirmation information display unit 34, and a data selection operation unit 35. The operation / confirmation information display unit 34 displays switchable tabs for "Enlarged View," "Download," and "Data Registration."

[0207] When the "Enlarged View" tab is selected, a portion of the area displayed on the digital twin display unit 33 is enlarged and displayed, similar to the example shown in Figure 8.

[0208] 14 shows an example of the display content when the "Download" tab is selected. The "Download" tab is selected when the data user 12 downloads data that he or she wants to use, and a GUI for specifying the data to be downloaded is displayed.

[0209] Specifically, a GUI is displayed for selecting the type of data to be downloaded. An input section 46 is also provided for specifying the "sensing range," "data certainty," "data density," etc.

[0210] When the type of data to be downloaded, as well as the "sensing range," "data certainty," and "data density" are specified, the corresponding data may be displayed on the digital twin display unit 33. When the data user 12 selects the download button 47, the specified data is downloaded.

[0211] 15 is a schematic diagram showing an example of the display content when the "Data Registration" tab is selected in the operation / confirmation information display unit 34. The "Data Registration" tab is selected when inputting missing data information, i.e., data information desired by the data user.

[0212] 15, a GUI is displayed that allows the user to select the type of data that is lacking as a "data collection request." An input section 48 is also provided for specifying the "sensing range," "data certainty," "data density," etc.

[0213] Furthermore, a comment input field 49 is provided for inputting comments such as what kind of data is desired, the degree of certainty of the data desired, and the degree of certainty of the data desired. When the data user 12 selects the send button 50, information on the missing data is sent to the server device 3 and registered.

[0214] In this example, the data utilization GUI 45 provides functions such as visualization of main data (automatic), selection of data to be visualized (manual), downloading data to be used / purchased, and registration of missing data. Of course, the GUI is not limited to these, and various other functions may be installed.

[0215] For example, any configuration that allows for specifying and acquiring at least one of detected data, predicted data (average), and likelihood (standard deviation) may be adopted as the data utilization GUI 45. Furthermore, any configuration that allows for input of desired data information regarding the digital twin 9 (virtual model) may be adopted as the data utilization GUI 45.

[0216] As described above, in the digital twin management system 1 according to this embodiment, a digital twin 9 of the detection target 8 is generated in virtual space so as to be synchronized with the detection target 8 in the real space, based on data of the detection target 8 obtained by the sensor system 2. In addition, recommended detection points for detecting data for the detection target 8 are calculated based on the data of the detection target 8. This makes it possible to generate a highly accurate digital twin 9.

[0217] There is a demand for sensing real space and faithfully recreating it in a virtual space. However, because real space changes constantly, it is difficult to keep track of everything and update the digital space.

[0218] For example, if it were possible to collect wide-area data with high accuracy using fewer sensing operations, this would be extremely effective in faithfully reproducing real space in a virtual space. Technology that can recognize which data should be updated in what order for overall efficiency would also be effective.

[0219] In this digital twin management system 1, by probabilistically modeling the virtual space based on data sensed in the real space, it becomes possible to highly accurately supplement data from areas that have not been sensed. By modeling probabilistically, it becomes clear that areas with low probability and uncertain data should be prioritized for sensing, further improving sensing efficiency.

[0220] For example, it is possible to prioritize sensing of areas far from the sensing position, areas with low accuracy due to sensor characteristics, areas where a long time has passed since the last sensing, etc., thereby improving sensing efficiency. Furthermore, it is possible to use optimization techniques to repeat sensing and modeling, further improving sensing efficiency. It is also possible to efficiently generate and update data over a wide area in virtual space by sensing the real space a small number of times.

[0221] There is also a demand for technology that can efficiently utilize data from virtual spaces. By handling data from virtual spaces probabilistically, this digital twin management system 1 makes it possible to utilize data based on its reliability. This makes it possible to improve the accuracy of predictions and learning.

[0222] For example, this technology can be applied not only to the picking system illustrated in FIG. 1 etc., but also to the reproduction of various real spaces (generation of a digital twin 9). For example, this technology can be applied to generating a digital twin 9 of a detection target 8 sensed by a satellite, using a satellite as the sensor system 2. When sensing using a satellite, it is possible to easily determine areas where the satellite's orbital period is insufficient and perform partial sensing, for example.

[0223] It is also possible to easily use a drone as the sensor system 2, first sensing the entire structure and then sensing finer points. This makes it possible to generate a highly accurate digital twin 9.

[0224] In addition, this technology can also be applied to generating digital twins 9 through sensing using various robots, or generating digital twins 9 through sensing by humans (operators).

[0225] This technology can also be used to build a platform business. For example, a business model can be built in which the data collector 11 shown in Figure 1 is provided with an index (acquisition function) of the points (locations) to be sensed, and the detected data is purchased at a price that varies depending on the size of the index. This would enable more necessary data to be sensed, resulting in a system in which data is collected in virtual space.

[0226] It is also possible to build a business model in which data is sold to data users 12 at prices that vary according to the reliability of the data (standard deviation). This makes it possible to effectively utilize all data held in virtual space. It also makes it possible to improve the reliability of the data.

[0227] For example, data for simulations, learning, and analysis are likely to be traded. For example, data on the movements of workers and forklifts, and the movement of pallets, etc., are likely to be traded as warehouse simulation data. Also, data on the density of people and their line of sight in commercial facilities are likely to be traded as marketing data. Of course, this data is not limited to these.

[0228] For the data user 12, the following data is considered to be highly valuable and will be purchased at a high price: Data with high probability (= data that is assumed to be close to the real space) Data with a low value of the acquisition function (= data with high accuracy, new data)

[0229] Furthermore, for the data collector 11, the following data is considered to be highly valuable and sellable at a high price: Data for which the value of the acquisition function becomes low when the data is shared with the system. For example, data with a high value of the acquisition function (= data that was insufficient, old data, data with low accuracy) and data with a low value of the acquisition function (= new data, data with high accuracy) are considered to be highly sellable data. It is also possible to adopt conditions such as data with a high value of the acquisition function and data with a low value of the acquisition function as conditions for data to sell at a high price.

[0230] The unit of data trading can be, for example, the number of data items. Specifically, it is also conceivable to trade data by specifying the sensing range (time and space), density, range of certainty, etc.

[0231] By using this technology, it will be possible to develop a platform business based on the above perspectives.

[0232] Furthermore, this technology makes it possible to create and propose simulation scenarios and learning data, taking into account the frequency of occurrence based on the reliability of the data, and to edit virtual spaces automatically or manually, making it possible to more faithfully reproduce real-world spaces for simulations and tests.

[0233] As a specific example of utilization that takes data reliability into consideration, for example, in a simulation to check the safety and work efficiency of a picking support robot based on the number of workers in each area of ​​a logistics warehouse, for areas where the reliability of the data on the number of workers is high, a simulation is run with one pattern of number of workers. For areas where the reliability is low, several scenarios with different numbers of workers are prepared and a simulation is run. Similarly, for learning, when the reliability is low, the number of patterns of learning data is increased and learning is carried out. This makes it possible to achieve simulations with extremely high accuracy.

[0234] This technology also makes it possible to analyze, visualize, and utilize the correlations between data, which makes it possible to determine the data to use for inference and learning based on the correlations, and to analyze and explain the reasons for inference and learning results.

[0235] This technology also makes it possible to use probabilistic data from virtual spaces to perform further inference and learning. For example, after inference and learning, it is possible to feed back the results from the real world into the virtual space data and acquisition function. This makes it possible to repeat the following loop: collect necessary data → improve the accuracy of inference and learning → get more use → increase feedback.

[0236] This technology also makes it possible to store probabilistic data in virtual space as metadata. For example, it makes it possible to compress the probability distribution and acquisition function for each point and perform visualization and calculations.

[0237] Simulations and predictions of real spaces using this technology can be applied to the following various fields. "Insurance": This technology can be applied to predicting when and where accidents are likely to occur, whether the risk is high, and calculating the probability of accidents occurring. "Sharing": This technology can be applied to predict where and how many robots should be deployed. "Billing": This technology can be applied to determining the appropriate billing method, such as subscription, pay-as-you-go, or dynamic pricing. "Community": This technology can be applied when you want to handle the same environment in virtual space / the real world from multiple locations. "Other uses": This technology can be applied when changing the placement of robots, products, cash registers, etc. depending on people's movement and congestion. For example, this technology can be applied to calculating the work efficiency of robots in a logistics warehouse.

[0238] Furthermore, for example, when integrating multiple digital twins 9, sensing of the real space may be necessary to compensate for differences in the accuracy (data quality) of each digital twin 9. In such cases, applying this technology makes it possible to calculate recommended detection points with high accuracy and efficiency, which is considered to be extremely effective. Of course, it is also possible to present multiple recommended detection points with a priority order.

[0239] Other Embodiments The present technology is not limited to the above-described embodiments, and various other embodiments can be realized.

[0240] This technology can be applied to the generation of any virtual model that is synchronized with the detection target 8 in the real world. For example, this technology can also be applied to the generation of virtual models that are not classified as "digital twins."

[0241] In each process described in the above embodiment, any machine learning algorithm using, for example, a DNN (Deep Neural Network), an RNN (Recurrent Neural Network), or a CNN (Convolutional Neural Network) may be used. For example, by using AI (artificial intelligence) that performs deep learning, each process can be executed with high accuracy. Of course, the application of a machine learning algorithm may be executed for any process within the present disclosure.

[0242] FIG. 16 is a block diagram showing an example of the hardware configuration of a computer 60 that can be used as the server device 3, the data manager terminal 4, the data collector terminal 5, the data user terminal 6, and the like.

[0243] The computer 60 includes a CPU 61, a ROM 62, a RAM 63, an input / output interface 65, and a bus 64 interconnecting these components. The input / output interface 65 is connected to a display unit 66, an input unit 67, a storage unit 68, a communication unit 69, a drive unit 70, and other components. The display unit 66 is a display device using, for example, an LCD or EL display. The input unit 67 is a keyboard, a pointing device, a touch panel, or other operating device. If the input unit 67 includes a touch panel, the touch panel may be integrated with the display unit 66. The storage unit 68 is a non-volatile storage device such as a HDD, flash memory, or other solid-state memory. The drive unit 70 is a device capable of driving a removable storage medium 71 such as an optical storage medium or magnetic recording tape. The communication unit 69 is a modem, router, or other communication device connectable to a LAN, WAN, or the like for communicating with other devices. The communication unit 69 may communicate via either a wired or wireless connection. The communication unit 69 is often used separately from the computer 60. Information processing by the computer 60 having the above-described hardware configuration is realized by cooperation between software stored in the storage unit 68 or the ROM 62, etc. and the hardware resources of the computer 60. Specifically, the information processing method according to the present technology is realized by loading a program constituting the software stored in the ROM 62, etc., into the RAM 63 and executing it. The program is installed in the computer 60 via, for example, the recording medium 71. Alternatively, the program may be installed in the computer 60 via a global network, etc. Alternatively, any computer-readable, non-transitory storage medium may be used.

[0244] The information processing method (digital twin management method / digital twin generation method / data acquisition method / data management method / data utilization method) and program according to the present technology may be executed by collaboration between multiple computers connected to each other via a network or the like, thereby constructing an information processing system or information processing device according to the present technology. In other words, the information processing method and program according to the present technology can be executed not only in a computer system composed of a single computer, but also in a computer system in which multiple computers operate in conjunction with each other. In this disclosure, a "system" refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

[0245] The execution of the information processing method and program according to the present technology by a computer system includes both cases where, for example, the generation of a digital twin (virtual model), the calculation of recommended detection points, the presentation of a GUI, the generation of plan information, etc. are performed by a single computer, and cases where each process is performed by a different computer. Furthermore, the execution of each process by a specific computer also includes having another computer execute part or all of the process and obtaining the results. In other words, the information processing method and program according to the present technology can also be applied to a cloud computing configuration in which a single function is shared and processed collaboratively by multiple devices via a network.

[0246] The digital twin management system, devices, GUI configurations, and processing flows for generating digital twins (virtual models), calculating recommended detection points, presenting GUIs, and generating planning information described with reference to the drawings are merely one embodiment and can be modified as desired without departing from the spirit of the present technology. In other words, any other configurations, algorithms, etc. for implementing the present technology may be adopted.

[0247] In this disclosure, terms such as "about," "approximately," "almost," and "roughly" may be used as appropriate to facilitate understanding of the description. However, there is no clear difference between using and not using terms such as "about," "approximately," "almost," and "approximately." In other words, in this disclosure, concepts that define shape, size, positional relationship, state, etc., such as "center," "middle," "uniform," and "equal," are concepts that include "substantially center," "substantially central," "substantially uniform," and "substantially equal." For example, states that fall within a predetermined range (e.g., a range of ±10%) based on "completely centered," "completely central," "completely uniform," and "completely equal" are also included. Therefore, even if terms such as "approximately," "almost," and "approximately" are not used, concepts expressed by adding "approximately," "almost," and "approximately" may be included. Conversely, states expressed by adding terms such as "approximately," "almost," and "approximately" do not necessarily exclude perfect states.

[0248] In the present disclosure, expressions using "than", such as "greater than A" and "smaller than A", are expressions that comprehensively include both concepts that include the case where it is equivalent to A and concepts that do not include the case where it is equivalent to A. For example, "greater than A" is not limited to cases that do not include equivalent to A, but also includes "A or greater". Furthermore, "smaller than A" is not limited to "less than A" but also includes "A or less". When implementing the present technology, specific settings and the like can be appropriately adopted from the concepts included in "greater than A" and "smaller than A" so that the effects described above can be achieved.

[0249] It is also possible to combine at least two of the features of the present technology described above. That is, the various features described in each embodiment may be arbitrarily combined without distinguishing between the embodiments. Furthermore, the various effects described above are merely examples and are not intended to be limiting, and other effects may also be achieved.

[0250] Note that the present technology may also have the following configurations. (1) An information processing system comprising: a generation unit that generates a virtual model of a detection target in a virtual space based on data of the detection target detected by a sensor system in real space so as to be synchronized with the detection target in the real space; and a calculation unit that calculates recommended detection points for detecting data for the detection target based on the data of the detection target. (2) The information processing system according to (1), wherein the generation unit generates the virtual model of the detection target based on data of the detection target at the recommended detection points. (3) The information processing system according to (1) or (2), wherein the detection target is configured in a predetermined three-dimensional area in the real space, the data of the detection target includes detection data for each detection point detected by the sensor system in the predetermined three-dimensional area, and the generation unit generates the virtual model by predicting data of each point in the predetermined three-dimensional area by probabilistic modeling based on the detection data for each detection point. (4) The information processing system according to (3), wherein the generation unit calculates the likelihood of prediction data, which is data for each of the points predicted by the probabilistic modeling. (5) The information processing system according to (4), wherein the calculation unit generates an acquisition function for each of the points based on the prediction data, which is data for each of the points predicted by the probabilistic modeling, and calculates the recommended detection points based on the acquisition function. (6) The information processing system according to (5), wherein the sensor system includes one or more sensors, and the calculation unit generates the acquisition function based on at least one of a distance of each of the points from the one or more sensors and a time elapsed since the most recent detection by the sensor system at each of the points. (7) The information processing system according to (5) or (6), wherein the calculation unit generates the acquisition function based on information of data desired by a user regarding the virtual model.(8) The information processing system according to any one of (5) to (7), wherein the calculation unit calculates a point at which the value of the acquisition function is maximum as the recommended detection point. (9) The information processing system according to any one of (5) to (8), further comprising: a GUI presentation unit that presents a GUI (Graphical User Interface) for managing data related to the virtual model, wherein the GUI for managing data related to the virtual model is configured to enable confirmation or adjustment of at least one of the virtual model, the detection data, the prediction data, the likelihood, and the acquisition function. (10) The information processing system according to any one of (1) to (9), further comprising: a GUI presentation unit that presents a GUI (Graphical User Interface) for collecting data of the detection target; and a plan information generation unit that generates plan information for executing detection of the data of the detection target by the sensor system, wherein the GUI for collecting the data of the detection target is configured to allow confirmation or correction of the plan information. (11) The information processing system according to (10), wherein the plan information generation unit generates the plan information based on the recommended detection points. (12) The information processing system according to any one of (4) to (9), further comprising: a GUI presentation unit that presents a GUI (Graphical User Interface) for utilizing data related to the virtual model, wherein the GUI for utilizing data related to the virtual model is configured to allow acquisition by specifying at least one of the detection data, the prediction data, and the likelihood. (13) The information processing system according to (12), wherein the GUI for utilizing data related to the virtual model is configured to enable input of desired data information related to the virtual model.(14) The information processing system according to any one of (1) to (13), wherein the virtual model is a digital twin of the detection target. (15) An information processing method executed by a computer system, based on data of the detection target detected by a sensor system in real space, generating a virtual model of the detection target in a virtual space so as to be synchronized with the detection target in the real space, and calculating recommended detection points for detecting data for the detection target based on the data of the detection target. (16) A program that causes a computer system to execute the following: based on data of the detection target detected by a sensor system in real space, generating a virtual model of the detection target in a virtual space so as to be synchronized with the detection target in the real space, and calculating recommended detection points for detecting data for the detection target based on the data of the detection target.

[0251] DESCRIPTION OF SYMBOLS 1...Digital twin management system 2...Sensor system 3...Server device 4...Data manager terminal 5...Data collector terminal 6...Data user terminal 8...Detection target 9...Digital twin 10...Data manager 11...Data collector 12...Data user 32...Data management GUI 33...Digital twin display section 34...Confirmation information display section 35...Data selection operation section 40...Data collection GUI 45...Data utilization GUI 60...Computer

Claims

1. An information processing system comprising: a generation unit that generates a virtual model of a detection target in a virtual space based on data of the detection target detected by a sensor system in real space so as to be synchronized with the detection target in the real space; and a calculation unit that calculates recommended detection points for detecting data for the detection target based on the data of the detection target.

2. An information processing system according to claim 1, wherein the generation unit generates the virtual model of the detection target based on data of the detection target at the recommended detection point.

3. An information processing system according to claim 1, wherein the detection target is configured in a predetermined three-dimensional area in the real space, the data of the detection target includes detection data for each detection point detected by the sensor system in the predetermined three-dimensional area, and the generation unit generates the virtual model by predicting data for each point in the predetermined three-dimensional area through probabilistic modeling based on the detection data for each detection point.

4. An information processing system according to claim 3, wherein the generation unit calculates the likelihood of predicted data, which is data for each point predicted by the probabilistic modeling.

5. An information processing system according to claim 4, wherein the calculation unit generates an acquisition function for each point based on predicted data, which is data for each point predicted by the probabilistic modeling, and calculates the recommended detection points based on the acquisition function.

6. An information processing system according to claim 5, wherein the sensor system includes one or more sensors, and the calculation unit generates the acquisition function based on at least one of the distance between the one or more sensors and each of the points and the time elapsed since the most recent detection by the sensor system at each of the points.

7. An information processing system according to claim 5, wherein the calculation unit generates the acquisition function based on information about data desired by a user regarding the virtual model.

8. An information processing system according to claim 5, wherein the calculation unit calculates the point at which the value of the acquisition function is maximum as the recommended detection point.

9. An information processing system according to claim 5, further comprising: a GUI presentation unit that presents a GUI (Graphical User Interface) for managing data related to the virtual model, wherein the GUI for managing data related to the virtual model is configured to enable confirmation or adjustment of at least one of the virtual model, the detection data, the prediction data, the likelihood, and the acquisition function.

10. An information processing system as claimed in claim 1, further comprising: a GUI presentation unit that presents a GUI (Graphical User Interface) for collecting data of the detection target; and a planning information generation unit that generates planning information for executing detection of the data of the detection target by the sensor system, wherein the GUI for collecting the data of the detection target is configured to allow confirmation or correction of the planning information.

11. An information processing system according to claim 10, wherein the plan information generation unit generates the plan information based on the recommended detection points.

12. An information processing system according to claim 4, further comprising: a GUI presentation unit that presents a GUI (Graphical User Interface) for utilizing data related to the virtual model; and the GUI for utilizing data related to the virtual model is configured to enable at least one of the detection data, the prediction data, and the likelihood to be specified and acquired.

13. An information processing system according to claim 12, wherein the GUI for utilizing data relating to the virtual model is configured to enable input of desired data information relating to the virtual model.

14. An information processing system according to claim 1, wherein the virtual model is a digital twin of the detection target.

15. An information processing method executed by a computer system, which generates a virtual model of a detection target in a virtual space based on data of the detection target detected by a sensor system in real space so as to be synchronized with the detection target in the real space, and calculates recommended detection points for detecting data for the detection target based on the data of the detection target.

16. A program that causes a computer system to execute the following: based on data of a detection target detected by a sensor system in real space, generate a virtual model of the detection target in a virtual space so that it is synchronized with the detection target in the real space; and based on the data of the detection target, calculate recommended detection points for detecting data for the detection target.

Images