Three-dimensional data processing system and three-dimensional data processing method

Abstract

To share a real-time situation on site and motions of multiple persons in remote locations in real time, while inhibiting the load on a computer.SOLUTION: A virtual three-dimensional data processing system is constituted of a computer having a calculation device that performs prescribed processing and an output device that outputs a result of the calculation of the calculation device. The virtual three-dimensional data processing system includes: an object recognition unit configured to identify an area of each object included in three-dimensional data acquired by a three-dimensional sensor; a data structuring unit configured to arrange three-dimensional data into a hierarchical structure; a motion determination unit configured to identify motion of a moving object included in the three-dimensional data; a relevance calculation unit configured to calculate a relevance of the motion of the moving object to each object; and a detail level control unit configured to control a level of detail at which each object is displayed, based on the calculated relevance. The output device outputs information according to the controlled level of detail.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a three-dimensional data processing system. [Background technology]

[0002] There are times when multiple people in different locations need to share information. For example, when equipment at a site breaks down, a skilled maintenance technician may travel to that site to provide maintenance guidance. In order for a skilled maintenance technician to travel to a remote site, schedule adjustments are required, which delays repair work and incurs travel costs. On the other hand, when receiving guidance from a skilled maintenance technician using a remote conference system, there is a problem that it is difficult to provide accurate guidance verbally or by sharing images.

[0003] Digital twins, which reproduce real-world conditions on a computer, only reproduce the on-site situation, and there is no mechanism for placing avatars on the digital twin and sharing the actions of multiple participants. Even if multiple avatars can interact with each other using a virtual three-dimensional space such as the metaverse, there is no mechanism for reflecting the on-site situation in the metaverse in real time.

[0004] The following prior art is included as background technology in this technical field. In Patent Document 1 (JP Patent Publication No. 2021-47610), when a worker wearing an MR-HMD observes a construction object in a space that is a construction site from various positions and in various directions, the three-dimensional shape of the construction object is measured by a terminal device from the image captured by the MR-HMD. The terminal device receives three-dimensional shape data representing the three-dimensional shape of the construction object, generates an image in which an input field for the inspection results regarding the construction of the construction object is superimposed on the three-dimensional shape of the construction object seen by the inspector in a virtual space in which the space and coordinate system are common, which is determined based on the three-dimensional shape data and the position and orientation of the VR-HMD worn by the inspector, and displays the image on the VR-HMD. A situation understanding support system is described in which the inspector enters the results of the inspection performed while viewing the three-dimensional shape of the construction object displayed on the VR-HMD into the input field.

[0005] Furthermore, Patent Document 2 (JP Patent Publication No. 2005-56075) describes a map display system having a map data processing unit that divides and processes 3D map data into scene graph data, in which the data structure of the 3D map is expressed in a tree structure, and drawing data for drawing objects included in the 3D map, and a map data display unit that identifies a display area by referring to the scene graph data, and reads and displays the drawing data corresponding to the identified display area.

[0006] Furthermore, Patent Document 3 (JP Patent Publication No. 2005-71285) describes a collision detection method including an interest level acquisition step of acquiring user interest level information for a subspace of a virtual space and / or an object in the virtual space, and a detail level determination step of dynamically changing and determining the level of detail of the object and the level of detail of collision detection according to the user interest level information acquired in the above step.

[0007] Also, Patent Document 4 (International Publication WO 2016 / 92698) describes a method for displaying image data more naturally according to the movement of an observer. The image processing device includes a storage means for storing a plurality of environment maps with different resolutions, an acquisition means for acquiring the observer's movement information, a storage means for storing a current area being displayed, a target area determination means for determining a target area based on the current area and the movement information, a route area determination means for determining an area including a route along which the line of sight changes from the current area to the target area as a route area, a setting means for setting a resolution corresponding to each of the route area and the target area, and a generation means for reading out image data corresponding to each of the route area and the target area from an environment map having a resolution set by the setting means, and generating image data to be displayed in the route area and image data to be displayed in the target area. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Patent Publication No. 2021-47610 [Patent Document 2] JP 2005-56075 A [Patent Document 3] JP 2005-71285 A [Patent Document 4] International Publication No. 2016 / 92698 Summary of the Invention [Problem to be solved by the invention]

[0009] The situation understanding support system described in the above-mentioned Patent Document 1 does not have a mechanism for sharing the real-time situation at the site and the actions of multiple people in remote locations in real time, and there is a problem that it is difficult to provide appropriate guidance to the site from a remote location. In addition, for real-time processing, it is desirable to reduce the load on the computer by displaying the necessary area in high resolution and the unnecessary area in low resolution. The technologies described in the above-mentioned Patent Documents 2 to 4 change the resolution for each area, but do not control the resolution of the area required depending on the human work.

[0010] The present invention aims to reduce the load on computers by lowering the resolution of areas with low relevance when sharing the real-time situation at a site and the actions of multiple workers in remote locations in real time. [Means for solving the problem]

[0011] A representative example of the invention disclosed in the present application is as follows: That is, a virtual three-dimensional data processing system is configured by a computer having an arithmetic unit that executes predetermined processing and an output device that outputs the results of the arithmetic unit, and has an object recognition unit that identifies the area of ​​each object included in three-dimensional data acquired by a three-dimensional sensor, a data structuring unit that arranges the three-dimensional data into a hierarchical structure, a motion determination unit that identifies the motion of a moving object included in the three-dimensional data, an association degree calculation unit that calculates the association degree between the motion of the moving object and the object, and a detail level control unit that controls the detail level of displaying the object based on the association degree, and the output device outputs information according to the controlled detail level. Effect of the Invention

[0012] According to one aspect of the present invention, it is possible to reduce the load on a computer when sharing three-dimensional data in real time. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0013] [Figure 1] 1 is a diagram illustrating a configuration of an information sharing system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing the physical configuration of a computer provided in the information sharing system of the present embodiment. [Figure 3A] FIG. 2 is a diagram showing an overview of on-site sensing in this embodiment. [Figure 3B] FIG. 13 is a diagram showing an overview of another on-site sensing in this embodiment. [Figure 4] FIG. 1 is a diagram illustrating an example of a logical configuration of an MEC server according to a first embodiment. [Diagram 5] 5 is a flowchart of a process executed by the MEC server shown in FIG. 4. [Figure 6] FIG. 13 is a diagram illustrating another example of a logical configuration of the MEC server of the first embodiment. [Figure 7] 7 is a flowchart of a process executed by the MEC server shown in FIG. 6. [Figure 8] FIG. 11 is a diagram illustrating an example of calculation of the relevance in the first embodiment. [Figure 9] FIG. 11 is a diagram illustrating an example of calculation of the relevance in the first embodiment. [Figure 10] FIG. 11 is a diagram illustrating an example of calculation of the level of detail in the first embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of control of the level of detail in the first embodiment. [Figure 12] FIG. 11 is a diagram illustrating an example of control of the level of detail in the first embodiment. [Figure 13] 13 is a flowchart of a process executed by an MEC server according to the second embodiment. [Figure 14] FIG. 11 is a diagram showing data (distance matrix) representing distances between objects in the second embodiment. [Figure 15] FIG. 11 is a diagram showing data (distance matrix) representing distances between objects in the second embodiment. [Figure 16] FIG. 11 is a diagram showing a distance difference according to the second embodiment. [Figure 17] FIG. 13 is a diagram showing the calculated relevance in Example 2. [Figure 18] A diagram showing the logical configuration of an MEC server of a third embodiment. [Figure 19] 13 is a flowchart of a process executed by an MEC server according to a third embodiment. [Figure 20] FIG. 13 is a diagram illustrating an example of control of the level of detail in the third embodiment. [Figure 21] A diagram showing the logical configuration of the MEC server of the fourth embodiment. [Figure 22] 13 is a flowchart of a process executed by an MEC server according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] <Example 1> FIG. 1 is a diagram showing a configuration of an information sharing system according to an embodiment of the present invention.

[0015] The information sharing system of this embodiment includes a plurality of 3D sensors 10, an edge processing device 20 connected to the 3D sensors 10, an MEC server 40 that processes observation results by the 3D sensors 10, a network 30 that connects the edge processing device 20 to the MEC server 40, an MR device 50, a VR device 60, a 3D sensor 61 that observes a wearer of the VR device 60, and an edge processing device 62 connected to the 3D sensor 61. The information sharing system may include an administrator terminal 70.

[0016] The 3D sensor 10 is a sensor that observes the situation of the site to be shared in the virtual three-dimensional space (metaverse space) 100. The 3D sensor 10 may be capable of acquiring three-dimensional point cloud data, and may be, for example, a TOF camera that outputs an image with distance in which RGB data is assigned a distance D for each pixel. The 3D sensor 10 may be a stereo camera equipped with two CMOS (Complementary Metal Oxide Semiconductor) image sensors, a structured light type sensor that combines a projection pattern light emitting element and an image sensor, or a sensor device that combines a distance sensor and a simple RGB camera to adjust the relationship between pixels. Furthermore, a sensor that implements a function of estimating distance information for each pixel from an RGB image using machine learning or the like may be used. A plurality of 3D sensors 10 may be provided to cover a wide range of the site including the work range of the worker, and the observation ranges of the 3D sensors 10 may be installed so as to overlap each other. The 3D sensor 10 observes objects such as static objects whose shape and position do not change, such as equipment installed on-site and room structures, and dynamic objects whose shape and position change, such as vehicles, construction machinery, robots, workers, tools, and work objects.

[0017] The edge processing device 20 is a computer that generates 3D sensing data including a plurality of three-dimensional plane data and a human skeletal model from the point cloud data acquired by the 3D sensor 10. By the edge processing device 20 generating the 3D sensing data from the point cloud data, the amount of communication between the edge processing device 20 and the MEC server 40 can be reduced, and congestion on the network 30 can be suppressed.

[0018] The MEC server 40 is a computer that is provided on the network 30 and realizes edge computing, and in this embodiment, generates a virtual three-dimensional space 100 from the 3D sensing data collected from the edge processing device 20.

[0019] The network 30 is a wireless network suitable for data communication that connects the edge processing device 20 and the MEC server 40, and may be, for example, a high-speed, low-latency 5G network. Note that, when the edge processing device 20 is fixedly installed, a wired network may be used.

[0020] The MR device 50 is a device worn by a worker at a site to share the virtual three-dimensional space 100, and includes a processor for executing a program, a memory for storing programs and data, a network interface for communicating with the MEC server 40, and a display for displaying an image transmitted from the MEC server 40 (described later with reference to FIG. 6). The display may be a transparent type so that the wearer can view the surroundings through the display superimposed on the image transmitted from the MEC server 40. The MR device 50 may also include a camera for capturing an image in front of the wearer, and transmit the image captured by the camera to the MEC server 40. The MR device 50 may also display an image captured by a camera for capturing an image in front of the wearer, superimposed on the image transmitted from the MEC server 40. The MR device 50 may also include a camera for capturing an image of the wearer's eyes, and detect the line of sight of the wearer from the image captured by the camera.

[0021] The VR device 60 is a device worn by a person (hereinafter referred to as a remote person, for example, an expert) at a remote location away from the site in order to share the virtual three-dimensional space 100, and has a processor that executes programs, a memory that stores programs and data, a network interface that communicates with the MEC server 40, and a display that displays images (described later with reference to FIG. 6) transmitted from the MEC server 40. The VR device 60 may also have a camera that captures an image in front of the wearer, and transmit the image captured by the camera to the MEC server 40. When the VR device 60 is provided outside the network in which the MEC server 40 is provided, the VR device 60 and the MEC server 40 may be connected via a public network such as the Internet 80 or another dedicated network.

[0022] The 3D sensor 61 is a sensor that observes the situation of the wearer of the VR device 60 to be shared in the virtual three-dimensional space 100. The 3D sensor 61 may be capable of acquiring three-dimensional point cloud data, similar to the 3D sensor 10, and may use, for example, a TOF camera that outputs an image with distance in which the distance D for each pixel is added to the RGB data.

[0023] The edge processing device 62 is a computer that generates 3D sensing data including a plurality of three-dimensional plane data (a human skeleton model) from the point cloud data acquired by the 3D sensor 61. By the edge processing device 62 generating the 3D sensing data from the point cloud data, the amount of communication between the edge processing device 62 and the MEC server 40 can be reduced.

[0024] The manager terminal 70 is a computer used by an on-site manager who uses the information sharing system, and can display information on the virtual three-dimensional space 100 (for example, an overhead image).

[0025] The information sharing system of this embodiment may include a cloud 90 that forms a large-scale virtual three-dimensional space for sharing three-dimensional information collected from multiple MEC servers 40. The large-scale virtual three-dimensional space formed in the cloud 90 is an integration of the virtual three-dimensional spaces formed by the multiple MEC servers 40, and a large-scale virtual three-dimensional space over a wide range can be formed.

[0026] Access to the MEC server 40 from the MR device 50, the VR device 60, and the administrator terminal 70 may be authenticated by ID and password or by the unique address (for example, MAC address) of these devices to ensure security of the information sharing system.

[0027] Fig. 2 is a block diagram showing the physical configuration of a computer provided in the information sharing system of this embodiment. In Fig. 2, the MEC server 40 is shown as an example of a computer, but the edge processing devices 20 and 62 and the administrator terminal 70 may also have the same configuration.

[0028] The MEC server 40 of this embodiment is configured by a computer having a processor (CPU) 1, a memory 2, an auxiliary storage device 3, and a communication interface 4. The MEC server 40 may also have an input interface 5 and an output interface 8.

[0029] The processor 1 is a calculation device that executes programs stored in the memory 2. The processor 1 executes various programs to realize various functional units (e.g., the metaverse analysis function 400, etc.) of the MEC server 40. Note that some of the processes performed by the processor 1 by executing the programs may be executed by other calculation devices (e.g., hardware such as ASIC and FPGA).

[0030] The memory 2 includes a ROM, which is a non-volatile storage element, and a RAM, which is a volatile storage element. The ROM stores unchanging programs (e.g., BIOS) and the like. The RAM is a high-speed, volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the processor 1 and data used when the programs are executed.

[0031] The auxiliary storage device 3 is, for example, a large-capacity, non-volatile storage device such as a magnetic storage device (HDD) or a flash memory (SSD). The auxiliary storage device 3 also stores data used by the processor 1 when executing a program and the program executed by the processor 1. That is, the program is read from the auxiliary storage device 3, loaded into the memory 2, and executed by the processor 1 to realize each function of the MEC server 40.

[0032] The communication interface 4 is a network interface device that controls communication with other devices (for example, the edge processing device 20, the cloud 90) in accordance with a predetermined protocol.

[0033] The input interface 5 is an interface to which input devices such as a keyboard 6 and a mouse 7 are connected and which receives input from an operator. The output interface 8 is an interface to which output devices such as a display device 9 and a printer (not shown) are connected and which outputs the results of program execution in a format that can be viewed by the user. Note that a user terminal connected to the MEC server 40 via a network may provide the input device and the output device. In this case, the MEC server 40 may have a web server function and the user terminal may access the MEC server 40 using a predetermined protocol (e.g., http).

[0034] The programs executed by the processor 1 are provided to the MEC server 40 via a removable medium (such as a CD-ROM or a flash memory) or a network, and are stored in a non-volatile auxiliary storage device 3, which is a non-transitory storage medium. For this reason, it is preferable that the MEC server 40 has an interface for reading data from the removable medium.

[0035] The MEC server 40 is a computer system configured on one physical computer, or on multiple logically or physically configured computers, and may operate on a virtual computer constructed on multiple physical computer resources. For example, each functional unit may operate on a separate physical or logical computer, or multiple functional units may be combined to operate on a single physical or logical computer.

[0036] In the embodiment of the present invention, as shown in Fig. 3A, a case will be mainly described in which an avatar of a worker wearing an MR device 50, a three-dimensional image of the site where the worker is working, and an avatar of a remote person in a remote location wearing a VR device 60 are constructed in a virtual three-dimensional space 100, but as another form shown in Fig. 3B, it is also applicable to a case in which an avatar of a worker simulating a robot working at the site, a three-dimensional image of the site where the robot is working, and an avatar of a remote person in a remote location wearing a VR device 60 are constructed in the virtual three-dimensional space 100. The moving object in this embodiment is an object that moves by itself to perform work, such as a worker or a robot.

[0037] FIG. 4 is a diagram showing an example of the logical configuration of the MEC server 40 in the first embodiment, and shows the functions of the MEC server 40 in the case shown in FIG. 3A, i.e., when the avatars of the field workers and the avatars of the remote parties are configured in a virtual three-dimensional space.

[0038] The MEC server 40 includes a data acquisition unit 401 that processes data acquired from the MR device 50, a self-position estimation unit 402, an information display unit 403, a motion determination unit 404, a task data management unit 405, a relevance calculation unit 406, a detail level determination unit 407, and a detail level control unit 408. The data acquisition unit 401 acquires image data output by the MR device 50, input data related to the task, and the like. The image data may be acquired in an RGBD format including distance data for each pixel. The self-position estimation unit 402 estimates the position and direction of the MR device 50. The position and direction of the MR device 50 may be estimated in a world coordinate system defined at the site, rather than in a local coordinate system of the MR device 50. The information display unit 403 superimposes three-dimensional object data stored in the 3D data management unit 415, and generates information to be displayed on the display of the MR device 50. As described above, the display of the MR device 50 does not have to be a see-through type, and when the wearer of the MR device 50 cannot see the outside world through the display, the information display unit 403 superimposes the front image captured by the camera and the three-dimensional object data stored in the 3D data management unit 415 to generate display data to be displayed on the display of the MR device 50. The motion determination unit 404 selects the next task to be performed by the worker, and determines the motion of the worker (e.g., the task being performed) from the worker's movement and input. For example, the motion determination unit 404 may select the next task to be performed by the worker from a work procedure manual that is previously tabulated, or may be configured with a machine learning model that is learned for each motion type by detecting the skeleton of the worker. The task data management unit 405 manages the data of the task determined by the motion determination unit 404 (e.g., whether the task is completed). The relevance calculation unit 406 calculates the relevance of the object in the vicinity of the worker to the task. For example, when the range of values ​​is from 0 to 1, if the object is the current task target, a value of 1 is set, and if not, a value less than 1 is set. The level of detail determination unit 407 determines the level of detail from the relevance calculated by the relevance calculation unit 406. The level of detail is, for example, a value indicating the depth level of the tree structure. The level of detail control unit 408 integrates data to be used in processing at a certain granularity based on the determined level of detail.For example, if the level of detail is low, multiple data are integrated and treated as a single piece of data, and if the level of detail is high, the data is treated at its original granularity.

[0039] The MEC server 40 also includes a data acquisition unit 411 for processing data acquired from the 3D sensor 10, a 3D map generation unit 412, an object recognition unit 413, a data structuring unit 414, and a 3D data management unit 415. The data acquisition unit 411 acquires image data output by the 3D sensor 10. The image data may be acquired in an RGBD format including distance data for each pixel. The 3D map generation unit 412 integrates point cloud data observed by a plurality of 3D sensors 10 to generate a three-dimensional map represented in a world coordinate system. The object recognition unit 413 recognizes objects included in the generated three-dimensional map. For example, the object recognition unit 413 may be configured with an object category and a machine learning model trained on a point cloud image. The data structuring unit 414 aligns data representing the recognized object in a tree structure such as an octree within a bounding box representing the object. The 3D data management unit 415 stores the three-dimensional map to which the object information is attached. The object information includes the object category and the created tree structure. The format in which the data structuring unit sorts data is not limited to a tree structure, but may be something like a pyramid structure, or may be any hierarchical structure in which a certain data granularity is defined for each hierarchical depth.

[0040] The MEC server 40 also has a data acquisition unit 421 that processes data acquired from the VR device 60, a coordinate conversion unit 422, and a rendering unit 423. The data acquisition unit 421 acquires image data output by the VR device 60 and operation data of a controller operated by a remote user (for example, a simulator that simulates an apparatus installed at the site, a wearable device worn by a remote user, etc.). The image data may be acquired in an RGBD format including distance data for each pixel. The coordinate conversion unit 422 converts a position in a local coordinate system held by the VR device 60 into a position in a world coordinate system defined at the site. The rendering unit 423 superimposes three-dimensional object data stored in the 3D data management unit 415 and generates display data to be displayed on the display of the VR device 60.

[0041] FIG. 5 is a flowchart of the process executed by the MEC server 40 shown in FIG.

[0042] The MEC server 40 judges whether an end command has been received from the user (S101), and ends the process when an end command has been received from the user.

[0043] On the other hand, if an end command has not been received from the user, the data acquisition unit 401 acquires image data, input data related to the work, and the like from the MR device 50 (S102).

[0044] Next, the self-position estimation unit 402 estimates the position and direction of the MR device 50 based on the information acquired from the MR device 50 (S103).

[0045] Next, the information display unit 403 superimposes the three-dimensional object data stored in the 3D data management unit 415, and generates information to be displayed on the display of the MR device 50 (S104).

[0046] Next, the movement determining unit 404 selects the next task that the worker is to perform, and determines the task that the worker is performing based on the movement and input of the worker (S105).

[0047] Next, the action determination unit 404 determines whether the task being performed by the worker is completed (S106). For example, the completion of the task can be determined based on an input by the worker to a touch panel or the MR device 50.

[0048] If the task is not completed, the action determination unit 404 refers to the task procedure manual and calculates the distance to the object that is used for the task (S107).

[0049] If the task is completed, the relevance calculation unit 406 calculates the relevance of the object (S108). For example, if the task for determining the relevance according to the work procedure manual is not completed, the relevance of the surrounding object is increased. Alternatively, the relevance of the object specified by the action determination unit 404 may be increased, or the relevance may be determined according to the distance from the MR device 50 (for example, the MR device 50 may calculate the range that the worker's hand can reach from the skeleton of the worker's hand estimated by the MR device 50, and the relevance of this range may be increased, or the position of the MR device 50 may be approximated as the position of the worker's hand, and the relevance of a range closer to the MR device 50 than a predetermined threshold may be increased).

[0050] Next, the level of detail determination unit 407 calculates a level of detail from the level of relevance calculated by the level of relevance calculation unit 406 (S109), and the level of detail control unit 408 sets the calculated level of detail for the object (S110).

[0051] Then, the rendering unit 423 renders the object based on the set level of detail (S111).

[0052] Figure 6 is a diagram showing another example of the logical configuration of the MEC server 40 in Example 1, and shows the functions of the MEC server 40 in the case of Figure 3B, i.e., when a worker avatar modeled after a robot working on site and a remote worker avatar are configured in a virtual three-dimensional space.

[0053] The MEC server 40 has a data acquisition unit 441, a self-location estimation unit 442, a work control unit 443, a work instruction unit 444, a work data management unit 445, a relevance calculation unit 446, a detail level determination unit 447, and a detail level control unit 448, which process data acquired from the MR device 50. The self-location estimation unit 442, the work data management unit 445, the relevance calculation unit 446, the detail level determination unit 447, and the detail level control unit 448 are the same as the self-location estimation unit 402, the work data management unit 405, the relevance calculation unit 406, the detail level determination unit 407, and the detail level control unit 408 shown in FIG. 4, respectively. The work data management unit 445 may have a procedure manual in advance as a table according to the specifications of the work robot. The data acquisition unit 441 acquires image data output by the work robot 110, data related to the work acquired by a sensor mounted on the work robot 110, and the like. The image data may be acquired in an RGBD format including distance data for each pixel. The work instructing unit 444 generates operation instructions for the work robot 110 based on data managed by the work data management unit 445. The work control unit 443 generates and outputs control signals for controlling the operation of the work robot 110 based on the work instructions generated by the work instructing unit 444. The relevance calculation unit 446 calculates the relevance of surrounding objects. For example, if a task for which a relevance is to be determined according to a work procedure manual has not been completed, the relevance of the surrounding objects is increased. Alternatively, even if the relevance of an object specified by the work instructing unit 444 is increased, the relevance may be determined according to the distance from the work robot 110 (for example, the closer to the arm of the work robot 110, the higher the relevance).

[0054] In the MEC server 40 configured as shown in FIG. 6, the level of detail of the 3D data used for collision judgment against an object can be changed in stages, thereby improving the accuracy of the collision judgment.

[0055] The logical configuration shown in FIG. 6 is implemented in MEC server 40, but it may also be implemented in the controller of work robot 110 or in a computer installed on-site.

[0056] FIG. 7 is a flowchart of the process executed by the MEC server 40 shown in FIG.

[0057] The MEC server 40 judges whether an end command has been received from the user (S121), and ends the process when an end command has been received from the user.

[0058] On the other hand, if an end command has not been received from the user, the data acquisition unit 441 acquires image data, input data related to the work, and the like from the MR device 50 (S122).

[0059] Next, the self-position estimation unit 442 estimates the position and direction of the MR device 50 based on the information acquired from the MR device 50 (S123).

[0060] Next, the work control unit 443 generates and outputs a control signal for controlling the operation of the work robot 110 based on the work instruction generated by the work instructing unit 444 (S124).

[0061] Next, relevance calculation unit 406 calculates the relevance of the object being worked on by work robot 110 (S125). The relevance calculation unit 406 also increases the relevance of objects in the vicinity of work robot 110 (for example, within a range where work robot 110 may collide), and decreases the relevance of objects far from work robot 110 (for example, within a range where work robot 110 is unlikely to collide). Note that a method of identifying an object being worked on by work robot 110 may involve implementing an object recognition algorithm that learns information about predetermined objects using information from a sensor implemented in work robot 110.

[0062] Next, the level of detail determination unit 407 calculates a level of detail from the level of relevance calculated by the level of relevance calculation unit 406 (S126), and the level of detail control unit 408 sets the calculated level of detail for the object (S127).

[0063] Next, the work control unit 443 determines whether the work robot 110 will collide with the object (S128). As a method for collision determination, for example, the minimum distance between the surface of its own CAD data held in advance and the surface of the detected object may be calculated, and the determination may be made based on whether the distance is less than a certain threshold value. If the distance is large, the determination may be made using each bounding box, and as the distance gets closer, the level of detail may be changed to a convex hull, non-convex hull, etc.

[0064] Next, the work control unit 443 determines whether the work has been completed (S129). As a result, if the work has been completed, the process proceeds to step S122. On the other hand, if the work is finished, the collision determination is continued (S128).

[0065] FIG. 8 and FIG. 9 are diagrams showing calculation examples of the relevance in the first embodiment.

[0066] First, the following conditions are set as the premise for the relevance calculation. · The relevance R of the object L in the work k kL is a value between 0 and 1, and is defined by the following formula. R kL ∈ [0, 1] R kL = 1 (when the object L is used in the work k) R kL < 1 (otherwise) · The related work k' (k' > k) performed after the work k ※ For reference to the control start point of the level of detail of the object L in the related work k'. · The related work k' is a related work of any work k < k'.

[0067] And for the object L in the work k', the relevance R in the work k" ∈ [k, k'] k"L is calculated by the following formula.

[0068]

Equation

[0069] In addition, if object L is not required for the subsequent task k (if object L is not associated with the task), the relevance R kL is calculated by the following formula:

number

[0070] Also, simply the standard time t k The relevance R when k"L is calculated by the following formula:

[0071]

number

[0072] Next, a specific calculation of the relevance will be described. Fig. 8 is a diagram showing an example of a task, and Fig. 9 is a diagram showing the relevance calculated for the task shown in Fig. 8.

[0073] As an example, calculate the relevance of object B in task 2. First, set the parameters k' and L of task 2 and object B to k'=2 and L=B. Since task 2 is a related task of task 1, set k=1. Then, the relevance R k"B is calculated by the following formula: R k"B =(k”-1+1) / (2-1+1)=k” / 2 At k”=1, R 1B is calculated by the following formula: R 1B =1 / 2=0.5

[0074] FIG. 10 is a diagram illustrating an example of calculation of the level of detail in the first embodiment.

[0075] First, the following conditions are set as prerequisites for calculating the level of specificity. ·Let the depth of the tree structure be D. For depth d∈[0,D], d=0 is the root node and d=D is the leaf node. d=0 corresponds to the bounding box of the object. · Level of detail of object L in task k kL is a value between 0 and 1, and is defined by the following formula. LoD kL ∈[0,1]

[0076] Then, the LoD of the object L in task k kL is calculated by the following formula, where min is a function that selects the minimum value, [ ] is a Gauss bracket, and is a function that rounds down the decimal part of the number in [ ] to an integer.

[0077]

number

[0078] Next, a specific calculation of the level of detail using the relevance shown in Fig. 9 will be described. As an example, when the depth D of the tree structure is 4, the level of detail for the object B in the task 1 is expressed by the following formula. LoD 1B =min([R 1B ×4+1],4) =min([0.5×4+1],4) =3 Similarly, LoD 1C =2, LoD 2C =3.

[0079] 11 and 12 are diagrams illustrating an example of controlling the level of detail in the first embodiment.

[0080] As shown in Fig. 11, when the depth of the tree structure is D=4, in the initial state, the area where the object is observed is controlled with the maximum level of detail d=4. When the level of detail LoD calculated using the relevance is 3, the grid of d=4 is integrated to match the size of d=3, and the area where the object exists is controlled and rendered with the maximum level of detail d=3, as shown in Fig. 12.

[0081] As described above, according to the first embodiment, the problem of in-field clips in which objects within the field of view that are less relevant to the work are displayed with a high level of detail, while objects outside the field of view that are more relevant to the work are displayed with a low level of detail or are not displayed at all, is resolved, and processing can be performed with a flexible level of detail according to the relationship to the work (high resolution for objects with a high level of relevance, and low resolution for objects with a low level of relevance), thereby reducing the load on the computer and enabling the real-time situation on site and the actions of multiple people in remote locations to be shared in real time.

[0082] <Example 2> Next, a description will be given of a second embodiment. In the second embodiment, the position of a detected object is updated at any time. In the second embodiment, differences from the first embodiment will be mainly described below, and the same configurations and processes as those in the first embodiment will be denoted by the same reference numerals, and the description thereof will be omitted.

[0083] FIG. 13 is a flowchart of a process executed by the MEC server 40 according to the second embodiment.

[0084] The MEC server 40 judges whether an end command has been received from the user (S141), and ends the process when an end command has been received from the user.

[0085] On the other hand, if no end command has been received from the user, the data acquisition unit 401 acquires image data, input data related to the work, and the like from the MR device 50 (S142).

[0086] Next, the self-position estimation unit 402 estimates the position and direction of the MR device 50 based on the information acquired from the MR device 50 (S143).

[0087] Next, the operation determination unit 404 refers to the work procedure manual and calculates the distance to the object that is the object used in the work. Also, the distance between the surrounding objects is calculated (S144). Since there are many combinations of objects, it is advisable to calculate the distances of the combinations of objects related to the current work, rather than all objects, and store them in a distance matrix in the 3D data management unit 415.

[0088] Next, the information display unit 403 places or moves the object according to the distance calculated in step S144, and updates the three-dimensional data (S145).

[0089] Next, the information display unit 403 superimposes the data of the object on the three-dimensional object data stored in the 3D data management unit 415, and generates information to be displayed on the display of the MR device 50 (S146).

[0090] Next, the movement determining unit 404 selects the next task that the worker is to perform, and determines the task that the worker is performing based on the movement and input of the worker (S147).

[0091] If the task is completed, the relevance calculation unit 406 calculates the relevance of the object (S148). For example, if the task of determining the relevance according to the work procedure manual is not completed, the relevance of the surrounding objects is increased. Alternatively, even if the relevance of the object specified by the action determination unit 404 is increased, the relevance may be determined according to the distance from the MR device 50 (for example, the relevance is increased for an object within the reach of the worker). Note that it is not necessary to calculate the relevance of an object that is beyond a predetermined distance (for example, 3 m).

[0092] Next, the level of detail determination unit 407 calculates a level of detail from the level of relevance calculated by the level of relevance calculation unit 406 (S149), and the level of detail control unit 408 sets the calculated level of detail for the object (S150).

[0093] Then, the object is rendered based on the set level of detail (S151).

[0094] 14 and 15 are diagrams showing data (distance matrix) indicating the distance between objects in the second embodiment.

[0095] Task 1 is related to tasks 2 and 3, and in task 1, object A is the related object, in task 2, objects A and B are the related objects, and in task 3, objects A and C are the related objects. Therefore, when calculating the distance between objects in task 1, the mutual distances between objects A, B and C are calculated. In this state (time 1), the distances between objects A, B and C are as shown in Figure 14. At time 2 when the task has progressed and objects A and B come close to each other, the distances between objects A, B and C are as shown in Figure 15.

[0096] FIG. 16 is a diagram showing the distance difference between time 1 (FIG. 14) and time 2 (FIG. 15), and FIG. 17 is a diagram showing the calculated relevance.

[0097] First, the following conditions are set as the premise for calculating the relevance. The distance between object L and object L' in task k is d LL' Let it be (k). d LL' (k)=d L'L (k)

[0098] In addition, the distance difference d between the object L and the object L' LL' and d L'L When the following relationship holds, it is not necessary to include object L or object L', or both, in the calculation of the relevance. d0 is a fixed distance (for example, 3 m). d LL' (k+1)-d LL' (k)=0 or d LL' (k+1)-d LL' (k)> d0

[0099] In the examples shown in Figures 14 and 15, only object A is moving, so only the distance to object A can be used. When multiple objects are moving nearby, the number of related tasks and objects increases, and simultaneous equations are set up for three or more tasks.

[0100] If the following formula is satisfied for the distance difference of all objects for which the relevance is calculated (i.e., the distance difference does not change between tasks), it is preferable to use a discrete change as in Example 1. Alternatively, the distance from the worker's own position may be used. d LL' (k+1)-d LL' (k)=0

[0101] Next, we will explain how to calculate the relevance. xB is calculated as follows:

[0102]

number

[0103] Calculate the relevance between tasks 1 and 2. R 1A =1, R 2A =1 R 1B =0.5, R 2B =1 R 1C = 0.33, R 2C =0.66 d AB (1)=0.7, d AB (2)=0 d AC (1)=0.85, d AC (2)=1(=d BC ) d BC (1)=0.1, d BC (2)=0.1 Therefore, the relevance is calculated as follows: R xB =-(5 / 7)×d AB (x)+1

[0104] In a task in which all distance differences are 0, the relevance may be calculated in the same manner as in the first embodiment.

[0105] As described above, according to the second embodiment, the position of a detected object is updated at any time, so that the relevance can be calculated at any time based on the distance between the worker and the object or between the objects, and step-like changes in the level of detail can be suppressed.

[0106] <Example 3> Next, a third embodiment will be described. In the third embodiment, if there is an area that does not meet the required level of detail, the position of the area is output for feedback to compensate for the level of detail. In the third embodiment, differences from the first embodiment will be mainly described, and the same configurations and processes as those in the first embodiment will be assigned the same reference numerals, and the description thereof will be omitted.

[0107] FIG. 18 is a diagram illustrating a logical configuration of the MEC server 40 according to the third embodiment.

[0108] The MEC server 40 includes a data acquisition unit 401 that processes data acquired from the MR device 50, a self-position estimation unit 402, an information display unit 403, a motion determination unit 404, an operation data management unit 405, a relevance calculation unit 406, a detail level determination unit 407, and a detail level control unit 408. The MEC server 40 also includes a data acquisition unit 411 that processes data acquired from the 3D sensor 10, a 3D map generation unit 412, an object recognition unit 413, a data structuring unit 414, and a 3D data management unit 415. The MEC server 40 also includes a data acquisition unit 421 that processes data acquired from the VR device 60, a coordinate conversion unit 422, and a rendering unit 423.

[0109] The detail control unit 408 determines the detail level of the display based on the determined detail level, and outputs information on the area that does not meet the required detail level to the 3D sensor 10. In the information sharing system of this embodiment, data acquired by multiple 3D sensors 10 is synthesized to generate a three-dimensional image that represents a virtual three-dimensional space. However, for example, if the number of pixels of the camera that captures each area is different, areas with low resolution may occur. In addition, there may be cases where the number of sensors is small and the entire circumference of the object cannot be covered, in which case data loss occurs at the relevant location. Furthermore, since the data density decreases according to the square of the distance, even if the number of sensors is sufficient, some data density may be reduced due to the convenience of placement. The 3D sensor 10 observes the area that does not meet the required detail level output from the MEC server 40. For example, the area that does not meet the required level of detail may be photographed by changing the camera's shooting direction, or the area that does not meet the required level of detail may be photographed by increasing the magnification of the area, or the movable 3D sensor 10 may be moved to a position where the area that does not meet the required level of detail can be photographed, and the area that does not meet the required level of detail may be photographed. The data acquisition unit 411 acquires image data output by the 3D sensor 10.

[0110] The level of detail control unit 408 may output information about the area that does not meet the required level of detail to the MR device 50. The MR device 50 may display information about the area that does not meet the required level of detail on a display device.

[0111] Furthermore, the MEC server 40 of the third embodiment may have a detail level complementing unit and may improve the detail level of an area that does not meet the required detail level by using newly acquired data from the area.

[0112] The other components and functions are the same as those in the first embodiment.

[0113] FIG. 19 is a flowchart of the process executed by the MEC server 40 according to the third embodiment.

[0114] The MEC server 40 determines whether an end command has been received from the user (S161), and ends the process when an end command has been received from the user.

[0115] On the other hand, if an end command has not been received from the user, the data acquisition unit 401 acquires image data, input data related to the work, and the like from the MR device 50 (S162).

[0116] Next, the self-position estimation unit 402 estimates the position and direction of the MR device 50 based on the information acquired from the MR device 50 (S163).

[0117] Next, the information display unit 403 superimposes the three-dimensional object data stored in the 3D data management unit 415, and generates information to be displayed on the display of the MR device 50 (S164).

[0118] Next, the movement determining unit 404 selects the next task that the worker is to perform, and determines the task that the worker is performing based on the movement and input of the worker (S165).

[0119] Next, the action determination unit 404 determines whether the task being performed by the worker is completed (S166). For example, the completion of the task can be determined based on an input by the worker to a touch panel or the MR device 50.

[0120] If the task is not completed, the action determination unit 404 refers to the task manual and calculates the distance to the object that is used in the task (S167).

[0121] If the task is completed, the relevance calculation unit 406 calculates the relevance of the object (S168). For example, if the task for determining the relevance according to the work procedure manual is not completed, the relevance of the surrounding objects is increased. Alternatively, even if the relevance of the object specified by the action determination unit 404 is increased, the relevance may be determined according to the distance from the MR device 50 (for example, the relevance of the object within the reach of the worker is increased).

[0122] Next, the detail level determination unit 407 calculates a detail level from the relevance level calculated by the relevance level calculation unit 406 (S169).

[0123] The level of detail control unit 408 judges whether the calculated level of detail is sufficient (S170). If there is an area where the calculated level of detail is insufficient, the level of detail control unit 408 identifies the area where the level of detail is insufficient, and outputs information about the identified area to the 3D sensor 10 (S171). An example of the judgment of whether the level of detail is sufficient will be described with reference to FIG. 20. Alternatively, a threshold value for the level of detail is determined in advance, and if there is no area where the maximum level of detail of the object is less than the threshold value, it is deemed to be sufficient.

[0124] If there is no area where the calculated level of detail is insufficient, level of detail control unit 408 sets the calculated level of detail for the object (S172).

[0125] Then, the rendering unit 423 renders the object based on the set level of detail (S173).

[0126] FIG. 20 is a diagram illustrating an example of controlling the level of detail in the third embodiment.

[0127] 20, when the depth D of the tree structure is 4, the area in which the object is observed is controlled with the maximum level of detail d=4. However, in some areas, the resolution is so low that the level of detail d=4 cannot be satisfied. For this reason, the level of detail control unit 408 identifies areas with insufficient level of detail, and outputs information on the identified areas to the 3D sensor 10.

[0128] As described above, according to the third embodiment, the position of an area that does not meet the required level of detail is output, making it possible to provide feedback to supplement the level of detail of the area, and additional 3D sensors can be used to add information to supplement the level of detail of the area, allowing the area to be displayed with the required level of detail.

[0129] <Example 4> Next, a fourth embodiment will be described. In the fourth embodiment, the level of detail is determined based on the distance between the object and the action estimated from the skeleton of the user. In the fourth embodiment, differences from the first embodiment will be mainly described, and the same configurations and processes as those in the first embodiment will be denoted by the same reference numerals, and the description thereof will be omitted.

[0130] FIG. 21 is a diagram illustrating a logical configuration of the MEC server 40 according to the fourth embodiment.

[0131] The MEC server 40 includes a data acquisition unit 401 that processes data acquired from the MR device 50, a self-position estimation unit 402, an information display unit 403, a motion determination unit 404, an operation data management unit 405, a relevance calculation unit 406, a detail level determination unit 407, and a detail level control unit 408. The MEC server 40 also includes a data acquisition unit 411 that processes data acquired from the 3D sensor 10, a 3D map generation unit 412, an object recognition unit 413, a data structuring unit 414, and a 3D data management unit 415. The MEC server 40 also includes a data acquisition unit 421 that processes data acquired from the VR device 60, a coordinate conversion unit 422, and a rendering unit 423. The MEC server 40 also includes a data acquisition unit 431 that processes data acquired from the skeleton acquisition device 120, and a distance calculation unit 432.

[0132] The data acquisition unit 431 acquires image data output by the skeleton acquisition device 120. The distance calculation unit 432 estimates the human skeleton from the position and direction of the MR device 50 estimated by the self-position estimation unit 402 and the image data output by the skeleton acquisition device 120, calculates the distance between the user's body and the object, and outputs the distance to the relevance calculation unit 406.

[0133] The relevance calculation unit 406 calculates the relevance of the object according to the distance calculated by the distance calculation unit 432 , and sends the calculated relevance to the detail level determination unit 407 .

[0134] The skeleton acquisition device 120 acquires data for estimating the skeleton of a person to be subjected to skeleton estimation and distance data for each pixel. For example, it may be a camera that captures a whole-body image of a person wearing the MR device 50 to be subjected to skeleton estimation.

[0135] By determining the relevance in this manner, for example, since the estimated skeleton is closely related to human actions such as field of view and viewpoint movement based on work procedures, the relevance related to actions can be calculated based on the results of skeleton estimation, and the level of detail related to actions can be determined.

[0136] The other components and functions are the same as those in the first embodiment.

[0137] FIG. 22 is a flowchart of the process executed by the MEC server 40 according to the fourth embodiment.

[0138] The MEC server 40 judges whether an end command has been received from the user (S181), and ends the process when an end command has been received from the user.

[0139] On the other hand, if no end command has been received from the user, the data acquisition unit 401 acquires image data, input data related to the work, and the like from the MR device 50 (S182).

[0140] Next, the self-position estimation unit 402 estimates the position and direction of the MR device 50 based on the information acquired from the MR device 50 (S183).

[0141] Next, the distance calculation unit 432 estimates the person's three-dimensional skeleton from the position and direction of the MR device 50 estimated by the self-position estimation unit 402 and the image data and distance data output by the skeleton acquisition device 120, and calculates the distance between the person's body (e.g., a hand) and the object (S184).

[0142] Next, the information display unit 403 superimposes the data of the object on the three-dimensional object data stored in the 3D data management unit 415, and generates information to be displayed on the display of the MR device 50 (S186).

[0143] Next, the movement determining unit 404 selects the next task that the worker is to perform, and determines the task that the worker is performing based on the movement and input of the worker (S187).

[0144] If the task is completed, the relevance calculation unit 406 calculates the relevance of the object according to the distance calculated by the distance calculation unit 432 (S188). For example, the relevance of an object within a predetermined distance from the estimated body is increased, and the relevance of an object outside the predetermined distance from the estimated skeleton is decreased. For example, a distance range (e.g., 0 m to 3 m) is determined in advance, and a value scaled from 0 to 1 according to the distance is calculated. Note that it is not necessary to calculate the relevance of an object that is even farther away (e.g., more than 3 m).

[0145] Next, the level of detail determination unit 407 calculates a level of detail from the level of relevance calculated by the level of relevance calculation unit 406 (S189), and the level of detail control unit 408 sets the calculated level of detail for the object (S190).

[0146] Then, the object is rendered based on the set level of detail (S191).

[0147] In the fourth embodiment, the relevance calculation unit 406 may calculate the relevance of an object not required for the first task to 0 indicating no relevance, and the relevance of an object required for the second task to 1 indicating relevance. Then, the relevance of an object required between the first task and the second task may be calculated to be a value between 0 and 1.

[0148] Furthermore, the relevance calculation section 406 may scan the time from the first task to the second task using a predetermined time window, and calculate the relevance at each time point scanned by the time window.

[0149] The present invention is not limited to the above-described embodiments, and includes various modified examples and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. Furthermore, a part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, a part of the configuration of each embodiment may be added, deleted, or replaced with another configuration.

[0150] In addition, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware, for example by designing some or all of them as an integrated circuit, or may be realized in software by a processor interpreting and executing a program that realizes each function.

[0151] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0152] In addition, the control lines and information lines shown are those considered necessary for the explanation, and do not necessarily show all the control lines and information lines necessary for implementation. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0153] 1 processor 2. Memory 3 Auxiliary storage 4. Communication Interface 5 Input Interface 6 Keyboard 7. Mouse 8 Output Interface 9 Display Devices 10, 61 3D sensor 20 Edge processing equipment 30 Network 40 MEC Server 50 MR Device 60 VR Devices 62 Edge Processing Equipment 70 Administrator terminal 80 Internet 90 Cloud 100 Virtual 3D Space 110 Working Robot 120 Skeleton Acquisition Device 200 On-site sensing function 210 Transmission Processing 220 3D sensing data generation processing 300 Remote sensing function 310 Motion Sensing Processing 400 Metaverse Analysis Function 401, 411, 421, 431 Data Acquisition Section 402 Self-position estimation part 403 Information display section 404 Operation determination section 405 Work Data Management Department 406 Relevance Calculation Unit 407 Level of Detail Determination Unit 408 Level of detail control section 410 Object Recognition Processing 412 3D Map Generation Unit 413 Object recognition unit 414 Data Structuring Department 415 3D Data Management Department 420 Motion Recognition Processing 422 Coordinate conversion section 423 Rendering Department 430 Skilled Sensor Processing 432 Distance calculation section 440 Motion Recognition Processing 441 Data Acquisition Department 442 Self-position estimation part 443 Work Control Section 444 Work Instruction Department 445 Work Data Management Department 446 Relevance Calculation Unit 447 Level of Detail Determination Unit 448 Level of detail control section 450 Task Recognition Processing 460 Accumulation Processing 470 Database 500 Feedback Function

Claims

1. A three-dimensional data processing system, comprising: A computer including an arithmetic unit for executing a predetermined process and an output unit for outputting a result of the arithmetic unit, an object recognition unit for identifying an area of ​​each object included in the three-dimensional data acquired by the three-dimensional sensor; a data structuring unit that arranges the three-dimensional data into a hierarchical structure; a motion determination unit for identifying a motion of a moving object included in the three-dimensional data; a relevance calculation unit for calculating a relevance between the motion of the moving object and the object; a level of detail control unit that controls a level of detail for displaying the object based on the relevance level, A three-dimensional data processing system, characterized in that the output device outputs information in accordance with the controlled level of detail.

2. 2. The three-dimensional data processing system according to claim 1, The relevance calculation unit is Calculating a relevance of the object based on a distance between the moving object and the object; A three-dimensional data processing system, characterized in that the degree of association is increased when the distance between the objects is short.

3. 3. A three-dimensional data processing system according to claim 2, The three-dimensional data processing system according to claim 1, wherein the motion determination unit calculates a distance to the moving object while the moving object is in motion.

4. 2. The three-dimensional data processing system according to claim 1, A three-dimensional data processing system, characterized in that the level of detail control unit outputs the position of at least a portion of an area of ​​the data for displaying the object when the level of detail of the area is insufficient to be controlled.

5. 3. A three-dimensional data processing system according to claim 2, The relevance calculation unit is The relevance of the object not required for the first action of the moving object is set to a first value indicating no relevance, A three-dimensional data processing system comprising: a calculation unit that calculates a degree of relevance of an object required for a second action of the moving object so that the degree of relevance becomes a second value indicating relevance.

6. 6. A three-dimensional data processing system according to claim 5, A three-dimensional data processing system characterized in that the relevance calculation unit calculates the relevance of the object required between the first action and the second action of the moving object so that the relevance is a value between the first value and the second value.

7. 7. A three-dimensional data processing system according to claim 6, The three-dimensional data processing system according to claim 1, wherein the relevance calculation unit scans a time from the first action to the second action using a predetermined time window, and calculates the relevance at each time point scanned by the time window.

8. 2. The three-dimensional data processing system according to claim 1, a distance calculation unit that estimates a skeleton of a worker, which is the moving object, and calculates a distance between the body of the worker and the object; The three-dimensional data processing system according to the present invention, wherein the relevance calculation unit calculates the relevance of the object by using the distance calculated by the distance calculation unit.

9. 9. A three-dimensional data processing system according to claim 8, A three-dimensional data processing system characterized in that the relevance calculation unit calculates the relevance so as to give a high relevance to objects within a predetermined distance from the body position obtained by skeleton estimation, and a low relevance to other objects.

10. 2. The three-dimensional data processing system according to claim 1, The three-dimensional data processing system is characterized in that the level of detail control unit outputs a level of detail at which the object is displayed in order to control the resolution of the output data from the three-dimensional sensor.

11. A computer-implemented three-dimensional data processing method, comprising the steps of: The computer includes an arithmetic unit that executes a predetermined arithmetic process, and an output unit that outputs a result of the arithmetic unit; The three-dimensional data processing method includes: an object identification step in which the computing device identifies the region of each object included in the three-dimensional data acquired by the three-dimensional sensor; a data structuring step in which the arithmetic device arranges the three-dimensional data into a hierarchical structure; a motion determination step in which the computing device identifies a motion of a moving object included in the three-dimensional data; a relevance calculation step in which the arithmetic device calculates a relevance between the motion of the moving object and the object; a level of detail control step in which the computing device controls a level of detail for displaying the object based on the relevance; and an output step in which the output device outputs information in accordance with the controlled level of detail.

Images