Three-dimensional data processing system and three-dimensional data processing method

Abstract

To share a real-time situation on site and motions of multiple persons in remote locations in real time, while inhibiting the load on a computer.SOLUTION: A virtual three-dimensional data processing system is constituted of a computer having a calculation device that performs prescribed processing and an output device that outputs a result of the calculation of the calculation device. The virtual three-dimensional data processing system includes: an object recognition unit configured to identify an area of each object included in three-dimensional data acquired by a three-dimensional sensor; a data structuring unit configured to arrange three-dimensional data into a hierarchical structure; a motion determination unit configured to identify motion of a moving object included in the three-dimensional data; a relevance calculation unit configured to calculate a relevance of the motion of the moving object to each object; and a detail level control unit configured to control a level of detail at which each object is displayed, based on the calculated relevance. The output device outputs information according to the controlled level of detail.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a three-dimensional data processing system. [Background technology]

[0002] There are situations where multiple people in different locations need to share information. For example, if equipment at a site breaks down, an experienced maintenance technician may need to travel to the site to provide maintenance instructions. Traveling to a remote site by an experienced maintenance technician requires scheduling, which delays the repair and incurs travel costs. However, when receiving instructions from an experienced maintenance technician using a remote conferencing system, there is a problem in that it is difficult to provide accurate instructions verbally or through image sharing.

[0003] Digital twins, which reproduce real-world situations on a computer, only reproduce the situation on-site, and do not have a mechanism for placing avatars on the digital twin and sharing the actions of multiple participants. Also, even if multiple avatars can interact with each other using a virtual three-dimensional space like the metaverse, there is no mechanism for reflecting the situation on-site in the metaverse in real time.

[0004] The following prior art exists as background technology in this technical field. Patent Document 1 (JP 2021-47610 A) describes a situation assessment support system in which an inspector enters the results of an inspection into an input field while viewing the three-dimensional shape of the construction work performed by the inspector in a virtual space with a common coordinate system, based on the three-dimensional shape data representing the three-dimensional shape of the construction work as seen by the inspector in a virtual space with a common coordinate system and determined based on the three-dimensional shape data and the position and orientation of the VR-HMD worn by the inspector.

[0005] Furthermore, Patent Document 2 (Japanese Patent Laid-Open Publication No. 2005-56075) describes a map display system having a map data processing unit that divides and processes three-dimensional map data into scene graph data that expresses the data structure of the three-dimensional map in a tree structure and drawing data for drawing objects included in the three-dimensional map, and a map data display unit that identifies a display area by referring to the scene graph data, and reads and displays the drawing data corresponding to the identified display area. description It has been done.

[0006] Furthermore, Patent Document 3 (JP Patent Publication No. 2005-71285) describes a collision detection method that includes an interest level acquisition step of acquiring user interest level information for a subspace of a virtual space and / or an object in the virtual space, and a detail level determination step of dynamically changing and determining the detail level of the object and the detail level of collision detection according to the user interest level information acquired in the step.

[0007] Furthermore, Patent Document 4 (WO 2016 / 92698) describes an image processing device for displaying image data in accordance with the viewer's line of sight, the image processing device including: a storage unit for storing a plurality of environment maps with different resolutions; an acquisition unit for acquiring movement information of the viewer; a storage unit for retaining a displayed current area; a target area determination unit for determining a target area based on the current area and the movement information; a route area determination unit for determining, as a route area, an area including a route along which the line of sight changes from the current area to the target area; a setting unit for setting resolutions corresponding to the route area and the target area; and a generation unit for reading image data corresponding to the route area and the target area from the environment map having the resolution set by the setting unit, and generating image data to be displayed in the route area and image data to be displayed in the target area. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Patent Publication No. 2021-47610 [Patent Document 2] Japanese Patent Application Laid-Open No. 2005-56075 [Patent Document 3] Japanese Patent Application Laid-Open No. 2005-71285 [Patent Document 4] International Publication No. 2016 / 92698 Summary of the Invention [Problem to be solved by the invention]

[0009] The situation understanding support system described in Patent Document 1 mentioned above does not have a mechanism for sharing the real-time situation at the site and the actions of multiple people in remote locations in real time, making it difficult to provide appropriate guidance to the site from a remote location. Furthermore, for real-time processing, it is desirable to reduce the computer load by displaying necessary areas at high resolution and unnecessary areas at low resolution. The technologies described in Patent Documents 2 to 4 mentioned above display different resolutions for each area, but do not control the resolution of the areas required depending on the person's work.

[0010] The present invention aims to reduce the computer load by lowering the resolution of areas with low relevance when sharing the real-time situation at a site and the actions of multiple workers in remote locations in real time. [Means for solving the problem]

[0011] A representative example of the invention disclosed in this application is as follows: three dimensionalA data processing system comprising a computer having an arithmetic unit that executes predetermined processing and an output device that outputs the results of the arithmetic unit, and further comprising an object recognition unit that identifies the area of ​​each object contained in three-dimensional data acquired by a three-dimensional sensor, a data structuring unit that arranges the three-dimensional data into a hierarchical structure, a movement determination unit that identifies the movement of a moving object contained in the three-dimensional data, a relevance calculation unit that calculates the relevance between the movement of the moving object and the object, and a detail control unit that controls the level of detail of displaying the object based on the relevance, and the output device outputs information according to the controlled level of detail. [Effects of the Invention]

[0012] According to one aspect of the present invention, it is possible to reduce the load on a computer when sharing three-dimensional data in real time. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a diagram illustrating a configuration of an information sharing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing the physical configuration of a computer provided in the information sharing system of the present embodiment. [Figure 3A] FIG. 1 is a diagram illustrating an overview of on-site sensing in this embodiment. [Figure 3B] FIG. 10 is a diagram illustrating an overview of another on-site sensing in this embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a logical configuration of an MEC server according to the first embodiment. [Figure 5] 5 is a flowchart of a process executed by the MEC server shown in FIG. 4. [Figure 6] FIG. 10 is a diagram illustrating another example of the logical configuration of the MEC server according to the first embodiment. [Figure 7] 7 is a flowchart of a process executed by the MEC server shown in FIG. 6. [Figure 8] FIG. 10 is a diagram illustrating an example of calculation of the relevance level in the first embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of calculation of the relevance level in the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of calculation of the level of detail in the first embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of controlling the level of detail in the first embodiment. [Figure 12] FIG. 10 is a diagram illustrating an example of controlling the level of detail in the first embodiment. [Figure 13] 10 is a flowchart of a process executed by an MEC server according to a second embodiment. [Figure 14] FIG. 10 is a diagram showing data (distance matrix) representing distances between objects in the second embodiment. [Figure 15] FIG. 10 is a diagram showing data (distance matrix) representing distances between objects in the second embodiment. [Figure 16] FIG. 10 is a diagram showing a distance difference according to the second embodiment. [Figure 17] FIG. 10 is a diagram showing the calculated relevance in Example 2. [Figure 18] FIG. 10 is a diagram illustrating the logical configuration of an MEC server according to a third embodiment. [Figure 19] 11 is a flowchart of a process executed by an MEC server according to a third embodiment. [Figure 20] FIG. 11 is a diagram illustrating an example of controlling the level of detail in the third embodiment. [Figure 21] FIG. 10 is a diagram illustrating the logical configuration of an MEC server according to a fourth embodiment. [Figure 22] 13 is a flowchart of a process executed by an MEC server according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Example 1 FIG. 1 is a diagram showing the configuration of an information sharing system according to an embodiment of the present invention.

[0015] The information sharing system of this embodiment includes a plurality of 3D sensors 10, an edge processing device 20 connected to the 3D sensors 10, an MEC server 40 that processes the observation results of the 3D sensors 10, a network 30 that connects the edge processing device 20 to the MEC server 40, an MR device 50, a VR device 60, a 3D sensor 61 that observes a wearer of the VR device 60, and an edge processing device 62 connected to the 3D sensor 61. The information sharing system may also include an administrator terminal 70.

[0016] The 3D sensor 10 is a sensor that observes the on-site situation to be shared in a virtual three-dimensional space (metaverse space) 100. The 3D sensor 10 can acquire three-dimensional point cloud data. For example, a time-of-flight (TOF) camera that outputs a distance-assigned image in which the distance D for each pixel is added to RGB data can be used. The 3D sensor 10 can also be a stereo camera equipped with two complementary metal oxide semiconductor (CMOS) image sensors, a structured light sensor that combines a projection pattern light-emitting element and an image sensor, or a sensor device that combines a distance sensor and a simple RGB camera to adjust the relationship between pixels. Furthermore, a sensor equipped with a function to estimate distance information for each pixel from an RGB image using machine learning or the like can also be used. Multiple 3D sensors 10 can be installed to cover a wide area of ​​the site, including the worker's work area, and the observation ranges of each 3D sensor 10 can be installed so that they overlap. The 3D sensor 10 observes static objects whose shape and position do not change, such as equipment installed on-site and room structures, and dynamic objects whose shape and position change, such as vehicles, construction machinery, robots, workers, tools, and work objects.

[0017] The edge processing device 20 is a computer that generates 3D sensing data including multiple three-dimensional plane data and a human skeletal model from the point cloud data acquired by the 3D sensor 10. By the edge processing device 20 generating 3D sensing data from the point cloud data, the amount of communication between the edge processing device 20 and the MEC server 40 can be reduced, and congestion on the network 30 can be alleviated.

[0018] The MEC server 40 is a computer that is provided on the network 30 and realizes edge computing, and in this embodiment, generates a virtual three-dimensional space 100 from the 3D sensing data collected from the edge processing device 20.

[0019] The network 30 is a wireless network suitable for data communication that connects the edge processing device 20 and the MEC server 40, and may be, for example, a high-speed, low-latency 5G network. Note that if the edge processing device 20 is installed in a fixed location, a wired network may also be used.

[0020] The MR device 50 is a device worn by a worker at a site to share the virtual three-dimensional space 100. The MR device 50 includes a processor for executing programs, a memory for storing programs and data, a network interface for communicating with the MEC server 40, and a display for displaying images transmitted from the MEC server 40 (described later with reference to FIG. 6). The display may be a transparent type so that the wearer can view the surroundings through the display, superimposed on the image transmitted from the MEC server 40. The MR device 50 may also include a camera for capturing a front view of the wearer, and transmit the image captured by the camera to the MEC server 40. The MR device 50 may also display an image captured by a camera for capturing a front view of the wearer, superimposed on the image transmitted from the MEC server 40. The MR device 50 may also include a camera for capturing a front view of the wearer, and detect the direction of the wearer's line of sight from the image captured by the camera.

[0021] The VR device 60 is a device worn by a person (hereinafter referred to as a remote person, for example, an expert) in a remote location away from the site in order to share the virtual three-dimensional space 100, and includes a processor for executing programs, memory for storing programs and data, a network interface for communicating with the MEC server 40, and a display for displaying images transmitted from the MEC server 40 (described later with reference to FIG. 6). The VR device 60 may also include a camera for capturing an image in front of the wearer, and transmit the image captured by the camera to the MEC server 40. When the VR device 60 is provided outside the network in which the MEC server 40 is provided, the VR device 60 and the MEC server 40 may be connected via a public network such as the Internet 80 or another dedicated network.

[0022] The 3D sensor 61 is a sensor that observes the situation of the wearer of the VR device 60 to be shared in the virtual three-dimensional space 100. The 3D sensor 61, like the 3D sensor 10, may be one that can acquire three-dimensional point cloud data, and for example, a TOF camera that outputs a distance image in which the distance D for each pixel is added to RGB data can be used.

[0023] The edge processing device 62 is a computer that generates 3D sensing data including a plurality of three-dimensional plane data (a human skeleton model) from the point cloud data acquired by the 3D sensor 61. By the edge processing device 62 generating the 3D sensing data from the point cloud data, the amount of communication between the edge processing device 62 and the MEC server 40 can be reduced.

[0024] The administrator terminal 70 is a computer used by an on-site administrator who uses the information sharing system, and can display information about the virtual three-dimensional space 100 (for example, an overhead image).

[0025] The information sharing system of this embodiment may include a cloud 90 that forms a large-scale virtual three-dimensional space for sharing three-dimensional information collected from multiple MEC servers 40. The large-scale virtual three-dimensional space formed in the cloud 90 is an integration of the virtual three-dimensional spaces formed by the multiple MEC servers 40, and can form a large-scale virtual three-dimensional space over a wide area.

[0026] Access to the MEC server 40 from the MR device 50, the VR device 60, and the administrator terminal 70 may be authenticated using an ID and password or the unique address (e.g., MAC address) of each device to ensure the security of the information sharing system.

[0027] 2 is a block diagram showing the physical configuration of a computer provided in the information sharing system of this embodiment. In FIG. 2, an MEC server 40 is shown as an example of a computer, but the edge processing devices 20, 62 and the administrator terminal 70 may also have the same configuration.

[0028] The MEC server 40 of this embodiment is configured by a computer having a processor (CPU) 1, a memory 2, an auxiliary storage device 3, and a communication interface 4. The MEC server 40 may also have an input interface 5 and an output interface 8.

[0029] The processor 1 is a computing device that executes programs stored in the memory 2. The processor 1 executes various programs to realize various functional units (e.g., the metaverse analysis function 400) of the MEC server 40. Note that some of the processing performed by the processor 1 by executing the programs may be executed by other computing devices (e.g., hardware such as ASIC and FPGA).

[0030] The memory 2 includes a ROM, which is a non-volatile storage element, and a RAM, which is a volatile storage element. The ROM stores unchanging programs (e.g., BIOS), etc. The RAM is a high-speed, volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the processor 1 and data used when the programs are executed.

[0031] The auxiliary storage device 3 is a large-capacity, non-volatile storage device such as a magnetic storage device (HDD) or flash memory (SSD). The auxiliary storage device 3 also stores data used by the processor 1 when executing a program and the program executed by the processor 1. That is, the program is read from the auxiliary storage device 3, loaded into the memory 2, and executed by the processor 1 to realize each function of the MEC server 40.

[0032] The communication interface 4 is a network interface device that controls communication with other devices (for example, the edge processing device 20, the cloud 90) in accordance with a predetermined protocol.

[0033] The input interface 5 is an interface to which input devices such as a keyboard 6 and a mouse 7 are connected and which receives input from an operator. The output interface 8 is an interface to which output devices such as a display device 9 and a printer (not shown) are connected and which outputs the results of program execution in a format that can be viewed by the user. Note that a user terminal connected to the MEC server 40 via a network may provide the input and output devices. In this case, the MEC server 40 may have web server functionality, and the user terminal may access the MEC server 40 using a predetermined protocol (e.g., http).

[0034] The programs executed by the processor 1 are provided to the MEC server 40 via removable media (CD-ROM, flash memory, etc.) or a network, and are stored in a non-volatile auxiliary storage device 3, which is a non-transitory storage medium. For this reason, the MEC server 40 should preferably have an interface for reading data from removable media.

[0035] The MEC server 40 is a computer system configured on a single physical computer or on multiple logically or physically configured computers, and may operate on a virtual computer constructed on multiple physical computer resources. For example, each functional unit may operate on a separate physical or logical computer, or multiple functional units may be combined to operate on a single physical or logical computer.

[0036] In the embodiment of the present invention, as shown in Fig. 3A, a virtual three-dimensional space 100 is mainly constructed in which an avatar of a worker wearing an MR device 50, a three-dimensional image of the site where the worker is working, and an avatar of a remote person in a remote location wearing a VR device 60 are constructed, but another form shown in Fig. 3B can also be applied to a virtual three-dimensional space 100 in which an avatar of a worker simulating a robot working on site, a three-dimensional image of the site where the robot is working, and an avatar of a remote person in a remote location wearing a VR device 60 are constructed. The moving object in this embodiment is an object that moves by itself to perform work, such as a worker or a robot.

[0037] Figure 4 is a diagram showing an example of the logical configuration of the MEC server 40 in Example 1, and shows the functions of the MEC server 40 in the case shown in Figure 3A, i.e., when the avatars of the field workers and the avatars of the remote parties are configured in a virtual three-dimensional space.

[0038] The MEC server 40 includes a data acquisition unit 401 that processes data acquired from the MR device 50, a self-location estimation unit 402, an information display unit 403, a motion determination unit 404, a task data management unit 405, a relevance calculation unit 406, a detail level determination unit 407, and a detail level control unit 408. The data acquisition unit 401 acquires image data output by the MR device 50, task-related input data, and the like. The image data may be acquired in an RGBD format that includes distance data for each pixel. The self-location estimation unit 402 estimates the position and orientation of the MR device 50. The position and orientation of the MR device 50 may be estimated using a world coordinate system defined on-site, rather than a local coordinate system of the MR device 50. The information display unit 403 superimposes three-dimensional object data stored in the 3D data management unit 415, and generates information to be displayed on the display of the MR device 50. As mentioned above, the display of the MR device 50 does not have to be transparent. If the wearer of the MR device 50 cannot see the outside world through the display, the information display unit 403 generates display data to be displayed on the display of the MR device 50 by superimposing a frontal image captured by a camera and three-dimensional object data stored in the 3D data management unit 415. The movement determination unit 404 selects the next task to be performed by the worker and determines the worker's movement (e.g., the task being performed) based on the worker's movements and input. For example, the movement determination unit 404 may select the next task to be performed by the worker from a pre-tabulated work procedure manual, or may be configured using a machine learning model trained for each type of movement by detecting the worker's skeleton. The task data management unit 405 manages the task data determined by the movement determination unit 404 (e.g., whether the task has been completed). The relevance calculation unit 406 calculates the relevance of objects around the worker to the task. For example, when the value ranges from 0 to 1, 1 is set if the object is currently being worked on, and a value less than 1 is set if it is not. The detail level determination unit 407 determines the detail level from the relevance calculated by the relevance level calculation unit 406. The detail level is, for example, a value indicating the depth level of the tree structure. The detail level control unit 408 integrates data to be used for processing at a certain granularity based on the determined detail level.For example, if the level of detail is low, multiple data are integrated and treated as one piece of data, and if the level of detail is high, the data is treated at its original granularity.

[0039] The MEC server 40 also includes a data acquisition unit 411, a 3D map generation unit 412, an object recognition unit 413, a data structuring unit 414, and a 3D data management unit 415 that process data acquired from the 3D sensor 10. The data acquisition unit 411 acquires image data output by the 3D sensor 10. The image data may be acquired in an RGBD format that includes distance data for each pixel. The 3D map generation unit 412 integrates point cloud data observed by multiple 3D sensors 10 to generate a 3D map expressed in a world coordinate system. The object recognition unit 413 recognizes objects included in the generated 3D map. For example, the object recognition unit 413 may be configured with an object category and a machine learning model trained using the point cloud image. The data structuring unit 414 arranges data representing the recognized object in a tree structure such as an octree within a bounding box representing the object. The 3D data management unit 415 stores the 3D map with object information attached. The object information includes the object category and the generated tree structure. The format in which the data structuring unit arranges data is not limited to a tree structure, but may be something like a pyramid structure, or any hierarchical structure in which a certain data granularity is defined for each hierarchical depth.

[0040] The MEC server 40 also has a data acquisition unit 421 that processes data acquired from the VR device 60, a coordinate conversion unit 422, and a rendering unit 423. The data acquisition unit 421 acquires image data output by the VR device 60 and operation data of a controller operated by a remote user (for example, a simulator that simulates equipment installed on-site, or a wearable device worn by the remote user). The image data may be acquired in an RGBD format that includes distance data for each pixel. The coordinate conversion unit 422 converts a position in a local coordinate system held by the VR device 60 into a position in a world coordinate system defined on-site. The rendering unit 423 superimposes three-dimensional object data stored in the 3D data management unit 415 and generates display data to be displayed on the display of the VR device 60.

[0041] FIG. 5 is a flowchart of the process executed by the MEC server 40 shown in FIG.

[0042] The MEC server 40 determines whether an end command has been received from the user (S101), and ends the process when an end command has been received from the user.

[0043] On the other hand, if an end command has not been received from the user, the data acquisition unit 401 acquires image data, input data related to the work, and the like from the MR device 50 (S102).

[0044] Next, the self-position estimation unit 402 estimates the position and direction of the MR device 50 based on the information acquired from the MR device 50 (S103).

[0045] Next, the information display unit 403 superimposes the three-dimensional object data stored in the 3D data management unit 415 and generates information to be displayed on the display of the MR device 50 (S104).

[0046] Next, the movement determining unit 404 selects the next task that the worker will perform, and determines the task that the worker is performing based on the worker's movements and inputs (S105).

[0047] Next, the operation determination unit 404 determines whether the work being performed by the worker is complete (S106). For example, the completion of the work can be determined based on an input by the worker to the touch panel or the MR device 50.

[0048] If the task is not completed, the action determination unit 404 refers to the task procedure manual and calculates the distance to the object that is used in the task (S107).

[0049] If the task is completed, the relevance calculation unit 406 calculates the relevance of the object (S108). For example, if the task for which the relevance is determined in accordance with the work procedure manual has not been completed, the relevance of the surrounding object is increased. Alternatively, even if the action determination unit 404 increases the relevance of the specified object, the relevance may be determined according to the distance from the MR device 50 (for example, the MR device 50 may calculate the range that the worker's hand can reach based on the estimated hand skeleton, and increase the relevance within this range, or the position of the MR device 50 may be approximated as the position of the worker's hand, and the relevance may be increased within a range that is closer to the MR device 50 than a predetermined threshold).

[0050] Next, the detail level determination unit 407 calculates the detail level from the relevance calculated by the relevance calculation unit 406 (S109), and the detail level control unit 408 sets the calculated detail level for the object (S110).

[0051] Then, the rendering unit 423 renders the object based on the set level of detail (S111).

[0052] Figure 6 is a diagram showing another example of the logical configuration of the MEC server 40 of Example 1, and shows the functions of the MEC server 40 in the case of Figure 3B, i.e., when a worker avatar modeled after a robot working on site and a remote worker avatar are configured in a virtual three-dimensional space.

[0053] The MEC server 40 includes a data acquisition unit 441, a self-location estimation unit 442, a work control unit 443, a work instruction unit 444, a work data management unit 445, a relevance calculation unit 446, a level of detail determination unit 447, and a level of detail control unit 448, which process data acquired from the MR device 50. The self-location estimation unit 442, the work data management unit 445, the relevance calculation unit 446, the level of detail determination unit 447, and the level of detail control unit 448 are the same as the self-location estimation unit 402, the work data management unit 405, the relevance calculation unit 406, the level of detail determination unit 407, and the level of detail control unit 408 shown in FIG. 4, respectively. The work data management unit 445 may be provided with a table containing a procedure manual tailored to the specifications of the work robot. The data acquisition unit 441 acquires image data output by the work robot 110, work-related data acquired by sensors implemented on the work robot 110, and the like. The image data may be acquired in an RGBD format including distance data for each pixel. The work instructing unit 444 generates operation instructions for the work robot 110 based on data managed by the work data management unit 445. The work control unit 443 generates and outputs control signals that control the operation of the work robot 110 based on the work instructions generated by the work instructing unit 444. The relevance calculation unit 446 calculates the relevance of surrounding objects. For example, if a task for which a relevance is determined in accordance with a work procedure manual has not been completed, the relevance of the surrounding objects is increased. Alternatively, even if the work instructing unit 444 increases the relevance of an object specified by the work instructing unit 444, the relevance may be determined according to the distance from the work robot 110 (for example, the closer the object is to the arm of the work robot 110, the higher the relevance).

[0054] In the MEC server 40 configured as shown in FIG. 6, the level of detail of the 3D data used for collision detection with an object can be changed in stages, thereby improving the accuracy of collision detection.

[0055] The logical configuration shown in FIG. 6 is implemented in MEC server 40, but it may also be implemented in the controller of work robot 110 or in a computer installed on-site.

[0056] FIG. 7 is a flowchart of the process executed by the MEC server 40 shown in FIG.

[0057] The MEC server 40 determines whether an end command has been received from the user (S121), and ends the process when an end command has been received from the user.

[0058] On the other hand, if an end command has not been received from the user, the data acquisition unit 441 acquires image data, input data related to the work, and the like from the MR device 50 (S122).

[0059] Next, the self-position estimation unit 442 estimates the position and direction of the MR device 50 based on the information acquired from the MR device 50 (S123).

[0060] Next, work control unit 443 generates and outputs control signals for controlling the operation of work robot 110 based on the work instructions generated by work instructing unit 444 (S124).

[0061] Next, the relevance calculation unit 406 calculates the relevance of the object being worked on by the work robot 110 (S125). The relevance calculation unit 406 also increases the relevance of objects in the vicinity of the work robot 110 (for example, within a range where the work robot 110 may collide), and decreases the relevance of objects far from the work robot 110 (for example, within a range where the work robot 110 is unlikely to collide). Note that the method of identifying the object being worked on by the work robot 110 may involve implementing an object recognition algorithm that learns information about predetermined objects using information from sensors implemented in the work robot 110.

[0062] Next, the detail level determination unit 407 calculates the detail level from the relevance calculated by the relevance calculation unit 406 (S126), and the detail level control unit 408 sets the calculated detail level for the object (S127).

[0063] Next, the work control unit 443 determines whether the work robot 110 will collide with the object (S128). For example, the collision determination method may be implemented by calculating the minimum distance between the surface of its own CAD data held in advance and the surface of the detected object, and determining whether it is below a certain threshold value. If the distance is far, the determination is made using each bounding box, and as the distance approaches, the level of detail may be changed to a convex hull, non-convex hull, etc.

[0064] Next, the work control unit 443 determines whether the work has been completed (S129). As a result, if the work has been completed, the process proceeds to step S122. On the other hand, if the work is completed, the collision determination is continued (S128).

[0065] FIG. 8 and FIG. 9 are diagrams showing calculation examples of the relevance in Example 1.

[0066] First, the following conditions are set as the premise for the relevance calculation. · The relevance R of the object L in the work k , is a value between 0 and 1, and is defined by the following formula. R kL ∈ [0, 1] R kL = 1 (when using the object L in the work k) R kL < 1 (otherwise) · The related work k' (k' > k) performed after the work k ※ For reference to the control start point of the level of detail of the object L in the related work k'. · The related work k' is a related work of any work k < k'.

[0067] And for the object L in the work k', the relevance R in the work k" ∈ [k, k'] k"L [[ID=4}} is calculated by the following formula.

[0068]

Equation

[0069] In addition, if object L is not required for the subsequent task k (if object L and the task are not related), the relevance R kL is calculated by the following formula:

number

[0070] Also, simply standard time t k Relevance R when influenced by k"L is calculated by the following formula:

[0071]

number

[0072] Next, a specific calculation of the relevance level will be described. Fig. 8 is a diagram showing an example of a task, and Fig. 9 is a diagram showing the relevance level calculated for the task shown in Fig. 8.

[0073] As an example, calculate the relevance of object B in task 2. First, set the parameters k' and L of task 2 and object B to k'=2 and L=B. Since task 2 is a related task of task 1, set k=1. Then, the relevance R k"B is calculated by the following formula: R k"B =(k”-1+1) / (2-1+1)=k” / 2 At k”=1, R 1B is calculated by the following formula: R 1B =1 / 2=0.5

[0074] FIG. 10 is a diagram illustrating an example of calculation of the level of detail in the first embodiment.

[0075] First, the following conditions are set as prerequisites for calculating the level of specificity. ·Let the depth of the tree structure be D. ·For depth d∈[0,D], d=0 is the root node and d=D is the leaf node. d=0 corresponds to the bounding box of the object. LoD of detail of object L in task k kL is a value between 0 and 1, and is defined by the following formula: LoD kL ∈[0,1]

[0076] Then, the level of detail LoD for object L in task k kL is calculated using the following formula: In the formula below, min is a function that selects the minimum value, [ ] is a Gauss bracket, and is a function that rounds down the decimal points of the number in [ ] to an integer.

[0077]

number

[0078] Next, a specific calculation of the level of detail using the relevance shown in Fig. 9 will be described. As an example, when the depth D of the tree structure is 4, the level of detail for object B in task 1 is expressed by the following formula. LoD 1B =min([R 1B ×4+1],4) =min([0.5×4+1],4) =3 Similarly, LoD 1C =2, LoD 2C =3.

[0079] 11 and 12 are diagrams illustrating an example of controlling the level of detail in the first embodiment.

[0080] As shown in Fig. 11, when the depth of the tree structure is D=4, in the initial state, the area where the object is observed is controlled with the maximum level of detail d=4. When the level of detail LoD calculated using the relevance is 3, the grid of d=4 is integrated to match the size of d=3, and as shown in Fig. 12, the area where the object exists is controlled and rendered with the maximum level of detail d=3.

[0081] As described above, according to the first embodiment, the problem of in-field clips in which objects in the field of view that are less relevant to the work are displayed with high detail, while objects outside the field of view that are more relevant to the work are displayed with low detail or are not displayed at all, is resolved, and processing can be performed with a flexible detail level according to the relationship to the work (high resolution for objects with high relevance, low resolution for objects with low relevance), and the real-time situation on site and the actions of multiple people in remote locations can be shared in real time while reducing the load on the computer.

[0082] <Example 2> Next, a second embodiment will be described. In the second embodiment, the position of a detected object is updated as needed. In the second embodiment, differences from the first embodiment will be mainly described, and the same configurations and processes as those in the first embodiment will be assigned the same reference numerals, and descriptions thereof will be omitted.

[0083] FIG. 13 is a flowchart of the process executed by the MEC server 40 according to the second embodiment.

[0084] The MEC server 40 determines whether an end command has been received from the user (S141), and ends the process when an end command has been received from the user.

[0085] On the other hand, if an end command has not been received from the user, the data acquisition unit 401 acquires image data, input data related to the work, and the like from the MR device 50 (S142).

[0086] Next, the self-position estimation unit 402 estimates the position and direction of the MR device 50 based on the information acquired from the MR device 50 (S143).

[0087] Next, the operation determination unit 404 refers to the work procedure manual and calculates the distance to the object that is the object used in the work. It also calculates the distance between surrounding objects (S144). Note that since there are many combinations of objects, it is preferable to calculate the distances of combinations of objects related to the current work, rather than all objects, and store them in a distance matrix in the 3D data management unit 415.

[0088] Next, the information display unit 403 places or moves the object according to the distance calculated in step S144, and updates the three-dimensional data (S145).

[0089] Next, the information display unit 403 superimposes the data of the object on the three-dimensional object data stored in the 3D data management unit 415, and generates information to be displayed on the display of the MR device 50 (S146).

[0090] Next, the movement determining unit 404 selects the next task that the worker will perform, and determines the task that the worker is performing based on the worker's movement and input (S147).

[0091] If the task is completed, the relevance calculation unit 406 calculates the relevance of the object (S148). For example, if the task for which the relevance is determined in accordance with the work procedure manual has not been completed, the relevance of the surrounding objects is increased. Alternatively, even if the action determination unit 404 increases the relevance of the specified object, the relevance may be determined according to the distance from the MR device 50 (for example, the relevance may be increased for objects within the worker's reach). Note that the relevance of objects beyond a predetermined distance (for example, 3 m) does not need to be calculated.

[0092] Next, the detail level determination unit 407 calculates the detail level from the relevance calculated by the relevance calculation unit 406 (S149), and the detail level control unit 408 sets the calculated detail level for the object (S150).

[0093] Then, the object is rendered based on the set level of detail (S151).

[0094] 14 and 15 are diagrams showing data (distance matrix) representing the distance between objects in the second embodiment.

[0095] Task 1 is related to tasks 2 and 3, and in task 1, object A is the related object, in task 2, objects A and B are the related objects, and in task 3, objects A and C are the related objects. Therefore, when calculating the distance between objects in task 1, the mutual distances between objects A, B, and C are calculated. In this state (time 1), the distances between objects A, B, and C are as shown in Figure 14. When the task progresses to time 2 and objects A and B come close to each other, the distances between objects A, B, and C become as shown in Figure 15.

[0096] FIG. 16 is a diagram showing the distance difference between time 1 (FIG. 14) and time 2 (FIG. 15), and FIG. 17 is a diagram showing the calculated relevance.

[0097] First, the following conditions are set as prerequisites for calculating the relevance. The distance between object L and object L' in task k is d LL' Let (k). ·d LL' (k)=d L'L (k)

[0098] In addition, the distance difference d between the object L and the object L' LL' and d L'L When the following relationship holds, it is not necessary to include object L or object L', or both, in the calculation of the relevance. d0 is a fixed distance (for example, 3 m). d LL' (k+1)-d LL' (k)=0 or d LL' (k+1)-d LL' (k)> d0

[0099] In the examples shown in Figures 14 and 15, only object A is moving, so it is sufficient to use only the distance to object A. When multiple objects are moving nearby, the number of related tasks and objects increases, and simultaneous equations are set up for three or more tasks.

[0100] If the following formula holds for the distance difference of all objects for which the relevance is calculated (i.e., the distance difference does not change between tasks), it is preferable to use a discrete change as in Example 1. Alternatively, the distance from the worker's own position may be used. d LL' (k+1)-d LL' (k)=0

[0101] Next, we will explain how to calculate the relevance. The relevance R between task x and object B is xB is calculated by the following formula:

[0102]

number

[0103] Calculate the relevance between tasks 1 and 2. R 1A =1, R 2A =1 R 1B =0.5, R 2B =1 R 1C =0.33, R 2C =0.66 d AB (1)=0.7, d AB (2)=0 d AC (1)=0.85, d AC (2)=1(=d BC ) d BC (1)=0.1, d BC (2)=0.1 Therefore, the relevance is calculated using the following formula: R xB =-(5 / 7)×d AB (x)+1

[0104] In a task where all distance differences are 0, the relevance may be calculated in the same manner as in the first embodiment.

[0105] As described above, according to the second embodiment, the position of the detected object is updated at any time, so that the relevance can be calculated at any time based on the distance between the worker and the object or between the objects, and step-like changes in the level of detail can be suppressed.

[0106] Example 3 Next, a third embodiment will be described. In the third embodiment, if there is an area that does not meet the required level of detail, the position of that area is output as feedback to compensate for the level of detail. In the following third embodiment, differences from the first embodiment will be mainly described, and the same configurations and processes as those in the first embodiment will be assigned the same reference numerals, and their description will be omitted.

[0107] FIG. 18 is a diagram illustrating a logical configuration of the MEC server 40 according to the third embodiment.

[0108] The MEC server 40 includes a data acquisition unit 401 that processes data acquired from the MR device 50, a self-position estimation unit 402, an information display unit 403, a motion determination unit 404, a task data management unit 405, a relevance calculation unit 406, a level of detail determination unit 407, and a level of detail control unit 408. The MEC server 40 also includes a data acquisition unit 411 that processes data acquired from the 3D sensor 10, a 3D map generation unit 412, an object recognition unit 413, a data structuring unit 414, and a 3D data management unit 415. The MEC server 40 also includes a data acquisition unit 421 that processes data acquired from the VR device 60, a coordinate conversion unit 422, and a rendering unit 423.

[0109] The level of detail control unit 408 determines the level of detail of the display based on the determined level of detail and outputs information on areas that do not meet the required level of detail to the 3D sensor 10. In the information sharing system of this embodiment, data acquired by multiple 3D sensors 10 is combined to generate a 3D image representing a virtual 3D space. However, for example, if the number of pixels of the cameras capturing each area differs, areas with low resolution may occur. Furthermore, there may be cases where the number of sensors is insufficient to cover the entire circumference of an object, resulting in data loss in the relevant areas. Furthermore, because data density decreases with the square of the distance, even if there are sufficient sensors, some data density may be low due to placement considerations. The 3D sensor 10 observes areas that do not meet the required level of detail output from the MEC server 40. For example, it is possible to change the shooting direction of the camera and shoot the area that does not meet the required level of detail with another camera, to shoot the area that does not meet the required level of detail with a higher magnification, or to move the movable 3D sensor 10 to a position where the area that does not meet the required level of detail can be shot, and then shoot the area that does not meet the required level of detail. The data acquisition unit 411 acquires the image data output by the 3D sensor 10.

[0110] The level of detail control unit 408 may output information about the area that does not meet the required level of detail to the MR device 50. The MR device 50 may display information about the area that does not meet the required level of detail on a display device.

[0111] Furthermore, the MEC server 40 of the third embodiment may have a detail level complementing unit, and may improve the detail level of an area by using data newly acquired from the area where the required detail level is not met.

[0112] The other components and functions are the same as those in the first embodiment.

[0113] FIG. 19 is a flowchart of the process executed by the MEC server 40 according to the third embodiment.

[0114] The MEC server 40 determines whether an end command has been received from the user (S161), and ends the process when an end command has been received from the user.

[0115] On the other hand, if an end command has not been received from the user, the data acquisition unit 401 acquires image data, input data related to the work, and the like from the MR device 50 (S162).

[0116] Next, the self-position estimation unit 402 estimates the position and direction of the MR device 50 based on the information acquired from the MR device 50 (S163).

[0117] Next, the information display unit 403 superimposes the three-dimensional object data stored in the 3D data management unit 415 and generates information to be displayed on the display of the MR device 50 (S164).

[0118] Next, the movement determining unit 404 selects the next task that the worker will perform, and determines the task that the worker is performing based on the worker's movement and input (S165).

[0119] Next, the action determination unit 404 determines whether the work being performed by the worker is complete (S166). For example, the completion of the work can be determined based on an input by the worker to the touch panel or the MR device 50.

[0120] If the task is not completed, the action determination unit 404 refers to the task procedure manual and calculates the distance to the object that is used in the task (S167).

[0121] If the task is completed, the relevance calculation unit 406 calculates the relevance of the object (S168). For example, if the task for which the relevance is determined according to the work procedure manual has not been completed, the relevance of the surrounding objects is increased. Alternatively, even if the action determination unit 404 increases the relevance of the object specified, the relevance may be determined according to the distance from the MR device 50 (for example, the relevance may be increased for objects within the worker's reach).

[0122] Next, the detail level determination unit 407 calculates the detail level from the relevance level calculated by the relevance level calculation unit 406 (S169).

[0123] The level of detail control unit 408 determines whether the calculated level of detail is sufficient (S170). If there is an area where the calculated level of detail is insufficient, the level of detail control unit 408 identifies the area where the level of detail is insufficient and outputs information about the identified area to the 3D sensor 10 (S171). An example of determining whether the level of detail is sufficient will be described with reference to FIG. 20. Alternatively, a threshold value for the level of detail may be determined in advance, and if there is no area where the maximum level of detail of the object is less than the threshold value, the level of detail is determined to be sufficient.

[0124] If there is no area where the calculated level of detail is insufficient, the level of detail control unit 408 sets the calculated level of detail for the object (S172).

[0125] Then, the rendering unit 423 renders the object based on the set level of detail (S173).

[0126] FIG. 20 is a diagram illustrating an example of controlling the level of detail in the third embodiment.

[0127] 20, when the depth D of the tree structure is 4, the area in which the object is observed is controlled with the maximum level of detail d=4. However, in some areas, the resolution is so low that the level of detail d=4 cannot be met. For this reason, the level of detail control unit 408 identifies areas where the level of detail is insufficient, and outputs information about the identified areas to the 3D sensor 10.

[0128] As described above, according to Example 3, the position of an area that does not meet the required level of detail is output, making it possible to provide feedback to supplement the level of detail of the area, and additional 3D sensors can be used to add information to supplement the level of detail of the area, allowing the area to be displayed with the required level of detail.

[0129] Example 4 Next, a fourth embodiment will be described. In the fourth embodiment, the level of detail is determined based on the distance between the object and the behavior estimated from the skeleton of the user. In the following fourth embodiment, differences from the first embodiment will be mainly described, and the same configurations and processes as those in the first embodiment will be assigned the same reference numerals, and descriptions thereof will be omitted.

[0130] FIG. 21 is a diagram illustrating a logical configuration of the MEC server 40 according to the fourth embodiment.

[0131] The MEC server 40 includes a data acquisition unit 401, a self-position estimation unit 402, an information display unit 403, a motion determination unit 404, a task data management unit 405, a relevance calculation unit 406, a level of detail determination unit 407, and a level of detail control unit 408, which process data acquired from the MR device 50. The MEC server 40 also includes a data acquisition unit 411, a 3D map generation unit 412, an object recognition unit 413, a data structuring unit 414, and a 3D data management unit 415, which process data acquired from the 3D sensor 10. The MEC server 40 also includes a data acquisition unit 421, a coordinate conversion unit 422, and a rendering unit 423, which process data acquired from the VR device 60. The MEC server 40 also includes a data acquisition unit 431, and a distance calculation unit 432, which process data acquired from the skeleton acquisition device 120.

[0132] The data acquisition unit 431 acquires image data output by the skeleton acquisition device 120. The distance calculation unit 432 estimates the human skeleton from the position and direction of the MR device 50 estimated by the self-position estimation unit 402 and the image data output by the skeleton acquisition device 120, calculates the distance between the user's body and the object, and outputs the calculated distance to the relevance calculation unit 406.

[0133] The relevance calculation unit 406 calculates the relevance of the object in accordance with the distance calculated by the distance calculation unit 432 , and sends the calculated relevance to the detail level determination unit 407 .

[0134] The skeleton acquisition device 120 acquires data for estimating the skeleton of a person whose skeleton is to be estimated and distance data for each pixel. For example, it may be a camera that captures a whole-body image of a person wearing the MR device 50 whose skeleton is to be estimated.

[0135] By determining the relevance in this manner, for example, since the estimated skeleton is closely related to human behavior such as field of view and viewpoint movements based on work procedures, the relevance related to behavior can be calculated based on the results of skeleton estimation, and the level of detail related to behavior can be determined.

[0136] The other components and functions are the same as those in the first embodiment.

[0137] FIG. 22 is a flowchart of the process executed by the MEC server 40 according to the fourth embodiment.

[0138] The MEC server 40 determines whether an end command has been received from the user (S181), and ends the process when an end command has been received from the user.

[0139] On the other hand, if an end command has not been received from the user, the data acquisition unit 401 acquires image data, input data related to the work, and the like from the MR device 50 (S182).

[0140] Next, the self-position estimation unit 402 estimates the position and direction of the MR device 50 based on the information acquired from the MR device 50 (S183).

[0141] Next, the distance calculation unit 432 estimates the person's three-dimensional skeleton from the position and direction of the MR device 50 estimated by the self-position estimation unit 402 and the image data and distance data output by the skeleton acquisition device 120, and calculates the distance between the person's body (e.g., hand) and the object (S184).

[0142] Next, the information display unit 403 superimposes the data of the object on the three-dimensional object data stored in the 3D data management unit 415, and generates information to be displayed on the display of the MR device 50 (S186).

[0143] Next, the movement determining unit 404 selects the next task that the worker will perform, and determines the task that the worker is performing based on the worker's movement and input (S187).

[0144] If this operation is completed, the relevance calculation unit 406 calculates the relevance of the object according to the distance calculated by the distance calculation unit 432 (S188). For example, the relevance of an object within a predetermined distance from the estimated body is increased, and the relevance of an object outside the predetermined distance from the estimated skeleton is decreased. For example, a distance range (e.g., 0 m to 3 m) is determined in advance, and a value scaled from 0 to 1 is calculated according to that distance. Note that it is not necessary to calculate the relevance of objects that are further away (e.g., greater than 3 m).

[0145] Next, the detail level determination unit 407 calculates the detail level from the relevance calculated by the relevance calculation unit 406 (S189), and the detail level control unit 408 sets the calculated detail level for the object (S190).

[0146] Then, the object is rendered based on the set level of detail (S191).

[0147] In the fourth embodiment, the relevance calculation unit 406 may calculate the relevance of an object not required for the first task to 0, indicating no relevance, and the relevance of an object required for the second task to 1, indicating relevance. The relevance calculation unit 406 may then calculate the relevance of an object required between the first task and the second task to be a value between 0 and 1.

[0148] Furthermore, the relevance calculation unit 406 may scan the time from the first task to the second task using a predetermined time window, and calculate the relevance at each time scanned by the time window.

[0149] The present invention is not limited to the above-described embodiments, but includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added, deleted, or replaced with other configurations.

[0150] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by a processor interpreting and executing a program that realizes each function.

[0151] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0152] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines that are necessary for implementation. In reality, it can be considered that almost all components are interconnected. [Explanation of symbols]

[0153] 1 processor 2. Memory 3 Auxiliary storage 4. Communication Interface 5 Input Interface 6 Keyboard 7. Mouse 8 Output Interfaces 9 Display Devices 10, 61 3D sensor 20 Edge processing equipment 30 Network 40 MEC Server 50 MR devices 60 VR devices 62 Edge Processing Equipment 70 Administrator terminal 80 Internet 90 Cloud 100 Virtual 3D Space 110 Work Robot 120 Skeleton Acquisition Device 200 On-site sensing function 210 Transmission Processing 220 3D sensing data generation processing 300 Remote sensing function 310 Motion Sensing Processing 400 Metaverse Analysis Function 401, 411, 421, 431 Data acquisition section 402 Self-position estimation part 403 Information display section 404 Operation determination section 405 Work Data Management Department 406 Relevance Calculation Unit 407 Level of Detail Determination 408 Level of Detail Control Unit 410 Object Recognition Processing 412 3D map generation unit 413 Object recognition unit 414 Data Structuring Department 415 3D Data Management Department 420 Motion Recognition Processing 422 Coordinate conversion unit 423 Rendering Department 430 Skilled Sensing Processing 432 Distance calculation part 440 Motion Recognition Processing 441 Data Acquisition Department 442 Self-position estimation part 443 Work Control Unit 444 Work Instruction Department 445 Work Data Management Department 446 Relevance Calculation Unit 447 Level of Detail Determination Unit 448 Level of Detail Control Unit 450 Task Recognition Processing 460 Accumulation Processing 470 databases 500 Feedback Function

Claims

1. A three-dimensional data processing system, a computer having an arithmetic unit that executes predetermined processing and an output unit that outputs the results of the arithmetic unit; an object recognition unit that identifies the area of ​​each object included in the three-dimensional data acquired by the three-dimensional sensor; a data structuring unit that arranges the three-dimensional data into a hierarchical structure; a motion determination unit for identifying a motion of a moving object included in the three-dimensional data; a relevance calculation unit that calculates a relevance between the motion of the moving object and the object; a level of detail control unit that controls a level of detail for displaying the object based on the relevance; The three-dimensional data processing system is characterized in that the output device outputs information in accordance with the controlled level of detail.

2. 2. The three-dimensional data processing system according to claim 1, The relevance calculation unit Calculating a relevance of the object based on the distance between the moving object and the object; A three-dimensional data processing system characterized in that the degree of association is increased when the distance between the objects is short.

3. 3. The three-dimensional data processing system according to claim 2, The three-dimensional data processing system is characterized in that the movement determination unit calculates the distance to the object while the moving object is in motion.

4. 2. The three-dimensional data processing system according to claim 1, A three-dimensional data processing system characterized in that the level of detail control unit outputs the position of at least a portion of the area of ​​the data for displaying the object when the level of detail of that area is insufficient to be controlled.

5. 3. The three-dimensional data processing system according to claim 2, The relevance calculation unit The relevance of the object not required for the first action of the moving object is set to a first value indicating no relevance, A three-dimensional data processing system, characterized in that the relevance of an object necessary for a second action of the moving object is calculated so that the relevance becomes a second value indicating relevance.

6. 6. A three-dimensional data processing system according to claim 5, a relevance calculation unit that calculates the relevance of the moving object required between the first action and the second action of the moving object so that the relevance is a value between the first value and the second value.

7. 7. A three-dimensional data processing system according to claim 6, a three-dimensional data processing system, characterized in that the relevance calculation unit scans the time from the first action to the second action using a predetermined time window and calculates the relevance at each time point scanned by the time window;

8. 2. The three-dimensional data processing system according to claim 1, a distance calculation unit that estimates the skeleton of the worker, which is the moving object, and calculates the distance between the worker's body and the object; The three-dimensional data processing system is characterized in that the relevance calculation unit calculates the relevance of the object using the distance calculated by the distance calculation unit.

9. 9. A three-dimensional data processing system according to claim 8, A three-dimensional data processing system characterized in that the relevance calculation unit calculates the relevance so that the relevance of objects within a predetermined distance from the body position obtained by skeleton estimation is high and the relevance of other objects is low.

10. 2. The three-dimensional data processing system according to claim 1, The three-dimensional data processing system is characterized in that the level of detail control unit outputs a level of detail at which the object is displayed so that the three-dimensional sensor controls the resolution of the output data.

11. A computer-implemented three-dimensional data processing method, comprising: the computer includes an arithmetic unit that executes predetermined arithmetic processing and an output unit that outputs a result of the arithmetic unit; The three-dimensional data processing method includes: an object identification step in which the computing device identifies the region of each object included in the three-dimensional data acquired by the three-dimensional sensor; a data structuring procedure in which the computing device arranges the three-dimensional data into a hierarchical structure; a motion determination step in which the computing device identifies a motion of a moving object included in the three-dimensional data; a relevance calculation step in which the arithmetic device calculates a relevance between the motion of the moving object and the object; a level of detail control step in which the computing device controls a level of detail for displaying the object based on the relevance; and an output procedure in which the output device outputs information in accordance with the controlled level of detail.

Images