Virtual three-dimensional space sharing system, virtual three-dimensional space sharing method, and virtual three-dimensional space sharing server

Abstract

To share movement of a plurality of persons who are at a location distant from a real-time situation of a site.SOLUTION: A virtual three-dimensional space sharing system comprises: a first display device which can be visually recognized by a first user in a first location; a first sensor that observes an object and the first user in the first location; a second sensor that observes movement of a second user in a second location different from the first location; and a server that collects data from the first sensor and the second sensor. The server maps the object and the first user observed by the first sensor and the second user observed by the second sensor in a virtual three-dimensional space, and transmits information about the position and the movement of the second user mapped in the virtual three-dimensional space to the first display device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a virtual three-dimensional space sharing system. [Background technology]

[0002] There are times when multiple people in different locations need to share information. For example, when equipment at a site breaks down, a skilled maintenance technician may travel to that site to provide maintenance instructions. In order for a skilled maintenance technician to travel to a remote site, schedules must be adjusted, which can delay repairs and incur costs. On the other hand, when receiving instructions from a skilled maintenance technician using a remote conference system, there is a problem that it is difficult to provide accurate instructions verbally or through image sharing.

[0003] On the other hand, there is the following prior art as a system for grasping the work situation using a virtual space. In Patent Document 1 (JP Patent Publication No. 2021-47610), when a worker wearing an MR-HMD observes a construction object in a space that is a construction site from various positions and in various directions, the three-dimensional shape of the construction object is measured by a terminal device from the image captured by the MR-HMD. The terminal device receives three-dimensional shape data representing the three-dimensional shape of the construction object, generates an image in which an input field for the inspection results of the construction of the construction object is superimposed on the three-dimensional shape of the construction object seen by the inspector in a virtual space in which the space and coordinate system are common, which is determined based on the three-dimensional shape data and the position and orientation of the VR-HMD worn by the inspector, and displays the image on the VR-HMD. A situation grasping support system is described in which the inspector enters the results of the inspection performed while viewing the three-dimensional shape of the construction object displayed on the VR-HMD into the input field.

[0004] Also, in Patent Document 2 (JP Patent Publication 2006-349578 A), a 3D laser scanner is used to scan the finished form surface, and 3D point cloud data of the finished form surface is synthesized in a virtual space constructed in a computer. Next, information on the center line defined in the workshop is synthesized in the virtual space, and a vertical virtual plane is constructed and moved to set a virtual structural surface. Then, a finished form confirmation system is described that changes the display form of the finished form surface, etc., on the front or back side of the set virtual structural surface and displays it on a screen. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-47610 [Patent Document 2] JP2006-349578A Summary of the Invention [Problem to be solved by the invention]

[0006] The situation understanding support system described in the aforementioned Patent Document 1 and the finished product confirmation system described in Patent Document 2 do not have a mechanism for sharing the real-time situation on site and the actions of multiple people in remote locations in real time, making it difficult to provide appropriate guidance to the site from a remote location.

[0007] The present invention aims to share in real time the situation at a site and the actions of multiple people in remote locations in real time. [Means for solving the problem]

[0008] A representative example of the invention disclosed in the present application is as follows: That is, a virtual three-dimensional space sharing system includes a first display device visible to a first user in a first location, a first sensor observing an object and the first user in the first location, a second sensor observing a movement of a second user in a second location different from the first location, and a server collecting data from the first sensor and the second sensor, wherein the server maps the object and the first user observed by the first sensor and the second user observed by the second sensor in a virtual three-dimensional space, and transmits information on the movement and position of the second user mapped in the virtual three-dimensional space to the first display device. Effect of the Invention

[0009] According to one aspect of the present invention, it is possible to share in real time the real-time situation of an on-site location and the actions of multiple people in remote locations. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a configuration of an information sharing system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing the physical configuration of a computer provided in the information sharing system of the present embodiment. [Diagram 3] 1 is a logical block diagram of an information sharing system according to an embodiment of the present invention. [Figure 4] FIG. 11 is a diagram showing details of on-site sensing processing in this embodiment. [Diagram 5] FIG. 2 is a diagram illustrating an example of a database configuration according to the present embodiment. [Figure 6] 1A and 1B are diagrams illustrating an example of an image displayed on the MR glasses of the present embodiment. [Figure 7] 11 is a diagram illustrating an example of an overhead image displayed on an administrator terminal of the present embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] FIG. 1 is a diagram showing a configuration of an information sharing system according to an embodiment of the present invention.

[0012] The information sharing system of this embodiment includes a plurality of three-dimensional sensors 10, an edge processing device 20 connected to the three-dimensional sensors 10, an MEC server 40 that processes the observation results by the three-dimensional sensors 10, a network 30 that connects the edge processing device 20 to the MEC server 40, MR glasses 50, VR glasses 60, a three-dimensional sensor 61 that observes a wearer of the VR glasses 60, and an edge processing device 62 connected to the three-dimensional sensor 61. The information sharing system may include an administrator terminal 70.

[0013] The three-dimensional sensor 10 is a sensor that observes the situation of the site to be shared in the virtual three-dimensional space (metaverse space) 100. The three-dimensional sensor 10 may be capable of acquiring three-dimensional point cloud data, and may be, for example, a TOF camera that outputs a distance-attached image in which RGB data is assigned a distance D for each pixel. A plurality of three-dimensional sensors 10 are provided to cover a wide range of the site including the work area of ​​the worker, and the observation ranges of the three-dimensional sensors 10 may be installed so as to overlap. The three-dimensional sensor 10 observes static objects, such as equipment installed at the site and structures in a room, whose shapes and positions do not change, and dynamic objects, such as vehicles, construction machinery, robots, workers, tools, and work objects, whose shapes and positions change, as objects. The three-dimensional sensor 10 observes the situation of the worker (for example, the movement and position of a remote worker).

[0014] The edge processing device 20 is a computer that generates three-dimensional information including a plurality of three-dimensional model data and a human skeletal model from the point cloud data acquired by the three-dimensional sensor 10. The edge processing device 20 generates three-dimensional information from the point cloud data, thereby reducing the amount of communication between the edge processing device 20 and the MEC server 40 and preventing congestion on the network 30. Note that if there is no problem with the bandwidth of the network 30, the point cloud data may be transmitted directly to the MEC server 40 and then the three-dimensional information may be generated.

[0015] The MEC server 40 is a computer provided on the network 30 to realize edge computing, and in this embodiment generates a virtual three-dimensional space 100 from three-dimensional information collected from one or more edge processing devices 20.

[0016] The network 30 is a wireless network suitable for data communication that connects the edge processing device 20 and the MEC server 40, and may be, for example, a high-speed, low-latency 5G network. Note that, when the edge processing device 20 is fixedly installed, a wired network may be used.

[0017] The MR glasses 50 are display devices that can be viewed by workers at the site, and may be worn on the head of the worker to share the virtual three-dimensional space 100. The MR glasses 50 have a processor that executes a program, a memory that stores programs and data, a network interface that communicates with the MEC server 40, and a display that displays an image transmitted from the MEC server 40 (described later with reference to FIG. 6). The display may be a transparent type, so that the wearer can view the surroundings through the display superimposed on the image transmitted from the MEC server 40. The MR glasses 50 may also have a camera that captures the front of the wearer, and transmit the image captured by the camera to the MEC server 40. The MR glasses 50 may also display an image captured by a camera that captures the front of the wearer superimposed on the image transmitted from the MEC server 40. The MR glasses 50 may also have a camera that captures the wearer's eyes, and may detect the direction of the wearer's line of sight from the image captured by the camera. The MR glasses 50 may also have a microphone that detects the sound the wearer is hearing.

[0018] Furthermore, a worker at the site may wear a wearable sensor (e.g., a tactile glove). The tactile glove detects the worker's sense of touch and transmits it to the MEC server 40. Furthermore, the wearable sensor may detect the movement of the worker's fingers, and a skeletal model of the worker may be generated from the finger movement detected by the wearable sensor to detect the worker's behavior.

[0019] The VR glasses 60 are display devices that can be viewed by a person (hereinafter, referred to as a remote person, for example, an expert) at a remote location away from the site, and may be worn on the head of a worker to share the virtual three-dimensional space 100. The VR glasses 60 have a processor that executes a program, a memory that stores programs and data, a network interface that communicates with the MEC server 40, and a display that displays an image transmitted from the MEC server 40 (described later with reference to FIG. 6). The VR glasses 60 may also have a camera that captures the front of the wearer, and transmit the image captured by the camera to the MEC server 40. When the VR glasses 60 are provided outside the network in which the MEC server 40 is provided, the VR glasses 60 and the MEC server 40 may be connected via a public network such as the Internet 80 or another dedicated network. The VR glasses 60 receive motion data including the movement and position of the on-site worker represented by a skeletal model from the MEC server 40, and display the virtual three-dimensional space 100 including the avatar of the on-site worker. The information on the virtual three-dimensional space 100 that the VR glasses 60 receive from the MEC server 40 includes information on the object observed by the three-dimensional sensor 10 in addition to the worker's avatar.

[0020] The three-dimensional sensor 61 is a sensor that observes the situation (e.g., the movement and position of the remote person) of the remote person wearing the VR glasses 60 to be shared in the virtual three-dimensional space 100. The three-dimensional sensor 61 may be capable of acquiring three-dimensional point cloud data, similar to the three-dimensional sensor 10, and may use, for example, a TOF camera that outputs a distanced image in which RGB data is assigned a distance D for each pixel. The remote person may wear a wearable sensor that detects the movement of fingers. The wearable sensor detects the movement of the remote person's fingers and transmits it to the MEC server 40. The MEC server 40 may generate a skeletal model of the worker from the movement of the fingers detected by the wearable sensor, and detect the behavior of the worker.

[0021] The edge processing device 62 is a computer that generates three-dimensional information including a plurality of three-dimensional model data (a human skeletal model) from the point cloud data acquired by the three-dimensional sensor 61. The edge processing device 62 generates three-dimensional information from the point cloud data, thereby reducing the amount of communication between the edge processing device 62 and the MEC server 40. Note that, if there is no problem with the amount of communication, the three-dimensional information may be generated after transmitting the point cloud data as is to the MEC server 40.

[0022] The manager terminal 70 is a computer used by an on-site manager who uses the information sharing system, and can display information on the virtual three-dimensional space 100 (for example, an overhead image).

[0023] The information sharing system of this embodiment may include a cloud 90 that forms a large-scale virtual three-dimensional space for sharing three-dimensional information collected from multiple MEC servers 40. The large-scale virtual three-dimensional space formed in the cloud 90 is an integration of the virtual three-dimensional spaces formed by the multiple MEC servers 40, and a large-scale virtual three-dimensional space over a wide range can be formed.

[0024] Access to the MEC server 40 from the MR glasses 50, the VR glasses 60, and the administrator terminal 70 may be authenticated by ID and password or by the unique address (e.g., MAC address) of these devices to ensure security of the information sharing system.

[0025] Fig. 2 is a block diagram showing the physical configuration of a computer provided in the information sharing system of this embodiment. In Fig. 2, the MEC server 40 is shown as an example of a computer, but the edge processing devices 20 and 62 and the administrator terminal 70 may also have the same configuration.

[0026] The MEC server 40 of this embodiment is configured by a computer having a processor (CPU) 1, a memory 2, an auxiliary storage device 3, and a communication interface 4. The MEC server 40 may also have an input interface 5 and an output interface 8.

[0027] The processor 1 is a calculation device that executes programs stored in the memory 2. The processor 1 executes various programs to realize various functional units (e.g., the metaverse analysis function 400, etc.) of the MEC server 40. Note that some of the processes performed by the processor 1 by executing the programs may be executed by other calculation devices (e.g., hardware such as a GPU, ASIC, or FPGA).

[0028] The memory 2 includes a ROM, which is a non-volatile storage element, and a RAM, which is a volatile storage element. The ROM stores unchanging programs (e.g., BIOS) and the like. The RAM is a high-speed, volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the processor 1 and data used when the programs are executed.

[0029] The auxiliary storage device 3 is, for example, a large-capacity, non-volatile storage device such as a magnetic storage device (HDD) or a flash memory (SSD). The auxiliary storage device 3 also stores data used by the processor 1 when executing a program and the program executed by the processor 1. That is, the program is read from the auxiliary storage device 3, loaded into the memory 2, and executed by the processor 1 to realize each function of the MEC server 40.

[0030] The communication interface 4 is a network interface device that controls communication with other devices (for example, the edge processing device 20, the cloud 90) in accordance with a predetermined protocol.

[0031] The input interface 5 is an interface to which input devices such as a keyboard 6 and a mouse 7 are connected and which receives input from an operator. The output interface 8 is an interface to which output devices such as a display device 9 and a printer (not shown) are connected and which outputs the results of program execution in a format that can be viewed by the user. Note that a user terminal connected to the MEC server 40 via a network may provide the input device and the output device. In this case, the MEC server 40 may have a web server function and the user terminal may access the MEC server 40 using a predetermined protocol (e.g., http).

[0032] The programs executed by the processor 1 are provided to the MEC server 40 via a removable medium (such as a CD-ROM or a flash memory) or a network, and are stored in a non-volatile auxiliary storage device 3, which is a non-transitory storage medium. For this reason, it is preferable that the MEC server 40 has an interface for reading data from the removable medium.

[0033] The MEC server 40 is a computer system configured on one physical computer, or on multiple logically or physically configured computers, and may operate on a virtual computer constructed on multiple physical computer resources. For example, each functional unit may operate on a separate physical or logical computer, or multiple functional units may be combined to operate on a single physical or logical computer.

[0034] FIG. 3 is a logical block diagram of the information sharing system of this embodiment.

[0035] The processing by the information sharing system of this embodiment is executed by an on-site sensing function 200, a remote sensing function 300, a metaverse analysis function 400, and a feedback function 500.

[0036] In the on-site sensing function 200, in on-site sensing and transmission processing 210, the three-dimensional sensor 10 observes the situation at the site, and transmits the observed point cloud data to the edge processing device 20. Then, in three-dimensional information generation processing 220, the edge processing device 20 generates three-dimensional information including the point cloud data and three-dimensional model data observed by the three-dimensional sensor 10.

[0037] 4, the edge processing device 20 integrates point cloud data observed by the multiple 3D sensors 10 based on the relationship between the positions and observation directions of the multiple 3D sensors 10 (221). When integrating the point cloud data, an image of the front of the wearer captured by the MR glasses 50 may also be integrated.

[0038] Then, a high-speed 3D modeling process for static objects is performed (222). For example, the outer surface of a static object can be constructed using an algorithm that generates a surface based on the positional relationship of adjacent point clouds. A high-speed 3D modeling process for dynamic objects is also performed (223). For example, a range in which the shape and position change is extracted from the point cloud data, a skeletal model obtained by skeletal estimation is generated, and a person is modeled. The generated skeletal model represents the position of the person (worker), and the time-series changes in the skeletal model represent the person's movement. Modeling of static objects and modeling of dynamic objects may be performed in any order, and either order may be performed first.

[0039] The 3D model is then segmented according to the continuity of the constructed surfaces and the extent of dynamic objects, distinguishing between dynamic and static objects and determining the extent to which objects are meaningful (224).

[0040] In addition, the edge processing device 20 collects the wearer's line of sight and the sounds the wearer is listening to from the MR glasses 50 and transmits them to the MEC server 40. In the MEC server 40, a metaverse analysis function 400 (described later) recognizes static objects and dynamic objects, and a virtual three-dimensional space 100 is generated.

[0041] In the remote side sensing function 300, in the motion sensing process 310, the three-dimensional sensor 61 observes the situation of the remote person and transmits the observed point cloud data to the edge processing device 62. Then, the edge processing device 62 executes a dynamic object high-speed three-dimensional modeling process on the point cloud data observed by the three-dimensional sensor 61 (310). For example, it extracts an area where the shape and position change from the point cloud data, generates a skeletal model obtained by skeletal estimation, and models the person. The generated skeletal model represents the position of the person (worker), and the time series changes in the skeletal model represent the movement of the person.

[0042] After that, the edge processing device 62 generates an avatar from the generated skeletal model (320). The edge processing device 62 also collects the gaze direction of the wearer and the sound heard by the wearer from the VR glasses 60 and transmits them to the MEC server 40. The generated skeletal model is transmitted to the MEC server 40 and treated as the action B of the remote person. The generated avatar is also transmitted to the MEC server 40 together with the sound data heard by the wearer of the VR glasses 60, incorporated into the virtual three-dimensional space 100, and fed back to the MR glasses 50. The generated avatar may be fed back directly to the MR glasses 50. The wearer of the MR glasses 50 can share with the remote person the virtual three-dimensional space 100 incorporating the actions and sensations represented by the movements and positions of the remote person, and can understand the actions of the remote person and can also converse with the remote person.

[0043] In the metaverse analysis function 400, the MEC server 40 generates an avatar of a field worker from a skeletal model of a dynamic object recognized by the field-side sensing function 200, and generates an avatar of a remote person from a skeletal model of the remote person generated by the remote-side sensing function 300. A virtual three-dimensional space 100 is generated by mapping these generated avatars and the three-dimensional model data of static objects recognized by the field-side sensing function 200.

[0044] In the object recognition process 410, the MEC server 40 recognizes the segmented three-dimensional model and identifies the object. For example, the type of the object can be estimated by a machine learning model that has learned images of the object installed at the site, or a model in which the three-dimensional shape of the object installed at the site is recorded.

[0045] In the action recognition process 420, the MEC server 40 recognizes the action A (type of action) of the worker from the motion data including the movement and position of the worker at the site represented by the skeletal model. For example, the action of the worker can be estimated by a machine learning model learned from the motion data based on the past changes in the skeletal model of the worker and the actions of the worker.

[0046] In the skill detection process 430, the MEC server 40 detects the skill level of a worker based on the direction of the worker's line of sight and the sounds the worker hears. For example, the skill level of a worker can be estimated by a machine learning model trained on the direction of the worker's line of sight and the sounds the worker hears while working, and the skill level of the worker. The skill level of a worker may also be estimated by comparing the work time of the worker with the standard work time. For example, if the work time is shorter than the standard work time, the skill level can be determined to be high.

[0047] In the action recognition process 440, the MEC server 40 recognizes the action B (type of action) of the remote person from the change in the skeletal model of the remote person. For example, the action of the remote person can be estimated by a machine learning model learned from the past changes in the skeletal model of the remote person and the actions of the remote person. The action recognition process 420 and the action recognition process 440 may use the same estimation model.

[0048] In the task recognition process 450, the MEC server 40 recognizes the task A of the worker from the object identified in the object recognition process 410 and the task A of the worker recognized in the action recognition process 420. For example, the task A of the worker can be estimated by a machine learning model learned from the object and the action A, or a knowledge graph associating the object and the action. Furthermore, the task A of the worker may be recognized by using the action B of a remote person recognized in the action recognition process 440.

[0049] In the structuring and storing process 460, the MEC server 40 records the task A recognized in the task recognition process 450 in a database 470. In the database 470, the object used to recognize the task A, the action A, motion data due to the change in the skeletal model in the action A, the action B, and the motion data including the movement and position of the worker on-site represented by the skeletal model in the action B are registered as related information. A detailed configuration example of the database 470 will be described with reference to FIG. 5.

[0050] In the feedback function 500, the MEC server 40 searches the database 470 using the recognized action A of the worker as a key, and transmits feedback information acquired from the database 470 to the MR glasses 50. The information fed back to the MR glasses 50 is an avatar generated from motion data of the same work in the same process previously performed, a video of the same work previously performed, and work instructions for the next process of the work. In particular, the avatar and the work video should provide data of the same work performed by a remote person. The information fed back to the MR glasses 50 should be changed according to the skill level estimated by the skill detection process 430 and the attributes of the worker. For example, detailed information should be provided to a less skilled worker, and general information should be provided to a more skilled worker. The feedback function 500 allows a worker wearing the MR glasses 50 to automatically obtain information related to his or her own action A.

[0051] The feedback function 500 may issue commands as feedback to equipment (e.g., robots, construction machinery, vehicles) in addition to the feedback to the MR glasses 50. This allows changes in the virtual three-dimensional space to be reflected in the real world, and various machines can be controlled.

[0052] Fig. 5 is a diagram showing an example of the configuration of the database 470 of this embodiment. Although the database 470 is shown in table format in Fig. 5, it may be configured in other data structures.

[0053] The database 470 includes pre-recorded task-related information 471 and task acquisition information 472 acquired in accordance with the actions of the worker.

[0054] Work-related information 471 stores a work ID, a work reference time, a work manual, work video content, and work text content in association with each other. The work ID is identification information of a work recorded in advance. The work reference time is a standard time for a work performed by a worker. The work manual is an instruction manual for the work performed by the worker, and link information for accessing the instruction manual may be recorded. The work video content is a video of the work performed by a skilled worker or a work previously performed by the worker, and link information for accessing the video may be recorded. The work text content is text information related to the work performed by the worker, and link information for accessing the text information may be recorded.

[0055] The task acquisition information 472 stores an action ID, actual task time, environmental object, worker motion, worker position, worker viewpoint, worker sound field, worker tactile sensation, worker vitals, worker proficiency, task ID, and task log in association with each other. The task ID is identification information of a task, which is a series of actions of a worker. The actual task time is the time required for the task of the worker. The environmental object is an object (e.g., a room, a floor, a device, a tool, a screw) photographed in relation to the task of the worker. The worker motion is the time change in the coordinates of the feature points (fingers, joints of arms, etc., head) of the skeletal model of the worker. The worker position is the positions of the feature points (head, left and right hands, etc.) of the worker and their positional relationship (distance, direction) with respect to the environmental object. The worker viewpoint is the line of sight of the worker and the intersection of the line of sight and the surface of an object present in the line of sight direction. The worker sound field is the sound heard by the worker, and information on a link for accessing sound data may be recorded. The worker's tactile sense is the worker's tactile sense obtained with a tactile glove. The worker's vitals are the worker's voice, facial expression, pulse rate estimated from changes in blood flow, etc., and are used to estimate the worker's emotions and attributes. The worker's proficiency is the worker's proficiency detected by the proficiency detection process 430. The task ID is the task of the worker recognized by the task recognition process 450. The task log is the result of the task execution, and records successful completion, retry, abnormal completion, etc.

[0056] FIG. 6 is a diagram showing an example of an image that is fed back to a field worker in the information sharing system of this embodiment and displayed on the MR glasses 50.

[0057] As shown in Fig. 6, the worker video is displayed so that the remote worker's VR glasses (head position) 601 and hands 602 are superimposed on the real landscape. In Fig. 6, the remote worker's avatar is configured with the VR glasses and hands, but an avatar representing the remote worker's entire body may be generated and displayed. Furthermore, the worker video may display worker attributes 611, a work manual 612, and work instructions 613. Furthermore, the worker video may display an avatar (not shown) generated from a skeletal model of another person present at the work site.

[0058] On-site workers can see the actions of remote workers through avatars via worker video, and can receive appropriate work guidance from experienced workers in remote locations.

[0059] FIG. 7 is a diagram showing an example of an overhead image displayed on the administrator terminal 70 in the information sharing system of this embodiment.

[0060] The overhead image displayed on the manager's terminal 70 shows the VR glasses (head position) 701 and hands 702 of a remotely located expert, the avatar 711 of a field worker, and an environmental object (work target) 721 superimposed on the image of the three-dimensional space.

[0061] The overhead image allows on-site managers to monitor events in the virtual 3D space, see the guidance that workers are receiving from experts, and supervise the work of workers.

[0062] The MEC server 40 may adjust the range of the image so that the feedback image (FIG. 6) or the overhead image (FIG. 7) includes only the image within the necessary range. The overhead image shown in FIG. 7 is an example in which only the work object, the worker's avatar, the remote worker's avatar, and the background around them are displayed, and other background is erased. For example, it is possible to adjust the range of the image to be fed back so that static objects and dynamic objects recognized in the segmentation process (224) do not include objects that are not related to the work, or to apply mosaic processing to the range that does not include objects that are not related to the work, thereby concealing the unrelated objects. When used at a factory or other site, it is possible to prevent the factory space and customer information from leaking to the outside.

[0063] As described above, according to the embodiment of the present invention, the real-time situation at the site and the actions of multiple people in remote locations can be shared in real time, making it possible to provide appropriate guidance to the site from a remote location.

[0064] The present invention is not limited to the above-described embodiments, and includes various modified examples and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. Furthermore, a part of the configuration of one embodiment may be replaced with a configuration of another embodiment. Furthermore, a configuration of another embodiment may be added to a configuration of one embodiment. Furthermore, a part of the configuration of each embodiment may be added, deleted, or replaced with another configuration.

[0065] Furthermore, each of the aforementioned configurations, functions, processing units, processing means, etc. may be realized in hardware, for example by designing some or all of them as an integrated circuit, or may be realized in software by a processor interpreting and executing a program that realizes each function.

[0066] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0067] In addition, the control lines and information lines shown are those considered necessary for the explanation, and do not necessarily show all the control lines and information lines necessary for implementation. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0068] 1 processor 2. Memory 3 Auxiliary storage 4. Communication Interface 5 Input Interface 6 Keyboard 7. Mouse 8 Output Interface 9 Display Devices 10 Three-dimensional sensor 20 Edge processing equipment 30 Network 40 MEC Server 50 MR Glass 60 VR Glasses 61 Three-dimensional sensor 62 Edge Processing Equipment 70 Administrator terminal 80 Internet 90 Cloud 100 Virtual 3D Space 200 On-site sensing function 210 Transmission Processing 220 3D Information Generation Processing 300 Remote sensing function 310 Motion Sensing Processing 400 Metaverse Analysis Function 410 Object Recognition Processing 420 Motion Recognition Processing 430 Expert Sensing Processing 440 Motion Recognition Processing 450 Task Recognition Processing 460 Accumulation Processing 470 Database 471 Work-related information 472 Work Acquisition Information 500 Feedback Function

Claims

1. A virtual three-dimensional space sharing system, a first display device viewable by a first user at a first location; a first sensor for observing an object that is a dynamic object that changes at least one of its shape and position at the first location and the first user; a second sensor that observes a movement of a second user at a second location different from the first location; a server that collects data from the first sensor and the second sensor; The server, Mapping the object and the first user observed by the first sensor and the second user observed by the second sensor into a virtual three-dimensional space; A virtual three-dimensional space sharing system characterized by transmitting information on the movement and position of the second user relative to an object that is a dynamic object mapped in the virtual three-dimensional space to the first display device in real time.

2. The virtual three-dimensional space sharing system according to claim 1, a second display device viewable by the second user at the second location; A virtual three-dimensional space sharing system characterized in that the server transmits information on the object mapped in the virtual three-dimensional space and the movement and position of the first user to the second display device.

3. The virtual three-dimensional space sharing system according to claim 1, a third sensor that detects at least one of a sound, a line of sight, and a touch sense perceived by the first user; The third sensor transmits detected information to the server; A virtual three-dimensional space sharing system, characterized in that the first sensor observes the movement of the first user.

4. The virtual three-dimensional space sharing system according to claim 1, a first edge device to which the first sensor is connected; The first sensor captures an image of the object installed at the first location; A virtual three-dimensional space sharing system characterized in that the first edge device transmits differential data between a frame of the image of the object captured by the first sensor and a previous frame to the server.

5. The virtual three-dimensional space sharing system according to claim 1, a first edge device to which the first sensor is connected; The first sensor acquires information on the movement and position of the first user; A virtual three-dimensional space sharing system characterized in that the first edge device transmits to the server a skeletal model generated from information on the movement and position of the first user acquired by the first sensor.

6. The virtual three-dimensional space sharing system according to claim 1, a fourth sensor that detects at least one of a sound, a line of sight, and a touch sense perceived by the second user; A virtual three-dimensional space sharing system, wherein the fourth sensor transmits detected information to the server.

7. The virtual three-dimensional space sharing system according to claim 1, a second edge device to which the second sensor is connected; The second sensor captures an image of the object installed at the second location; A virtual three-dimensional space sharing system characterized in that the second edge device transmits differential data between a frame of the image of the object captured by the second sensor and a previous frame to the server.

8. The virtual three-dimensional space sharing system according to claim 1, a second edge device to which the second sensor is connected; The second sensor acquires information on the movement and position of the second user; The second edge device A virtual three-dimensional space sharing system, characterized in that a skeletal model generated from information on the movement and position of the second user acquired by the second sensor is transmitted to the server.

9. The virtual three-dimensional space sharing system according to claim 1, A virtual three-dimensional space sharing system, wherein the server records in a database skeletal models generated from images of the first user and the second user.

10. The virtual three-dimensional space sharing system according to claim 9, The server, Recognizing the object from a result of observing the object by the first sensor; Identifying an operation of the first user based on a relationship between a skeleton model generated from the video of the first user and the recognized object; A virtual three-dimensional space sharing system characterized in that the identified task is recorded in the database.

11. The virtual three-dimensional space sharing system according to claim 1, a fifth sensor for detecting at least one of a voice, a blood flow, and a facial expression of the first user; A virtual three-dimensional space sharing system characterized in that at least one of the proficiency and attributes of the first user is estimated from at least one of the voice, blood flow, and facial expression detected by the fifth sensor.

12. The virtual three-dimensional space sharing system according to claim 11, A virtual three-dimensional space sharing system, characterized in that the server changes information to be transmitted to the first display device according to at least one of the estimated skill level and attributes.

13. The virtual three-dimensional space sharing system according to claim 10, The server retrieves information related to the identified task from the database; A virtual three-dimensional space sharing system, characterized in that information acquired from the database is transmitted to the first display device.

14. The virtual three-dimensional space sharing system according to claim 1, A terminal connected to the server, A virtual three-dimensional space sharing system characterized in that the server transmits data of the virtual three-dimensional space in which the object, the first user, and the second user are mapped to the terminal.

15. A computer-implemented virtual three-dimensional space sharing method, comprising: The computer includes: A computing device that executes a predetermined computation process and a storage device that can be accessed by the computing device, a first display device that is viewable by a first user at a first location, a first sensor that is installed at the first location, and a second sensor that is installed at a second location different from the first location; The virtual three-dimensional space sharing method includes: The computing device collects data of the first user and an object that is a dynamic object changing in at least one of its shape and position and that is observed by the first sensor at the first location, and data of the second user that is observed by the second sensor at the second location; the computing device maps the object and the first user observed by the first sensor and the second user observed by the second sensor into a virtual three-dimensional space; A virtual three-dimensional space sharing method characterized in that the computing device transmits to the first display device in real time information on the movement and position of the second user relative to the object, which is a dynamic object, mapped in the virtual three-dimensional space.

16. A virtual three-dimensional space sharing server, A computing device that executes a predetermined computation process and a storage device that can be accessed by the computing device, a first display device that is viewable by a first user at a first location, a first sensor that is installed at the first location, and a second sensor that is installed at a second location different from the first location; collecting data of the first user and an object that is a dynamic object changing in at least one of its shape and position, observed by the first sensor at the first location, and data of the second user that is observed by the second sensor at the second location; Mapping the object and the first user observed by the first sensor and the second user observed by the second sensor into a virtual three-dimensional space; A virtual three-dimensional space sharing server characterized by transmitting information on the movement and position of the second user relative to an object that is a dynamic object mapped in the virtual three-dimensional space to the first display device in real time.

Images