Method and device for providing virtual content in a virtual space based on a common coordinate system

By detecting the basic and target marks in the actual job space and calculating the model matrix in the virtual job space, the problem of inconsistent coordinate systems in remote collaboration is solved, and the unified and accurate information transmission of the virtual collaboration space is achieved, and the collaboration efficiency is improved.

CN113383371BActive Publication Date: 2024-11-08KOREA INST OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202080011789.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-08-14
Filing Date
2020-07-27
Publication Date
2024-11-08
Estimated Expiration
2040-07-27

AI Technical Summary

Technical Problem

In remote collaboration, due to the inconsistent coordinate system and scale of the virtual collaboration space, job participants are unable to accurately reflect and understand each other's job information, which affects collaboration efficiency.

Method used

By detecting the basic markers and target markers in the actual job space, the initial and current model matrix of the target markers in the virtual job space is calculated to determine their position and direction, and update the position and direction of the virtual job object according to these matrices, ensuring that All participants collaborated under a unified coordinate system and scale.

Benefits of technology

Accurate information transmission and collaboration between job participants in the virtual collaboration space is realized, preventing camera movement from being misjudged as a job object movement, and ensuring the unity and accuracy of the virtual job space.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113383371B_ABST
    Figure CN113383371B_ABST
Patent Text Reader

Abstract

The present invention relates to a method for providing virtual content in a virtual space based on a common coordinate system, wherein a basic mark as a mark for identifying a fixed location in the actual work space and a target mark as a mark for identifying the actual work object are detected from initial image data representing an initial state of an actual work object in the actual work space, an initial model matrix of the target mark representing the initial position and orientation of the target mark in the virtual work space having a common coordinate system based on the basic mark and a model matrix of the basic mark representing the position and orientation of the basic mark are calculated from the initial image data, a virtual work object having a three-dimensional model shape of the actual work object is generated, and a current model matrix of the target mark representing the current position and orientation of the target mark is calculated using the calculated model matrix of the basic mark, thereby being able to provide a virtual work space based on the common coordinate system to on-site workers and remote workers.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a method for providing virtual content in a virtual space, and more particularly to a method for providing virtual content in a virtual space based on a common coordinate system with a base marker as a reference. Background Art

[0002] Virtual space refers to a three-dimensional virtual space that reflects part or all of the actual space. Virtual space includes non-real virtual objects and spaces generated by computers. This virtual space can provide users with corresponding space through augmented reality (AR) technology, virtual reality (VR) technology or mixed reality (MixedReality) technology. This virtual space is currently used in many fields such as games, aviation, and military. A representative example of the application of virtual space is remote collaboration.

[0003] Remote collaboration refers to multiple workers who are physically separated from each other communicating and working together to solve the same problem. In other words, remote collaboration means that multiple work participants share information and perform work at the same time in different spaces instead of gathering in the same space. This type of remote collaboration can remove the spatial restriction that all work participants need to gather in the same space.

[0004] In this remote collaboration, the work participants are connected to each other through the network and share the same screen. Examples of remote collaboration include video conferencing or sharing the same screen so that participants in remote work can work on the same task together. In the past, remote collaboration was a way for work participants to communicate and collaborate through voice and chat. Recent remote collaboration uses augmented reality (AR, Augmented Reality), virtual reality (VR, Virtual Reality) or mixed reality (MixedReality) to provide more information to provide a shared collaboration space between remote collaborative work participants. The collaboration space that applies augmented reality, virtual reality or mixed reality is a virtual space that reflects part or all of the space where the real collaboration objects are actually located. The virtual collaboration space is provided to the work participants, and the work participants can work together in the virtual collaboration space.

[0005] In this remote collaboration service, when one of the multiple work participants performs any task in the virtual collaboration space, the work done by the task participant will be reflected in the virtual collaboration space and transmitted to all work participants. Among them, when the coordinate system and scale in the virtual collaboration space are not unified among multiple work participants, there is a problem that the work performed by a task participant cannot be correctly reflected in the virtual collaboration space. In the virtual collaboration space, in the example where the virtual collaboration object is moved by any one of the work participants, if the coordinate scales between the work participants are different, then no matter which work participant moves the collaboration object, the other work participants will not be able to know the correct moving distance of the collaboration object. In addition, in the above example, if the coordinate system of the virtual collaboration space is inconsistent, then even if the collaboration object is moved by any one of the work participants, the other work participants will not be able to know the moving direction of the collaboration object.

[0006] As described above, since the coordinate system and scale are not unified in the virtual collaborative space of the remote collaborative system, accurate information about the work of the work participant cannot be provided to other work participants, thus hindering smooth collaboration among multiple work participants in the virtual collaborative space.

[0007] In existing remote collaboration systems, in order to solve this problem, when the coordinate systems and scales between the work participants are inconsistent, each work participant adjusts the coordinate system and scale of the virtual collaboration space in his or her own device to make the coordinate system and scale between the work participants consistent. However, in these existing methods, there is a problem that since each of the multiple work participants manually adjusts the coordinate system and scale of the virtual collaboration space in its device, the coordinate systems and scales of all the work participants cannot be perfectly matched. Summary of the invention

[0008] Technical issues

[0009] The present invention aims to provide a method for providing virtual content in a virtual space based on a common coordinate system. Furthermore, the present invention is not limited to the above-mentioned technical problem, and another technical problem can be derived from the following description.

[0010] Technical Solution

[0011] According to one aspect of the present invention, a method for providing virtual content in a virtual space based on a common coordinate system includes: a step of detecting a base mark as a mark for identifying a fixed location of the actual work space and a target mark as a mark for identifying the actual work object from initial image data representing an initial state of the actual work object in the actual work space; a step of calculating an initial model matrix of the target mark representing the initial position and orientation of the target mark in a virtual work space having a common coordinate system based on the detected base mark and a model matrix of the base mark representing the position and orientation of the detected base mark; and a step of calculating a current model matrix of the target mark representing the current position and orientation of the target mark using the calculated model matrix of the base mark.

[0012] The step of detecting the target marker includes extracting a shape included in the initial image data, and detecting the base marker and the target marker using the extracted shape.

[0013] The step of detecting the target marker comprises comparing the extracted shape with the shape of the base marker and the shape of the target marker, detecting a shape matching the shape of the base marker as the base marker, and detecting a shape matching the shape of the target marker as the target marker.

[0014] The step of calculating the model matrix of the basic mark includes: the step of calculating, based on the initial image data, an initial model viewing angle matrix of the basic mark that represents the position and orientation of the basic mark based on the viewing angle of a camera module used to shoot the initial image data, and an initial model viewing angle matrix of the target mark that represents the position and orientation of the target mark based on the viewing angle of the camera module; the step of calculating the initial model matrix of the target mark on the virtual workspace; and the step of calculating the model matrix of the basic mark using the calculated initial model viewing angle matrix of the basic mark, the calculated initial model matrix of the target mark, and the calculated initial model viewing angle matrix of the target mark.

[0015] The step of calculating the model matrix of the base marker using the calculated initial model perspective matrix of the base marker, the calculated initial model matrix of the target marker, and the calculated initial model perspective matrix of the target marker calculates the model matrix of the base marker according to the following mathematical formula.

[0016] <Mathematical formula>

[0017] M b =M o *(MV o )-1*MV b

[0018] (M b : The model matrix of the base label, MV b : Initial model view matrix of base marker, M o : The initial model matrix of the target label, MV o : The initial model view matrix of the target marker)

[0019] The step of calculating the current model matrix of the target marker includes: the step of calculating the current model viewing matrix of the base marker and the current model viewing matrix of the target marker based on the current image data; and the step of calculating the current model matrix of the target marker using the calculated current model viewing matrix of the base marker, the calculated current model viewing matrix of the target marker and the calculated model matrix of the base marker.

[0020] Using the calculated current model perspective matrix of the base tag, the calculated current model perspective matrix of the target tag and the calculated model matrix of the base tag, the step of calculating the current model matrix of the target tag calculates the current model matrix of the target tag according to the following mathematical formula.

[0021] <Mathematical formula>

[0022] M o `=M b *(MV b `)-1*MV o `

[0023] (M o `: Current model matrix of target label, MV o `: The current model view matrix of the target marker, MV b `: The current model view matrix of the base marker, M b : The model matrix of the base label)

[0024] According to one aspect of the present invention, a method for providing virtual content in a virtual space based on a common coordinate system also includes: a step of determining whether an instruction for changing the position and direction of a virtual work object is received from a remote device of the remote operator; and when an instruction for changing the position and direction of the virtual work object is received, a step of reflecting the change in the position and direction of the virtual work object on the virtual work space according to the received instruction.

[0025] According to one aspect of the present invention, the method for providing virtual content in a virtual space based on a common coordinate system further includes: updating the position and direction of the virtual work object in the virtual work space according to the calculated current model matrix of the target mark.

[0026] The device of another aspect of the present invention includes a mark detection unit that detects a basic mark as a mark for identifying a fixed location in the actual working space and a target mark as a mark for identifying the actual operating object from initial image data representing an initial state of the actual operating object in the actual working space; a matrix calculation unit that calculates an initial model matrix of the target mark representing the initial position and orientation of the target mark and a model matrix of the base mark representing the position and orientation of the detected base mark in a virtual working space having a common coordinate system based on the detected base mark, and calculates a current model matrix of the target mark representing the current position and orientation of the target mark using the calculated model matrix of the base mark.

[0027] Technical Effects

[0028] The on-site equipment of the on-site operator designates a fixed location in the actual work space as a basic mark, and determines the position and direction of the virtual work object corresponding to the actual work object in the virtual work space based on the fixed basic mark. That is, the virtual content providing method of the present invention detects changes in the actual work object based on the basic mark attached to a fixed location in the actual work space, and reflects it (mirroring) to the virtual work space. The changes in the position and direction of the actual work object are calculated based on the fixed basic mark, and reflected to the virtual work object in the virtual work space, so that the user device can generate a virtual work space with a certain coordinate system and scale. In other words, a virtual work space with a common coordinate system can be generated for remote operators. As a result, correct information about changes in the position and direction of the actual work object can be provided to remote operators located in different spaces.

[0029] The virtual content providing method of the embodiment of the present invention can distinguish the movement of the actual work object and the movement of the camera module by grasping the movement of the actual work object and the movement of the camera module based on the fixed basic mark. The user device calculates the model matrix of the target mark and the viewing angle matrix of the camera module based on the basic mark, so that the movement of the actual work object and the movement of the camera module can be distinguished in the virtual work space. Therefore, the user device of the embodiment of the present invention can prevent the problem of mistakenly reflecting the camera movement as the movement of the actual work object on the virtual collaboration space.

[0030] As described above, the virtual content providing method of the embodiment of the present invention generates a virtual workspace with a common coordinate system and scale, and provides the generated virtual workspace to multiple work participants, so that the work participants can perform virtual work smoothly and correctly. Furthermore, since the coordinate system of the virtual workspace based on the common coordinate system is consistent, it can be displayed in multiple extended realities such as virtual reality, augmented reality, or mixed reality. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a diagram of a virtual operation system according to an embodiment of the present invention;

[0032] Figure 2 yes Figure 1 A diagram showing the composition of the field equipment;

[0033] Figure 3 is based on Figure 1 A flowchart of a method for providing virtual content in a virtual space of a common coordinate system between an on-site worker and a remote worker is shown;

[0034] Figure 4 It is calculated Figure 3 A detailed flow chart of the steps of the initial model matrix of the base markers and target markers shown;

[0035] Figure 5 is a schematic diagram illustrating a method for calculating an initial model matrix of a base marker and an initial model matrix of a target marker;

[0036] Figure 6 is a schematic diagram showing an example screen before and after a virtual work object is moved, which is displayed on a display unit of a field device;

[0037] Figure 7 It is calculated Figure 3 A detailed flowchart of the steps of the current model matrix of the target label shown;

[0038] Figure 8 is a schematic diagram illustrating a method for calculating a current model matrix of a target marker;

[0039] Fig. 9 is a schematic diagram showing example screens displayed on the display units of the on-site device and the remote device before and after the actual work object is moved;

[0040] Fig.10 1 is a schematic diagram showing an example screen in which the virtual work space of the present embodiment is displayed on the display units of the on-site device and the remote device. DETAILED DESCRIPTION

[0041] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. The embodiments of the present invention to be described below relate to a method for providing virtual content in a virtual space based on a common coordinate system. More specifically, it relates to a method for providing virtual content between an on-site operator and a remote operator using a virtual space based on a common coordinate system. A person of ordinary skill in the technical field to which the embodiments of the present invention belong can understand that the "Mixed Reality (MR)" to be described below refers to a technology that combines reality and virtuality to create a new environment in which real physical objects and virtual objects coexist, and users can interact in real time in the environment.

[0042] According to the following embodiments, the devices of the work participants who are located at remote locations are connected to each other through a communication network, and the work participants share the work progress status through the AR (Augmented Reality) content or VR (Virtual Reality) content described below. Hereinafter, the virtual object corresponding to the actual work object will be referred to as the "virtual work object", and the virtual space corresponding to the actual work space will be referred to as the "virtual work space". At least one participant who is located in the actual work space and works on the actual work object will be referred to as an "on-site operator", and at least one participant who is located at a remote location of the actual work space and participates in the work on the actual work object will be referred to as a "remote operator". In addition, the coordinate system of the virtual space that can be expressed in augmented reality or virtual reality and is shared by the equipment of the on-site operator and the equipment of the remote operator is called a "common coordinate system". Here, the virtual space can be provided to the user in virtual reality or augmented reality according to the equipment.

[0043] Figure 1 FIG. 1 is a diagram showing the structure of a virtual operation system 1 according to an embodiment of the present invention. Figure 1 The virtual work system 1 of this embodiment is composed of the device 11 of the on-site worker 10 and the device 21 of the remote worker 20. The device 11 of the on-site worker 10 and the device 21 of the remote worker 20 can be the same device. In order to distinguish the device 11 of the on-site worker 10 from the device 11 of the remote worker 20, the device 11 of the on-site worker 10 is referred to as "on-site device 11", and the device 11 of the remote worker 20 is referred to as "remote device 21". The on-site device 11 and the remote device 21 are connected via a network to communicate with each other. Multiple work participants 10, 20 can perform work while sharing the work status in a mixed reality environment by displaying the contents of their respective devices 11, 21.

[0044] The on-site operator 10 in the actual work space can operate the actual work object through joint work with the remote operator 20, such as joint work such as repair work on the faulty part or component replacement work, or work such as games in the virtual space. More specifically, the on-site device 11 uses its camera module 112 to shoot the actual work space, that is, shoot the actual work object at the work site, generate a virtual work object corresponding to the actual work object shot together with it in the virtual work space, and transmit data about the virtual work object in the virtual work space to the remote device 21 of the remote operator 20 at a remote location.

[0045] The remote operator 20 can operate the virtual operation object using various input mechanisms while observing the virtual operation object in the virtual operation space displayed on the screen of the remote device 21. By reflecting the operation of the virtual operation object by the remote operator 20 in the virtual operation space, the information of the operation of the virtual operation object is transmitted to all the on-site operators 10 and the remote operator 20 who use the virtual operation application. Based on the changes of the virtual operation object in the virtual operation space, the on-site operator 10 can timely know the operation performed by the remote operator 20 in the virtual operation space. The on-site operator 10 can operate the actual operation object at the operation site with reference to the operation in the virtual operation space. As described above, the virtual operation technology is to share the virtual operation object representing the actual operation object between the remote operator 20 who is an expert far away from the operation object and the on-site operator 10 located at the operation site, so that the on-site operator 10 can operate the actual operation object after observing the operation performed by the remote operator 20 in the virtual operation space.

[0046] As with the on-site operator 10, the remote operator 20 can grasp the movement of the actual operation object operated by the on-site operator 10 while observing the virtual operation object in the virtual operation space displayed on the screen of the remote device 21. By reflecting the operation performed by the on-site operator 10 on the actual operation object in the virtual operation space, the information on the operation of the actual operation object is transmitted to all on-site operators 10 and remote operators 20 using the virtual operation application. The remote operator 20 can timely know the operation performed by the on-site operator 10 in the actual operation space based on the change of the virtual operation object in the virtual operation space. The remote operator 20 can operate the virtual operation object in the virtual operation space with reference to the operation in the actual operation space. As described above, the virtual operation technology is to share the virtual operation object that displays the actual operation object between the remote operator 20 who is an expert far away from the operation object and the on-site operator 10 located at the operation site, so that the remote operator 20 can observe the operation performed by the on-site operator 10 in the actual operation space in the virtual operation space and operate the virtual operation object.

[0047] The field device 11 is a device capable of realizing virtual work between multiple work participants 10 and 20 located in places far away from each other. The field device 11 executes a virtual work application for performing a preset remote collaboration service, and provides remote collaboration services to users through the virtual work application. The virtual work application is a field device 11 connected to multiple work participants through a network, which enables multiple work participants located in places far away from each other to perform virtual work. In the following, it is assumed that the virtual work application is pre-set in the field device 11 before performing the virtual work. A representative example of the field device 11 can be a tablet PC, and HMD (Head Mounted Display), VR (virtual reality) equipment, smart phone, laptop, etc. can also be exemplified. In addition, the field device 11 can be connected to other field devices 11 by accessing wireless communication networks such as WCDMA and LTE. As described above, the remote device 21 can be the same device as the field device 11.

[0048] Figure 2 yes Figure 1 The structure diagram of the field device 11 is shown in FIG. Figure 2 The field device 11 is composed of a processor 111, a camera module 112, a marker detection unit 113, a virtual space processor 114, a matrix calculation unit 115, a display unit 116, an input module 117, a communication module 118, and a memory 119. The processor 111 processes general tasks of the field device 11. The camera module 112 captures the actual work space where the actual work object is located, and generates image data representing the actual work object and the actual work space. The marker detection unit 113 analyzes the image data generated from the camera module 112, and detects a base marker indicating a location in the actual work space where the actual work object is located that serves as a reference point of the virtual work space, and a target marker indicating the actual work object in the virtual work space. The virtual space processing unit 114 generates a virtual work space as a three-dimensional virtual space that reflects a part or the whole of the actual work space based on the image data representing the actual work space. Furthermore, the virtual space processing unit 114 generates a virtual operation object having the same three-dimensional model shape as the actual operation object for the virtual operation of the on-site operator 10 and the remote operator 20, and arranges the generated virtual operation object in the virtual operation space. The virtual space processing unit 114 detects changes in the actual operation object or the virtual operation object, and reflects the detected changes in the actual operation object or the virtual operation object in the virtual operation space.

[0049] The matrix calculation unit 115 calculates a matrix that represents the state of any object in the virtual work space according to the reference point, that is, the position and orientation. The matrix calculation unit 115 calculates the model matrix, the view matrix and the model view matrix. Here, the matrix is ​​a mathematical tool used to distinguish the position and direction in the virtual space based on any origin of the virtual object, and calculate the position and direction of the independent virtual object. The model matrix is ​​a matrix that represents the state (position and direction) of the virtual object in the virtual space. For example, the model matrix of the virtual work object is a matrix that represents the position and direction of the virtual work object based on the origin on the virtual work space. The model matrix, the view matrix and the model view matrix will be described in detail below. Here, the mark detection unit 113, the virtual space processing unit 114 and the matrix calculation unit 115 can also be implemented by another dedicated processor different from the processor 111, or by executing a computer program of the processor 111.

[0050] The display unit 116 displays a virtual work space including virtual operation objects corresponding to actual operation objects. The input module 117 receives input signals from the user to operate virtual operation objects in the virtual work space. The input module 117 can be, for example, a keyboard, a mouse, a touch screen, a motion remote controller, etc. The communication module 118 supports a communication function, so that it is connected to an LTE base station or a WIFI repeater and can communicate with other remote devices 21 through a wide area network such as the Internet. The memory 119 stores applications for virtual operations and data for these applications.

[0051] The field device 11 includes additional components in addition to the components described above. Figure 2 As shown, it includes a bus for transmitting data between multiple components, Figure 2 Although omitted from the illustration, it also includes a power module that supplies driving power to each component. As such, the description of components known to those skilled in the art of the present embodiment obscures the features of the present embodiment, and thus the description is omitted. In the following, in the process of describing a method for providing virtual content in a virtual space based on a common coordinate system according to an embodiment of the present invention, each component of the field device 11 is described in detail.

[0052] Figure 3 yes Figure 1 The flowchart of the method for providing virtual content between the on-site operator and the remote operator in the virtual space based on the common coordinate system is shown. Figure 3 The virtual content providing method of this embodiment is provided by Figure 1 The illustrated field device 11 is structured in steps that are executed in chronological order.

[0053] In step 301, the processor 111 of the field device 11 executes the virtual work application. More specifically, when the processor 111 receives an execution instruction of the virtual work application from the field operator 10 through the input module 117, the processor 111 executes the virtual work application. Figure 1 When the on-site device 11 of the on-site worker 10 receives an execution instruction of the virtual work application from the on-site worker 10, the virtual work application is executed. The remote device 21 of the remote worker 20 also executes the virtual work application when receiving an execution instruction of the virtual work application from the remote worker 20. The on-site device 11 of the on-site worker 10 executing the virtual work application is connected to the remote device 21 of the remote worker 20 via a network.

[0054] In step 302, the camera module 112 of the field device 11 generates initial image data representing the initial state of the actual operation object in the actual operation space. The image data is an electronic signal representing the characteristics of a two-dimensional or three-dimensional shape, and is data generated by an input device such as the camera module 112. In order to realize a virtual operation with at least one remote operator 20 that is far away, the remote operator 20 uses the camera module 112 of the field device 11 to shoot the actual operation object that is the object of the virtual operation. The camera module 112 generates initial image data representing the initial state of the actual operation object in the actual operation space. The initial image data represents the initial state of the actual operation object including the position and direction of the actual operation object in the actual operation space before the virtual operation is performed. Since the initial image data represents the state of the actual operation object before the virtual operation is performed, it becomes a reference for determining whether the state of the actual operation object changes during the execution of the virtual operation. The camera module 112 inputs the generated initial image data into the mark detection unit 113 of the field device 11.

[0055] In step 303, the mark detection unit 113 of the field device 11 detects a basic mark and a target mark from the initial image data input by the camera module 112. More specifically, the mark detection unit 113 detects a basic mark for identifying a fixed location in the actual work space and a target mark for identifying the actual work object from the initial image data representing the initial state of the actual work object in the actual work space. The mark detection unit 113 detects a basic mark as a reference point in the virtual work space coordinate system from the initial image data representing the initial state of the actual work object in the actual work space, and detects a target mark representing the actual work object. As described above, the basic mark is a mark that serves as a reference point in the virtual work space and is a mark for identifying a fixed location in the actual work space. Since the basic mark serves as a reference point, the mark detection unit 113 detects a fixed location or object as a basic mark. The target mark is a mark for identifying an object that is an actual work object of the virtual work. The basic mark and the target mark can be preset by the field operator 10. The detection of the basic mark and the target mark will be described in detail below.

[0056] The marker detection unit 113 of one embodiment of the present invention extracts the shape contained in the image data through image processing, and uses the extracted shape to detect the basic marker and the target marker. More specifically, the marker detection unit 113 extracts the shape contained in the initial image data representing the initial state of the actual work object in the actual work space. For example, the marker detection unit 113 extracts feature points (feature point, keypoint) from the initial image data. Feature points are locations in the image that are distinguished from the surrounding background and are easily recognizable. Feature points are locations in the image data that can be easily recognized even if the shape, size or position of the object changes, and locations that can be easily recognized even if the camera's perspective and lighting change. Feature points can be, for example, corners, edges, etc. The marker detection unit 113 analyzes the extracted feature points to extract the shape contained in the image data. The marker detection unit 113 can use an artificial neural network (Artificial Neural Network) technique to extract the shape contained in the image data. The method of extracting feature points and shapes from image data is well known to a person skilled in the art of the present embodiment, so a detailed description thereof will not be repeated to prevent confusion about the features of the present embodiment.

[0057] The marker detection unit 113 extracts a basic marker and a target marker from the shapes extracted from the initial image data. More specifically, the marker detection unit 113 compares the extracted shape with the basic marker shape and the target marker shape pre-stored in the memory 119. Based on the comparison result, the marker detection unit 113 determines a shape that is consistent with the shape of the basic marker among the shapes extracted from the image data as a basic marker, and determines a shape that is consistent with the shape of the target marker as a target marker. The marker detection unit 113 extracts a shape that is identical to the shape of the basic marker pre-stored in the memory 119 of the field device 11 as a basic marker, and extracts a shape that is identical to the shape of the target marker pre-stored in the memory 119 as a target marker.

[0058] The shape of the basic mark and the shape of the target mark are pre-stored in the memory 119 of the field device 11. As described above, the shape of the basic mark is the shape of a place or object that can serve as a reference point in the virtual work space. Since the basic mark serves as a reference point in the virtual work space, the place or object that can be designated as the basic mark should not be moved and needs to be fixed. For example, the ground or desk of the space where the actual work object is located, the surrounding wallpaper pattern, the QR code (Quick Response Code) attached to the fixed object in the actual work space, the bar code, the specific pattern, etc. can be used as the basic mark. In the case of storing a QR code with a specific pattern as the shape of the basic mark, the field operator 10 can predetermine the basic mark in the actual work space by attaching the QR code with such a specific pattern to a fixed object in the actual work space. The memory 119 of the field device 11 stores image data representing the shape of the basic mark as the shape of a fixed place or object in the actual work space. The marker detection unit 113 compares the shape of the basic marker stored in the memory 119 with the shape extracted from the image data input by the camera module 112 (i.e., image data representing the state of the actual work object in the actual work space), and determines the shape that is consistent with the shape of the pre-stored basic marker as the basic marker.

[0059] Furthermore, as described above, the shape of the target mark is the shape of the actual work object that the on-site operator 10 and the remote operator 20 will work together in the virtual work space. The on-site operator 10 determines the actual work object that will perform virtual work with the remote operator 20 in the actual work space. In one embodiment of the present invention, the on-site operator 10 uses the camera module 112 of the on-site device 11 to generate image data representing the shape of the determined actual work object. The on-site device 11 pre-stores image data representing the shape of the target mark as the shape of the actual work object in the memory. The mark detection unit 113 compares the shape of the actual work object stored in the memory 119 with the shape extracted from the image data representing the state of the actual work object in the actual work space, and determines the shape that is consistent with the shape of the pre-stored target mark as the target mark (that is, the work object in the virtual work space). For example, the actual work object of the virtual work is a powder cylinder of a 3D printer, and the memory 119 of the on-site device 11 pre-stores the shape of the powder cylinder as the actual work object. The mark detection unit 113 of the on-site device 11 compares the shape extracted from the image data representing the actual work object in the actual work space with the shape of the powder cylinder pre-stored in the memory 119. As a result of the comparison, a shape that matches the shape of the powder cartridge stored in advance as the shape of the target mark in the memory 119 is identified as the target mark. The mark detection unit 113 extracts the identified shape as the target mark.

[0060] In another embodiment of the present invention, the memory 119 of the field device 11 is capable of storing any code such as a QR code or a bar code as the shape of a target mark. The field collaborator 10 can attach a QR code or a bar code stored as the shape of a target mark to the actual work object of the virtual work to be performed to specify the actual work object. Taking the case where the actual work object of the virtual collaboration is a powder barrel as an example, the field operator 10 attaches the QR code stored in the memory 119 in the shape of a target mark to the powder barrel as the actual work object. The mark detection unit 113 compares the shape of the QR code or bar code stored in the memory 119 with the shape extracted from the image data representing the actual work object in the actual work space, and can determine the shape consistent with the shape of the pre-stored QR code or bar code as the target mark (i.e., the work object).

[0061] In another embodiment of the present invention, the target mark or base mark includes an electronic signal that cannot be recognized by the naked eye. For example, the field device 11 can identify the target mark or base mark by a signal sent by a small wireless communication device. For example, a wireless communication device that sends a beacon signal including information about the shape and position of the base mark is attached to a fixed location in the actual working space, and the wireless communication device attached to the fixed location in the actual working space sends a signal for the base mark to the field device 11. The field device 11 can identify the shape and position of the base mark based on the received signal for the base mark. In addition, the target mark is also attached to the actual working object by sending a wireless communication device including a beacon signal including information about the shape and position of the actual working object, and the wireless communication device attached to the actual working object sends a signal about the target mark to the field device 11. The field device 11 can identify the shape and position of the target mark based on the received signal about the target mark. In addition to the above examples, the field device 11 can also identify the target mark and the base mark by a variety of methods.

[0062] In step 304, the virtual space processing unit 114 of the field device 11 generates a virtual work space for realizing a virtual operation with the remote operator 20 and a virtual work object corresponding to the actual work object. The virtual space processing unit 114 generates a virtual work space including a virtual work object corresponding to the actual work object of the virtual operation. The virtual work space is a three-dimensional virtual space that reflects (mirroring) a part or the whole of the actual work space based on image data representing the state of the actual work object in the actual work space. For example, the virtual work space as a three-dimensional space may have an origin and X, Y, and Z axes. The virtual space processing unit 114 generates a virtual work object having a three-dimensional model shape of the actual work object. For example, when the actual work object is a cylindrical powder cylinder, the virtual space processing unit 114 generates a virtual work object having a cylindrical shape. In addition, the virtual space processing unit 114 configures a virtual work object as a three-dimensional model shape of the actual work object at an arbitrary position in the three-dimensional virtual work space. Among them, the arbitrary position of the virtual work object configured as a three-dimensional model shape of the actual work object can be preset by the designer. As will be described below, an embodiment of the present invention can generate a virtual work space consisting of a coordinate system having a common reference with the actual work space by using a base marker attached to a fixed location in the actual work space to determine the state (position and direction) of the target marker in the virtual work space.

[0063] In step 305, the matrix calculation unit 115 of the field device 11 calculates the model matrix of the base mark sensed in step 303 and the initial model matrix of the target mark from the initial image data generated in step 302. The model matrix represents the position and direction of any place or object configured in the virtual work space. For example, the model matrix of the base mark represents the position and direction of the base mark in the virtual work space, and the initial model matrix of the target mark is the model matrix of the target mark calculated first after the target mark is sensed, representing the initial state (position or direction) of the virtual work object of the virtual work in the virtual work space. Therefore, the initial model matrix of the target mark becomes the basis for determining the change of the target mark in the virtual work space. When the state (position and direction) of the actual work object changes, the state of the target mark changes, so the initial model matrix representing the target mark and the current model matrix of the current state of the target mark differ.

[0064] More specifically, when the position or direction of the actual work object changes, the field device 11 calculates the model matrix of the target mark using the image data representing the state of the changed actual work object, and compares it with the initial model matrix of the target mark, thereby being able to know the change in the position and / or direction of the target mark, i.e., the actual work object, in the virtual work space. The initial model matrix of the target mark is a reference for determining whether a change such as movement of the actual work object occurs in the virtual work.

[0065] When the model matrix of the base mark and the initial model matrix of the target mark are calculated based on the camera module 112 of the on-site device 11, even if the actual work object does not change, the position or direction of the base mark and the target mark in the virtual work space also changes with the change of the position of the on-site device 11, which may lead to the problem of transmitting erroneous information about the actual work object to the remote operator 20.

[0066] More specifically, in order to realize the virtual operation of the on-site operator 10 and the remote operator 20, the on-site device 11 of the on-site operator 10 generates a virtual operation space including a virtual operation object corresponding to the actual operation object. In order to display the virtual operation space including the virtual operation object, the on-site device 11 needs to understand the position and direction of the virtual operation object corresponding to the actual operation object configured in the virtual operation space. The virtual operation object is configured in a predetermined position and direction in the virtual operation space. The position and direction of the virtual operation object in the virtual operation space can be represented by a model matrix. The model matrix of the virtual operation object represents the position and direction of the virtual operation object in the virtual operation space.

[0067] Furthermore, in order to configure the virtual work object corresponding to the actual work object in the virtual work space in the same state as the actual work object, the field device 11 needs to know the direction in which the camera module 112 of the field device 11 that photographs the actual work object in the actual work space observes the actual work object. The viewing angle matrix of the camera module 112 represents the direction in which the camera module 112 observes the virtual work object corresponding to the actual work object in the virtual work space.

[0068] In the actual work space, the object observed is different depending on the viewing direction when the user observes the object as the actual work object. In the virtual work space, the virtual work object is observed differently depending on the direction in which the camera module 112 observes the virtual work object. The model view (ModelView) matrix characterizes the state (position and direction) of the virtual operation object in the virtual work space with the camera module 112 of the field device 11 as a reference. In other words, the model view matrix of the virtual work object is a matrix that characterizes the position and direction of the virtual work object with the view of the camera module 112 used when capturing image data as a reference. The model view matrix of the virtual work object can be represented by the product of the model matrix that characterizes the position and direction of the virtual work object in the virtual work space and the view matrix that characterizes the direction in which the camera module 112 observes the operation object. The model view matrix, the model matrix, and the view matrix can be represented as the following mathematical formulas.

[0069]

Mathematical formula 1

[0070] Model View matrix=View matrix*Model matrix

[0071] As described above, the model viewing angle matrix of the virtual work object represents the position and direction of the virtual work object in the virtual work space based on the viewing direction of the camera module 112. For such a model viewing angle matrix of the virtual work object, the matrix calculation unit 115 of the field device 11 can calculate the model viewing angle matrix of the virtual work object from the image data representing the actual work object in the actual work space. Since the model viewing angle matrix of the virtual work object calculated in this way is based on the camera module 112, it represents the relative position and direction of the virtual work object with respect to the position and shooting direction of the camera module 112.

[0072] The model view matrix of the virtual work object can be calculated in the above manner, but the model matrix of the virtual work object representing the absolute position and direction of the virtual work object in the virtual work space and the view matrix of the camera module 112 representing the direction in which the camera module 112 observes the virtual work object in the virtual work space cannot be known. As described above, since the model view matrix of the virtual work object is the relative position and direction of the virtual work object based on the position and direction of the camera module 112, it is impossible to know whether the camera module 112 and the actual work object move due to the movement of the camera module 112 or the movement of the actual work object. Due to this uncertainty, the remote operator 20 cannot know the correct information about the actual work object in the virtual work.

[0073] In order to solve this problem, an embodiment of the present invention calculates a model matrix of a base label and an initial model matrix of a target label as follows. Figure 4 It is calculated Figure 3 The specific flow chart of the steps of the model matrix of the base marker and the initial model matrix of the target marker is shown, Figure 5 is a schematic diagram illustrating the calculation method of the model matrix of the base marker and the initial model matrix of the target marker. Figure 4 and Figure 5 In step 3051, the virtual space processing unit 114 specifies an origin in the virtual workspace generated in step 304. The virtual space processing unit 114 specifies an arbitrary point as the origin in the generated virtual workspace.

[0074] In step 3052, the matrix calculation unit 115 calculates an initial model view matrix (MV b matrix) and the initial model view matrix (MV o As described above, the model view matrix represents the position and direction of the object in the virtual workspace based on the camera module 112. The matrix calculation unit 115 uses an algorithm for analyzing image data to calculate the initial model view matrix (MV) of the base mark according to the direction in which the camera module 112 observes the base mark and the target mark. b matrix) and the initial model view matrix (MV o matrix).

[0075] In step 3053, the matrix calculation unit 115 calculates the initial model matrix (M oAs described above, it is assumed that the target mark is a mark for identifying the actual work object and is arranged at an arbitrary position in the virtual work space reflecting the actual work space. The matrix calculation unit 115 calculates the initial model matrix (M) of the target mark arranged at an arbitrary position with respect to the origin determined in step 3051. o matrix).

[0076] In step 3054, the matrix calculation unit 115 uses the initial model view matrix (MV o matrix) and the initial model matrix (M) of the target marker calculated in step 3053 o matrix), calculate the initial view matrix (View matrix) of the camera module 112. As described in [Formula 1] above, the model view matrix is ​​the product of the view matrix and the model matrix. According to [Formula 1], the initial model view matrix (MV o The initial model matrix (M matrix) can be used to represent the position and orientation of the target marker in the virtual workspace. o The product of the model matrix and the initial view matrix (View matrix) representing the direction in which the camera module 112 observes the virtual work object. Since the model matrix is ​​invertible, if the inverse matrix of the model matrix is ​​multiplied on both sides of [Math 1], it is organized into the following math formula.

[0077]

Mathematical formula 2

[0078] View matrix=MV o matrix*(M o matrix) -1

[0079] The matrix calculation unit 115 may use the above-mentioned equation 2 to calculate the initial viewing matrix (Viewmatrix) of the camera module 112 .

[0080] In step 3055, the matrix calculation unit 115 uses the initial view matrix (View matrix) of the camera module 112 calculated in step 3054 and the initial model view matrix (MV b matrix) to calculate the model matrix (M b As mentioned above, the initial model view matrix (MV b The model matrix (M matrix) can be used to represent the position and orientation of the basic marker in the virtual work space. bmatrix) and the initial view matrix (Viewmatrix) representing the direction in which the camera module 112 observes the virtual operation object. b matrix) can be represented as follows.

[0081]

Mathematical formula 3

[0082] M b matrix=(View matrix) -1 *MV b matrix

[0083] Mathematical formulas 2 and 3 can be rearranged as follows.

[0084]

Mathematical formula 4

[0085] M b matrix=(View matrix) -1 *MV b matrix

[0086] =(MV o matrix*(M o matrix) -1 ) -1 *MV b matrix

[0087] =M o matrix*(MV o matrix) -1 *MV b matrix

[0088] The matrix calculation unit 115 calculates the model matrix (Mbmatrix) of the base mark by using the above-mentioned mathematical formula 4. As described above, the base mark is a mark that serves as a reference point in the virtual work space. It is equivalent to a fixed place or object in the actual work site, so the base mark does not move. Therefore, the model matrix (Mbmatrix) of the base mark b matrix) is fixed.

[0089] The matrix calculation unit 115 of the field device 11 can calculate the model matrix of the base mark and the initial model matrix of the target mark in the virtual work space having a common coordinate system based on the base mark, using the mathematical formula described above.

[0090] In step 306, the processor 111 of the field device 11 determines whether an instruction for changing at least one of the position and the direction of the virtual work object in the virtual work space is received from the remote device 21 of the remote operator 20 through the communication module 118. The remote operator 20 can operate the position and direction of the virtual work object in the virtual work space through the virtual work application executed in the remote device 21. When the remote operator 20 receives an instruction for changing at least one of the position and the direction of the virtual work object in the virtual work space through the communication module 118 of the field device 11, step 307 is executed, otherwise step 308 is executed.

[0091] In step 307, when the virtual space processing unit 114 of the field device 11 receives an instruction to change at least one of the position and direction of the virtual work object from the remote device 21 of the remote operator 20, the virtual space processing unit 114 reflects the change of the position and direction of the virtual work object to the virtual work space according to the received instruction. When the remote operator 20 changes the virtual work object, the virtual space processing unit 114 of the field device 11 reflects the change of the virtual work object to the virtual work space. More specifically, when the remote operator 20 inputs an instruction to move the position or rotate the virtual work object in the virtual work space to the remote device 21, the remote device 21 sends an instruction about the change of the virtual work object to the field device 11 through the communication module of the remote device 21. The field device 11 that receives such an instruction moves the position or rotates the virtual work object in the virtual work space according to the received instruction, and then reflects the operation of the remote operator 20 on the virtual work object to the virtual work space. For example, when the remote operator 20 inputs an instruction for moving the virtual work object to the remote device 21, the field device 11 of the field operator 10 receives the instruction for moving the virtual work object from the remote device 21 of the remote operator 20. The field device 11 of the field operator 10 moves the virtual work object on the virtual work space according to the received instruction.

[0092] In connection with this, Figure 6 is a schematic diagram showing an example screen displayed on the display unit of the user device before and after the virtual operation object moves. Figure 6 , Figure 6 (a) is an example of a screen showing that a powder cartridge as an actual work object and a virtual work object of the powder cartridge are displayed on the display unit 116 of the field device 11. Figure 6In the example of (a), the field device 11 is photographing the powder barrel as the actual operation object and the actual operation space where the powder barrel is located through the camera module, and the field device 11 displays the photographed powder barrel and the actual operation space through the display unit 116. In the screen displayed by the display unit 116, the powder barrel as the actual operation object and the cylindrical virtual operation object of the same shape as the powder barrel are arranged in the same position and direction. In other words, the actual powder barrel and the cylindrical virtual operation object are displayed overlappingly on the display unit 116.

[0093] Here, when the remote worker 20 inputs an instruction to move upward the cylindrical virtual work object on the virtual work space through the remote device 21, the cylindrical virtual work object corresponding to the powder cartridge moves upward on the virtual work space. Figure 6 (b) shows an example of a screen displayed on the display unit 116 of the on-site device 11 of the on-site worker 10 when the virtual work object of the powder cartridge is moved upward by the remote worker 20. Figure 6 (b), the on-site device 11 is photographing the actual work object and the actual work space, and the display unit 116 of the on-site device 11 displays a powder barrel as the actual work object and a cylindrical virtual work object corresponding to the powder barrel. As described above, in the example in which the remote operator 20 inputs an instruction to move the cylindrical virtual work object corresponding to the powder barrel upward in the virtual work space, the virtual work object displayed on the display unit 116 of the on-site device 11 moves upward. After observing the cylindrical virtual work object separated upward, the on-site operator 10 is able to separate and move the powder barrel as the actual work object upward. The on-site operator 10 can lift and remove the powder barrel in the actual work space according to the operation of the remote operator 20 in the virtual work space. In the above manner, the on-site operator 10 and the remote operator 20 can perform joint work through the virtual work space.

[0094] In step 308, when the field device 11 does not receive an instruction for changing at least one of the position and direction of the virtual work object from the remote device 21 of the remote worker 20, the camera module 112 of the field device 11 generates current image data representing the current state of the work object and the work site. During the virtual work performed by the field worker 10 and the remote worker 20, since the position and direction of the actual work object continue to change, the field worker 10 uses the camera module 112 of the field device 11 to continuously photograph the changing actual work object in order to perform the virtual work with the remote worker 20. The camera module 112 generates current image data representing the current state of the actual work object in the actual work space.

[0095] In step 309 , the mark detection unit 113 of the field device 11 detects the base mark and the target mark from the current image data input by the camera module 112 . The detailed description of the method of detecting the base mark and the target mark is replaced with the content described in step 303 .

[0096] In step 310, the matrix calculation unit 115 of the field device 11 calculates the current model matrix of the target mark sensed in step 309. The matrix calculation unit 115 calculates the current model matrix representing the current state (position and orientation) of the target mark. When the state (position and orientation) of the work object changes, the model matrix representing the state (position and orientation) of the target mark corresponding to the work object may change.

[0097] In connection with this, Figure 7 It is calculated Figure 3 A detailed flowchart of the steps of the current model matrix of the target label is shown, Figure 8 is a schematic diagram for explaining the calculation method of the current model matrix of the target label. Figure 7 and Figure 8 As mentioned above, the model view matrix is ​​represented by the product of the model matrix and the view matrix. In step 3101, the matrix calculation unit 115 calculates the current model view matrix (MV b `matrix) and the current model view matrix (MV o `matrix). The detailed description of the method for calculating the model view matrix is ​​replaced by the content recorded in step 3052.

[0098] In step 3102, the matrix calculation unit 115 uses the model matrix (M b matrix) to calculate the current view matrix (View`matrix) of the camera module 112. As mentioned above, the current model view matrix (MV b `matrix) can be composed of a model matrix (M b The current view matrix of the camera module 112 is represented by the product of the current view matrix (View`matrix) representing the direction in which the camera module 112 observes the work object. Here, the basic mark is a fixed mark with no state change, so the model matrix of the basic mark remains unchanged. Therefore, the current view matrix (View`matrix) of the camera module 112 can be represented by the following formula.

[0099]

Mathematical formula 5

[0100] View`matrix=MV b`matrix*(M b matrix) -1

[0101] The matrix calculation unit 115 calculates the current viewing angle matrix (View`matrix) of the camera module 112 using the mathematical formula 5.

[0102] In step 3103, the matrix calculation unit 115 uses the current view matrix (View`matrix) of the camera module 112 calculated in step 3102 and the current model view matrix (MV o `matrix) calculates the current model matrix (M) of the target marker o `matrix). As mentioned above, the current model view matrix (MV o `matrix) can be composed of the current model matrix (M) representing the current position and orientation of the target marker in the virtual workspace o The current model matrix (M `matrix) of the target marker is represented by the product of the current viewing angle matrix (View `matrix) representing the direction in which the camera module 112 observes the current working object. o `matrix) can be expressed as follows.

[0103]

Mathematical formula 6

[0104] M o `matrix = (View`matrix) -1 *MV o `matrix

[0105] Mathematical formulas 5 and 6 can be rearranged as follows.

[0106]

Mathematical formula 7

[0107] M o `matrix=(MV b `matrix*(M b matrix) -1 ) -1 *MV o `matrix

[0108] =M b matrix*(MV b `matrix) -1 *MV o `matrix

[0109] The matrix calculation unit 115 calculates the current model matrix (M) representing the current position and direction of the target mark by using the above-mentioned mathematical formula 7.o `matrix).

[0110] In step 311, the virtual space processing unit 114 of the field device 11 processes the target marker according to the current model matrix (M o `matrix) updates the state of the virtual job object in the virtual job space. As mentioned above, the target tag indicates the job object. The current model matrix (M o `matrix) represents the current position and direction of the work object in the virtual work space. The current model matrix (M o `matrix) and the initial model matrix (M o If the actual operation object is different after comparison, it can be considered that the actual operation object has moved. The change in the position or direction of the actual operation object is displayed in the current model matrix (M) of the target marker calculated in step 310. o Thus, in order to reflect the changes of the actual work object in the virtual work space, the virtual space processing unit 114 can use the virtual work object in the virtual work space to change the virtual work object according to the current model matrix (M o The state of the work object in the virtual work space is updated by reconfiguring the position and direction of the target matrix. In this way, the field device 11 can reflect the changes of the actual work object in the actual work space to the virtual work space in real time. When the state of the actual work object changes, the virtual space processing unit 114 of the field device 11 operates the virtual work object according to the current model matrix of the target mark, thereby reflecting the state change of the actual work object to the virtual work space.

[0111] In connection with this, Fig. 9 Schematic diagram showing example screens displayed on the display units of the on-site device and the remote device before and after the actual work object is moved. Fig. 9 , Fig. 9 (a) is an example of a screen showing a powder cartridge as an actual work object and a virtual work object of the powder cartridge displayed on the display unit 116 of the on-site device 11 and a screen showing a virtual work object of a virtual work space displayed on the display unit 116 of the remote device 21. Fig. 9(a), the on-site device 11 is photographing the powder barrel as the actual operation object and the actual operation space where the powder barrel is located through the camera module, and the on-site device 11 displays the photographed powder barrel and the actual operation space through the display unit 116. In the screen displayed on the display unit 116, the powder barrel as the actual operation object and the cylindrical virtual operation object of the same shape as the powder barrel are arranged in the same position and direction. In other words, the actual powder barrel and the cylindrical virtual operation object are overlapped and displayed on the display unit 116. In addition, in the screen displayed on the display unit of the remote device 21, the virtual operation object is arranged at an arbitrary position in the virtual operation space.

[0112] When the on-site worker 10 moves the powder cartridge as the actual work object in the actual work space, the virtual work object corresponding to the powder cartridge also moves in the virtual work space by an amount corresponding to the movement of the powder cartridge. Fig. 9 (b) shows an example of a screen displayed on the display unit of the on-site device 11 of the on-site worker 10 and the remote device 21 of the remote worker 20 when the on-site worker 10 moves the powder cartridge as the actual work object in the actual work space. Fig. 9 (b), the on-site device 11 is photographing the actual work object, and the display units of the on-site device 11 and the remote device 21 display a powder barrel corresponding to the actual work object and a cylindrical virtual work object corresponding to the powder barrel. As mentioned above, in the example where the on-site operator 10 moves the powder barrel as the actual work object in the actual work space, the powder barrel as the actual work object and the cylindrical virtual work object displayed on the display units of the on-site device 11 and the remote device 21 all move in the same direction. The remote operator 20 can confirm the operation of the on-site operator 10 on the actual work object through the changes in the virtual work object. As described above, by timely reflecting the operation of the actual work object on the virtual work object in the virtual work space, the remote operator 20 can accurately know the current state of the actual work object in the actual work space. As a result, precise joint operations between the on-site operator 10 and the remote operator 20 can be performed.

[0113] In step 312, the display unit 116 of the field device 11 displays the current state of the virtual work object reflected in step 307 or updated in step 310 in the virtual work space to the field worker 10, and transmits the current state of the virtual work object reflected in step 307 or updated in step 310 in the virtual work space to the remote device 21 through the communication module 118, so that the virtual content in the virtual space based on the common coordinate system can be provided to the field worker 10 and the remote worker 20. As described above, the remote worker 20 reflects the change of the position and direction of the virtual work object in the virtual work space according to the instruction input through the remote device 21, or the field worker 10 updates the state of the virtual work object in the virtual work space according to the change of the position and direction of the actual work object in the actual work space, so that the state of the virtual work object in the virtual work space is changeable. The display unit 116 of the field device 11 displays the current state of the virtual work object that has changed in the virtual work space. The field worker 10 can refer to the current state of the virtual work object displayed on the display unit 116 of the field device 11 and perform work on the actual work object, such as repair work on a faulty part or replacement work of a component.

[0114] Furthermore, the field device 11 sends data indicating the current state of the virtual work object in the virtual work space to the remote device 21 through the communication module 118. The remote device 21 displays the current state of the virtual work object in the virtual work space on the display unit of the remote device 21 according to the received data. The remote operator 20 can know the state of the actual work object in the actual work space through the current state of the virtual work object displayed on the display unit of the remote device 21. The remote operator 20 refers to the state of the actual work object in the actual work space and performs work on the virtual work object in the virtual work space, thereby being able to provide the field operator 10 with work information on the actual work object. As described above, the embodiment of the present invention provides the state of the virtual work object obtained as a virtual content in the virtual work space based on the common coordinate system to the field operator 10 and the remote operator 20 through the field device 11 and the remote device 21.

[0115] In connection with this, Fig.10 1 is a schematic diagram showing an example screen in which the virtual work space according to the present embodiment is displayed on the display unit of the on-site device and the remote device. Fig.10 (a) is a schematic diagram showing a screen of a field device 11 of a field worker 10 in which a virtual work space is displayed. Fig.10 (b) is a schematic diagram showing a virtual work space displayed on a screen of a remote device 21 of a remote operator 20. Fig.10(a) In the case where the field device 11 is a device supporting AR (augmented reality), a screen showing a virtual work object corresponding to an actual work object and an actual work space where the actual work object is located is provided to the field worker 10. The virtual work object corresponding to the actual work object is matched with the actual work object and displayed overlappingly at the same position.

[0116] The remote device 21 receives data on the virtual work space and the virtual work object sent from the on-site device 11 via the network 30, and can provide a screen showing the status of the virtual work object in the virtual work space to the remote operator using the sent data. Fig.10 (b) When the remote device 21 is a device supporting VR (virtual reality), a screen displaying a virtual work object and a virtual work space is provided to the remote worker 20 .

[0117] Here, the screen displaying the virtual work object in the virtual work space may change according to the virtual environment supported by the device. As in the example described above, when the on-site device is a device that supports AR, the on-site device displays an augmented reality screen, and when the remote device is a device that supports VR, the remote device displays a virtual reality screen. Conversely, when the on-site device is a device that supports VR, the on-site device displays a virtual reality screen, and when the remote device is a device that supports AR, the remote device displays an augmented reality screen.

[0118] Steps 306 to 312 are repeatedly performed during the virtual operation between the on-site worker 10 and the remote worker 20. According to the present embodiment, during the virtual operation between the on-site worker 10 and the remote worker 20, when the remote worker 20 operates the virtual operation object through the remote device 21, the operation of the virtual operation object is reflected in the virtual operation space, thereby providing the on-site worker 20 with information about the operation performed by the remote worker 20 on the virtual operation object. In addition, the on-site device 11 uses a camera module to photograph the actual operation object in the actual operation space in real time, and reflects the operation performed by the on-site worker 10 on the actual operation object in the actual operation space to the virtual operation object in the virtual operation space, thereby providing the remote worker 20 with information about the status of the actual operation object in real time.

[0119] According to the above-mentioned embodiment of the present invention, in the method for providing virtual content in a virtual space based on a common coordinate system, a fixed location in the actual work space is designated as a basic mark, and a model matrix of a target mark representing the state (position and direction) of the target mark used to indicate the actual work object is calculated based on the fixed basic mark. The embodiment of the present invention determines the position and direction of the actual work object reflected in the virtual work space based on a fixed location in the actual work space, so the coordinate system and scale of the virtual work space are fixed. Thus, the coordinate system and scale of all work participants who share the virtual work space are unified. By fixing the coordinate system and scale on the virtual work space, it is possible to provide correct information about the actual work object to remote workers participating in the virtual work.

[0120] Furthermore, in the method for providing virtual content in a virtual space based on a common coordinate system, according to an embodiment of the present invention, the state (position and direction) change of the actual work object and the movement of the camera module are grasped based on fixed basic marks, so that the user device can distinguish the movement of the work object from the movement of the camera module. The user device calculates the model matrix of the target mark and the viewing angle matrix of the camera module based on the basic mark, so that the movement of the work object and the movement of the camera module can be distinguished in the virtual work space. As a result, the user device of the embodiment of the present invention can prevent the problem of mistakenly reflecting the camera movement as the movement of the work object in the virtual work space. By distinguishing the movement of the work object from the movement of the camera module, the user device of the embodiment of the present invention can convey the correct information about the work object to the remote operator participating in the virtual work.

[0121] In addition, the embodiments of the present invention as described above can be made into a program that can be run on a processor of a computer, and can be implemented using a computer-readable storage medium on a general-purpose digital computer that runs the program. In addition, the data structure used for the above-mentioned embodiments of the present invention can be recorded on the computer-readable storage medium by various means. The computer-readable storage medium includes storage media such as magnetic storage media (e.g., ROM, floppy disk, hard disk, etc.), optically readable media (e.g., CD-ROM, DVD, etc.).

[0122] The present invention is described above with the preferred embodiment as the center. A person skilled in the art of the present invention should understand that various modifications can be made without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be understood from the perspective of explanation rather than from the perspective of limitation. The scope of the present invention is not shown in the above description, but the scope of the rights, and all differences within the equivalent scope should be understood to be included in the present invention.

Claims

1. A method for providing virtual content in a virtual space based on a common coordinate system, characterized in that: include: detecting, from initial image data representing an initial state of an actual work object in an actual work space, a base mark as a mark for identifying a fixed location in the actual work space and a target mark as a mark for identifying the actual work object; Calculating an initial model matrix of the target mark representing an initial position and orientation of the target mark in a virtual work space having a common coordinate system based on the detected base mark and a model matrix of the base mark representing a position and orientation of the detected base mark; Calculate a current model matrix of the target marker representing the current position and orientation of the target marker using the calculated model matrix of the base marker; generating a virtual work object including a three-dimensional model shape of the actual work object in the virtual work space; as well as By updating at least one of the position and the direction of the virtual working object according to the difference between the current model matrix and the initial model matrix of the target marker, the change of the position and the direction of the actual working object is reflected to the virtual working object.

2. The method for providing virtual content in a virtual space based on a common coordinate system according to claim 1, characterized in that: The target marker is detected, a shape included in the initial image data is extracted, and the base marker and the target marker are detected using the extracted shape.

3. The method for providing virtual content in a virtual space based on a common coordinate system according to claim 2, characterized in that: The target marker is detected, the extracted shape is compared with the shapes of the base marker and the target marker, a shape matching the shape of the base marker is detected as the base marker, and a shape matching the shape of the target marker is detected as the target marker.

4. The method for providing virtual content in a virtual space based on a common coordinate system according to claim 1, characterized in that: Calculating the model matrix of the basic marker includes: Calculate, based on the initial image data, an initial model viewing angle matrix of the base mark representing the position and orientation of the base mark based on the viewing angle of the camera module used for shooting the initial image data, and an initial model viewing angle matrix of the target mark representing the position and orientation of the target mark based on the viewing angle of the camera module; Calculating the initial model matrix of the target mark on the virtual work space; and The model matrix of the base marker is calculated using the calculated initial model view matrix of the base marker, the calculated initial model matrix of the target marker, and the calculated initial model view matrix of the target marker.

5. The method for providing virtual content in a virtual space based on a common coordinate system according to claim 4, characterized in that: The model matrix of the base marker is calculated using the calculated initial model perspective matrix of the base marker, the calculated initial model matrix of the target marker, and the calculated initial model perspective matrix of the target marker. The model matrix of the base marker is calculated according to the following mathematical formula: M b =M o *(MV o )-1*MV b Among them, M b : The model matrix of the base label, MV b : The initial model view matrix of the base marker, M o : The initial model matrix of the target label, MV o : The initial model view matrix of the target marker.

6. The method for providing virtual content in a virtual space based on a common coordinate system according to claim 1, characterized in that: Calculating the current model matrix of the target marker includes: Calculating a current model viewing matrix of the base marker and a current model viewing matrix of the target marker based on the current image data; and The current model matrix of the target marker is calculated using the calculated current model view matrix of the base marker, the calculated current model view matrix of the target marker, and the calculated model matrix of the base marker.

7. The method for providing virtual content in a virtual space based on a common coordinate system according to claim 6, characterized in that: Using the calculated current model perspective matrix of the base tag, the calculated current model perspective matrix of the target tag and the calculated model matrix of the base tag, the current model matrix of the target tag is calculated according to the following mathematical formula: M o `=M b *(MV b `)-1*MV o ` M o `: Current model matrix of target label, MV o `: The current model view matrix of the target marker, MV b `: The current model view matrix of the base marker, M b : The model matrix of the basis labeling.

8. The method for providing virtual content in a virtual space based on a common coordinate system according to claim 1, characterized in that: Also includes: determining whether an instruction for changing the position and direction of the virtual working object is received from the remote device of the remote worker; as well as When a command for changing the position and direction of the virtual work object is received, the change in the position and direction of the virtual work object is reflected on the virtual work space according to the received command.

9. The method for providing virtual content in a virtual space based on a common coordinate system according to claim 1, characterized in that: Also includes: The position and direction of the virtual work object in the virtual work space are updated according to the calculated current model matrix of the target mark.

10. A computer-readable storage medium, characterized in that: A program for executing the method according to claim 1 on a computer is recorded therein.

11. An information device, characterized in that: include: a mark detection unit for detecting a base mark as a mark for identifying a fixed point in the actual work space and a target mark as a mark for identifying the actual work object from initial image data representing an initial state of the actual work object in the actual work space; A matrix calculation unit that calculates an initial model matrix of the target mark representing an initial position and orientation of the target mark and a model matrix of the base mark representing the position and orientation of the detected base mark in a virtual work space having a common coordinate system based on the detected base mark, and calculates a current model matrix of the target mark representing a current position and orientation of the target mark using the calculated model matrix of the base mark; A virtual work object including a three-dimensional model shape of the actual work object is generated in the virtual work space, and changes in the position and direction of the actual work object are reflected in the virtual space processing unit of the virtual work object by updating at least one of the position and direction of the virtual work object according to the difference between the current model matrix and the initial model matrix of the target mark.