Information processing method, information processing device, and information processing program

Abstract

The information processing device constructs, in a virtual space, a three-dimensional model of an object to be worked by a worker in an actual space, acquires position information and orientation information of the worker in the actual space, and calculates, on the basis of the position information and the orientation information, a three-dimensional model of the object to be worked by the worker in the actual space. The position and orientation of the virtual camera in the virtual space are synchronized with the position and orientation of the operator, a virtual camera image representing the three-dimensional model captured by the synchronized virtual camera is generated, and output information for displaying an output image including the virtual camera image is output.

Description

Technical Field

[0001] The present disclosure relates to a technology for assisting a worker's work. Background Art

[0002] Patent Document 1 discloses a technology in which a 3D model generation device generates a 3D model of a subject by visual volume intersection based on camera images of a near camera and a far camera, and generates a 3D model by assuming that the subject exists outside the viewing angle range of the near camera.

[0003] However, Patent Document 1 only generates a virtual camera image enclosed in a virtual space, and does not have the function of generating a virtual camera image associated with a real space. Thus, under the condition that the image of the work site cannot be output, the remote manager cannot confirm the place observed by the operator, so further improvement is needed.

[0004] Prior Art Literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2022-29730 Summary of the invention

[0007] The present disclosure aims to solve such a problem, and an object of the present disclosure is to provide a technology that allows a remote manager to check a location observed by a worker under the condition that an image of a work site cannot be output.

[0008] An information processing method in one embodiment of the present invention is an information processing method in an information processing system for assisting an operator's work. In the information processing method, a three-dimensional model of an object that the operator works on in a real space is constructed in a virtual space, and position information and posture information of the operator in the real space are obtained. Based on the position information and the posture information, the position and posture of a virtual camera in the virtual space are synchronized with the position and posture of the operator to generate a virtual camera image, the virtual camera image representing the three-dimensional model captured by the synchronized virtual camera, and output information for displaying an output image including the virtual camera image is output.

[0009] According to the present disclosure, even if it is impossible to output an image of the work site, a remote manager can confirm the location observed by the worker. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a block diagram showing an example of the overall configuration of the information processing system in Embodiment 1 of the present disclosure.

[0011] Figure 2It is a diagram showing an overview of processing of the information processing system.

[0012] Figure 3 This is a diagram showing an example of a real space representing a scene.

[0013] Figure 4 It is a diagram showing a situation of spatial calibration.

[0014] Figure 5 It is shown Figure 4 A diagram of the situation following the spatial calibration.

[0015] Figure 6 This is a diagram showing an example of a virtual camera image included in an output image.

[0016] Figure 7 It is a flowchart showing the overall situation of the processing of the information processing system.

[0017] Figure 8 is a flowchart showing details of the processing of spatial calibration.

[0018] Fig. 9 is a flowchart showing details of the stabilization process.

[0019] Fig.10 is a flowchart showing the details of the synchronization process.

[0020] Fig.11 This is a diagram showing an example of a bird's-eye view image.

[0021] Fig.12 The diagrams show bird's-eye view images displayed on the manager terminal and the operator terminal respectively.

[0022] Fig.13 This is a flowchart showing the processing of spatial calibration in Modification 1 of Implementation Example 1.

[0023] Fig.14 This is a flowchart showing an example of synchronization processing in Modification 1 of Implementation Example 1.

[0024] Fig.15 This is a flowchart showing the processing of space calibration in Modification 2 of Implementation 1.

[0025] Fig.16 This is an explanatory diagram of the processing in the second variation of the first embodiment.

[0026] Fig.17 This is an explanatory diagram of the processing in the second variation of the first embodiment.

[0027] Fig.18 This is an explanatory diagram of the processing in the second variation of the first embodiment.

[0028] Fig.19 This is a block diagram showing an example of the configuration of an information processing system in the second embodiment.

[0029] Fig. 20 This is a flowchart showing details of the processing of spatial calibration in the second embodiment.

[0030] Fig.21 This is a flowchart showing the details of the stabilization process in the third embodiment.

[0031] Fig. 22 This is a block diagram showing an example of the configuration of an information processing system in Embodiment 3.

[0032] Fig.23 This is a flowchart showing details of the processing of spatial calibration in the third embodiment.

[0033] Fig.24 This is a flowchart showing the details of the stabilization process in the third embodiment. DETAILED DESCRIPTION

[0034] (Knowledge that forms the basis of this disclosure)

[0035] When an operator is working on an object at a work site, the operator sometimes advances the work while confirming instructions from a manager outside the work site. In this case, if the manager can confirm which part of the object the operator is observing, the manager can smoothly instruct the operator. To achieve this, for example, if a shooting device such as a sports camera or smart glasses is worn on the operator's head, and the image of the work site captured by the shooting device is sent in real time to the manager terminal owned by the manager, the manager can confirm which part of the object the operator is observing.

[0036] However, there are work sites where it is prohibited to output images to the outside from the viewpoint of safety. In this case, there is a problem that the manager cannot confirm the part observed by the worker.

[0037] The present disclosure aims to solve such problems.

[0038] (1) An information processing method in one aspect of the present disclosure is an information processing method in an information processing system for assisting an operator's work. In the information processing method, a three-dimensional model of an object that the operator works on in a real space is constructed in a virtual space, and position information and posture information of the operator in the real space are obtained. Based on the position information and the posture information, the position and posture of a virtual camera in the virtual space are synchronized with the position and posture of the operator to generate a virtual camera image, the virtual camera image representing the three-dimensional model captured by the synchronized virtual camera, and output information for displaying an output image including the virtual camera image is output.

[0039] According to this structure, the position and posture of the virtual camera are synchronized with the position and posture of the operator in the real space, and a virtual camera image of the three-dimensional model of the object photographed by the synchronized virtual camera is generated. Thus, the manager can confirm the virtual camera image as the image of the three-dimensional model observed in the line of sight of the operator. As a result, under the condition that the image of the work site cannot be output to the outside, the remote manager can confirm the place observed by the operator.

[0040] (2) In the information processing method described in (1) above, the position information and the posture information may be actual mark information indicating the position and posture of an actual mark worn by the operator. In the information processing method, initial mark information is obtained, the initial mark information indicating the position and posture of an actual mark set in the actual space. Based on the initial mark information and the relative position of the object and the actual mark, a spatial calibration is performed to establish a correspondence between the actual space and the virtual space, and actual mark information is obtained. The actual mark information indicates the position and posture of the actual mark worn by the operator in the actual space. Based on the actual mark information, the position and posture of the virtual camera are synchronized with the position and posture of the actual mark indicated by the actual mark information.

[0041] According to this structure, initial marking information indicating the position and posture of the actual marking is obtained, and spatial calibration is performed to establish correspondence between the actual space and the virtual space based on the initial marking information and the relative position of the object and the actual marking. Therefore, the actual space and the virtual space can be accurately established to correspond, thereby generating a virtual camera image that accurately reproduces the place observed by the operator without using the actual image of the work site.

[0042] (3) In the information processing method described in (1) or (2) above, it may also be that, in the spatial calibration, based on the initial marking information, a virtual marker corresponding to the actual marker is set at an arbitrary position in the virtual space, and based on the relative position, the three-dimensional model is set in the virtual space with the virtual marker as a reference so as to maintain the relative position relationship between the actual marker and the object, and calibration information is generated, the calibration information including an actual reference vector indicating the relative position of the actual marker relative to the object when the spatial calibration is performed, and a virtual reference vector indicating the relative position of the virtual marker relative to the three-dimensional model.

[0043] According to this structure, the position of the actual mark is set at an arbitrary position in the virtual space, and the three-dimensional model is set in the virtual space based on the virtual mark, so that the relative position relationship between the actual mark and the object is maintained. Therefore, the coordinate system of the virtual space can be set so that the position of the virtual mark essentially becomes the origin, and the position of the object can be determined using the coordinate system. Therefore, in an environment using a sensor that indicates the position of the actual mark by its relative position to itself, the actual mark and the virtual mark can be accurately established. In addition, since the actual reference vector and the virtual reference vector are generated as calibration information, the actual reference vector and the virtual reference vector can be used to accurately synchronize the virtual camera and the actual mark during operation.

[0044] (4) In the information processing method described in (3) above, it may also be that, during the synchronization, based on the actual mark information, an actual space vector representing the relative position of the actual mark with respect to the object is calculated, and a difference vector between the actual space vector and the actual reference vector is calculated. The position of the virtual mark relative to the three-dimensional model is determined based on the virtual reference vector and the difference vector, and the posture of the virtual mark is determined based on the posture of the actual mark included in the actual mark information. The position of the virtual marker is determined as the position of the virtual camera, and the posture of the virtual marker is set as the posture of the virtual camera.

[0045] According to this structure, during operation, the coordinate system of the virtual space can be set so that the position of the virtual marker set during calibration is substantially the origin, and the position of the virtual marker can be determined using this coordinate system. Therefore, in an environment where a sensor that indicates the position of the actual marker by its relative position to itself is used, the actual marker and the virtual marker can be accurately associated during operation.

[0046] (5) In the information processing method described in (3) above, the initial marker information and the actual marker information may be information acquired by sensing the surrounding environment to detect the relative position of the self.

[0047] According to this structure, in an environment where the position of the actual marker is detected by using a method of sensing the surrounding environment to detect its relative position, the actual marker and the virtual marker can be accurately associated. An example of such a sensor is an inertial sensor or a sensor using SLAM technology.

[0048] (6) In the information processing method described in (2) above, it may also be that, in the space calibration, based on the initial marking information, a virtual mark corresponding to the actual mark is set in the virtual space, and based on the relative position, the three-dimensional model is set in the virtual space so that the relative position relationship between the actual mark and the object is maintained, and the coordinates of the set three-dimensional model are generated as calibration information.

[0049] According to this structure, the three-dimensional model is set in the virtual space so as to maintain the relative positional relationship between the actual marker and the object, and the coordinates of the set three-dimensional model are generated as calibration information. Therefore, in an environment where the position of the actual marker is detected using a sensor that represents the position of the object using the coordinate system of the actual space, the position and posture of the virtual marker can be set by maintaining the position and posture represented by the actual marker information as it is.

[0050] (7) In the information processing method described in (6) above, in the synchronization, the position of the actual mark may be determined as the position of the virtual camera, and the posture of the actual mark may be determined as the posture of the virtual camera.

[0051] According to this configuration, in an environment where the position of the real marker is detected using a sensor that indicates the position of the object using a coordinate system in real space, the position and posture of the virtual marker can be set using the position and posture indicated by the real marker information as is.

[0052] (8) In the information processing method described in (6) above, the initial label information and the actual label information may be information acquired using a sensor that indicates a position of an object using a coordinate system of the actual space.

[0053] According to this configuration, in an environment where the position of the real marker is detected using a sensor that expresses the position of the object using a coordinate system in real space, the real marker and the virtual marker can be accurately associated. An example of such a sensor is an optical sensor.

[0054] (9) In the information processing method described in any one of (2) to (8) above, the initial mark information may include first initial mark information indicating the position and posture of a first actual mark set at a position different from the object, and second initial mark information indicating the position and posture of a second actual mark set at the position of the object. In the spatial calibration, based on the first initial mark information, a first virtual mark corresponding to the first actual mark is set in the virtual space, and based on the second initial mark information, a second virtual mark corresponding to the second actual mark is set in the virtual space, so that the relative positional relationship between the first actual mark and the second actual mark is maintained.

[0055] According to this configuration, the first real marker and the second real marker can be used to accurately associate the first real marker with the first virtual marker.

[0056] (10) In the information processing method described in (9) above, in the synchronization, the position of the first real marker may be determined as the position of the virtual camera, and the posture of the first real marker may be determined as the posture of the virtual camera.

[0057] According to this configuration, in an environment where the position of the real marker is detected using a sensor that indicates the position of the object using a coordinate system in real space, the position and posture of the virtual marker can be set using the position and posture indicated by the real marker information as is.

[0058] (11) In the information processing method described in any one of (1) to (10) above, the virtual camera image may further change in synchronization with changes in the position and posture of the virtual camera.

[0059] According to this configuration, since the virtual camera image changes in synchronization with changes in the position and posture of the virtual camera, a virtual camera image that represents the place observed by the operator in real time can be obtained.

[0060] (12) In the information processing method described in (1) to (11) above, it may also be that, further, the output image includes an overhead image of the three-dimensional model and a camera icon displayed in correspondence with the overhead image, and the display mode of the camera icon changes synchronously with the position and posture of the virtual camera.

[0061] According to this configuration, it is possible to provide information indicating which part of the object the operator is observing from a bird's-eye view.

[0062] (13) In the information processing method described in (12) above, the output image may be displayed on the operator terminal of the operator and the manager terminal of the manager. In the information processing method, further, display object information is obtained from the manager terminal, and the display object information is used to display a display object indicating a position specified by the manager in the three-dimensional model shown in the bird's-eye view image or the virtual camera image. Furthermore, based on the display object information, the display object is displayed overlappingly on the bird's-eye view image or the virtual camera image.

[0063] According to this configuration, since the display object indicating the position designated by the administrator is displayed in a superimposed manner on the bird's-eye view image or the virtual camera image, the administrator can share the position designated by the administrator with the operator.

[0064] (14) In the information processing method described in any one of (1) to (13) above, the three-dimensional model may include a main body and a movable part installed on the main body. In the information processing method, operation information is further obtained from an administrator terminal, and the operation information represents an operation from the administrator on the movable part. The image representing the movable part included in the virtual camera image changes based on the operation information.

[0065] According to this configuration, the administrator can remotely operate the movable portion of the three-dimensional model.

[0066] (15) In the information processing method described in any one of (2) to (8) above, the initial marker information and the actual marker information may be information acquired from a sensor capable of detecting a position and a posture of the actual marker.

[0067] According to this configuration, the position and posture of the actual marker can be measured using a sensor, thereby obtaining initial marker information and actual marker information.

[0068] (16) In another embodiment of the present invention, an information processing device assists an operator in performing a task, and the information processing device comprises: a construction unit that constructs a three-dimensional model of an object that the operator performs a task on in a real space in a virtual space; a first acquisition unit that acquires position information and posture information of the operator in the real space; a synchronization unit that synchronizes the position and posture of a virtual camera in the virtual space with the position and posture of the operator based on the position information and the posture information; a generation unit that generates a virtual camera image, wherein the virtual camera image represents the three-dimensional model captured by the synchronized virtual camera; and an output unit that outputs output information for displaying an output image including the virtual camera image.

[0069] According to this configuration, it is possible to provide an information processing device capable of generating a virtual camera image that accurately reproduces a location observed by a worker without using an actual image of the work site.

[0070] An information processing program in another embodiment of the present invention enables a computer to function as an information processing system that assists an operator in his or her work, and the information processing program enables the computer to perform the following processing: constructing a three-dimensional model of an object that the operator is working on in a real space in a virtual space, obtaining position information and posture information of the operator in the real space, synchronizing the position and posture of a virtual camera in the virtual space with the position and posture of the operator based on the position information and the posture information, generating a virtual camera image, the virtual camera image representing the three-dimensional model captured by the synchronized virtual camera, and outputting output information for displaying an output image including the virtual camera image.

[0071] According to this configuration, it is possible to provide an information processing program capable of generating a virtual camera image that accurately reproduces a location observed by a worker without using an actual image of the work site.

[0072] The present disclosure can also be realized as an information updating system operated by such an information processing program. In addition, such a computer program can of course be circulated via a computer-readable non-transitory recording medium such as a CD-ROM or a communication network such as the Internet.

[0073] In addition, the embodiments described below all represent a specific example of the present disclosure. The numerical values, shapes, structural elements, steps, the order of steps, etc. shown in the following embodiments are an example and are not intended to limit the present disclosure. In addition, for the structural elements in the following embodiments that are not recorded in the independent technical solutions representing the highest concept, they are described as arbitrary structural elements. In addition, in all embodiments, each content can also be combined.

[0074] (Implementation method 1)

[0075] Figure 1 1 is a block diagram showing an example of the overall structure of the information processing system 100 in the first embodiment of the present disclosure. The information processing system 100 is a system in which a manager located at a remote location assists a worker who performs work on an object. Specifically, the system reproduces an image visually recognized by a worker located on site in a virtual space, and enables the manager to visually recognize the image in the virtual space, thereby sharing information between the worker and the manager, and assisting the worker's work from a remote location.

[0076] Figure 2 1 is a diagram showing an overview of the processing of the information processing system 100. Figure 2 In the figure, the left figure shows the scene, and the right figure shows the remote location. An object 501 is set in the scene. An operator 504 performs some operation on the object 501 on the scene. The information processing system 100 reproduces the image in such a field of view 502 in the virtual space. Specifically, the information processing system 100 constructs a three-dimensional model 221 obtained by simulating the object 501 in the virtual space, and generates a virtual camera image by shooting the three-dimensional model 221 using a virtual camera configured in the virtual space. The generated virtual camera image is displayed on the display of the administrator terminal 3 located at the remote location. Here, the virtual camera changes the posture in the virtual space in synchronization with the change of the line of sight of the operator 504, and shoots the virtual camera image. As a result, the administrator 514 who visually recognizes the virtual camera image can confirm what kind of place the operator 504 is looking at in the object 501. As a result, the administrator 514 can provide appropriate assistance to the operator. In addition, the information processing system 100 can also display the operator 504's gaze point superimposed on the virtual camera image.

[0077] The site is, for example, an actual space where an object is set. An example of a site is a factory, an experimental site, a test site, a chemical plant, etc. A factory can be a factory that manufactures electrical products such as televisions and washing machines, or a factory that manufactures automobiles, iron, etc. These sites are examples, and any site can be any site as long as it is a place where an operator performs work on an object. For example, a site can also be a place where maintenance of a machine or equipment is performed.

[0078] An example of an object is a device to be maintained and a product manufactured by a manufacturing line. An example of a product is an electric product, a car, etc. An example of a device is a manufacturing line in a factory, etc. The structure of the information processing system 100 is described in detail below. Figure 1 . The information processing system 100 includes a detection device 1, an operator terminal 2, and an administrator terminal 3. The detection device 1 and the operator terminal 2 are located at a site where an object 501 is set. The administrator terminal 3 is located at a remote location. The detection device 1 and the operator terminal 2 are communicatively connected via a communication path C1. The communication path C1 can be, for example, a wireless communication path such as a wireless LAN or Bluetooth (registered trademark), or a wired communication path. The operator terminal 2 and the administrator terminal 3 are connected via a network. The network is, for example, a wide area communication network including the Internet.

[0079] The detection device 1 is installed at a site, and is a sensor capable of detecting the position and posture of an actual marker installed at the site.

[0080] The real marker is an object used to perform space calibration to establish correspondence between the real space and the virtual space. Figure 3 1 is a diagram showing an example of an actual space 600 representing a scene. An actual mark M1 and an object 501 are provided in the actual space 600. The actual mark M1 includes a base portion 611 and three protrusions 612 protruding upward from the base portion 611. A pedestal 601 is installed on the base portion 611 to keep the height of the actual mark M1 from the ground at a fixed height. In addition, the shape of the actual mark M1 shown here is an example, and other shapes may also be used without particular limitation.

[0081] The detection device 1 may be any device as long as it is a sensor device that detects the position and posture of the actual marker M1. In Embodiment 1, the detection device 1 is a sensor that senses the surrounding environment and detects its own position. An example of a method of estimating such a self-position is a method using SLAM (Simultaneous Localization and Mapping) technology using camera images or LiDAR. In addition, in Embodiment 1, the detection device 1 may also be a sensor for an inertial motion capture system (inertial sensor) or a sensor for a magnetic motion capture system (magnetic sensor).

[0082] The detection device 1 outputs marker information indicating the position and posture of the detected actual marker M1 to the operator terminal 2. The position of the actual marker M1 is indicated by, for example, three-dimensional vector data relative to the object 501 or three-dimensional vector data based on an initial position.

[0083] The detection device 1 outputs the marking information to the operator terminal 2 via the communication path C1.

[0084] The above is the description of the detection device 1. Next, the operator terminal 2 is described. The operator terminal 2 includes a processor 21, a memory 22, a display 23, and a communication unit 24. The operator terminal 2 is, for example, a portable information terminal carried by the operator. The portable information terminal is, for example, a smart phone, a tablet computer, and smart glasses.

[0085] The processor 21 is constituted by, for example, a central processing unit. The processor 21 includes a construction unit 211, a first acquisition unit 212, an execution unit 213, a second acquisition unit 214, a synchronization unit 215, a generation unit 216, and an output unit 217. The construction unit 211 to the output unit 217 may be implemented by the central processing unit executing an information processing program that enables the computer to function as the operator terminal 2, or may be implemented by a dedicated hardware circuit.

[0086] The construction unit 211 constructs a three-dimensional model 221 of the object 501 that the operator works on in the real space in a three-dimensional virtual space. The construction unit 211 is a three-dimensional model obtained by simulating the shape of the object 501. The construction unit 211 only needs to obtain three-dimensional CAD (Computer-Aided Design) data of the object 501 or BIM (Building Information Modeling) data of the object 501, and generate the obtained three-dimensional CAD data or BIM data into the three-dimensional model 221.

[0087] Alternatively, the construction unit 211 may generate an object 501 including a main body and a movable part attached to the main body. The movable part is a three-dimensional model configured to be movable based on operation information representing an operation inputted by the administrator 514 to the administrator terminal 3. In this case, the three-dimensional model 221 is generated using, for example, a game engine. The construction unit 211 may also generate the three-dimensional model 221 using a visual volume intersection method. The three-dimensional model 221 may be either point group data or voxel data. The construction unit 211 stores the generated three-dimensional model 221 in the memory 22.

[0088] The first acquisition unit 212 acquires initial marker information indicating the position and posture of the actual marker M1 in the actual space 600. Figure 3 . In the actual space 600, the actual marker M1 is set at a position separated from the object 501 by a reference distance D1 and at a reference height h from the ground. In addition, the actual marker M1 is set so that the posture D2 is parallel to the ground. The posture D2 is represented by a three-dimensional vector representing the posture of the actual marker M1. In addition, the actual marker M1 is set at a position where the posture D2 is parallel to the front direction (not shown) of the object 501. In addition, the actual marker M1 is set so that a straight line extending from the position P1 in the direction of the posture D2 intersects with the center position O1 of the object 501. The position P1 is, for example, the front position of the actual marker M1. The front position is the center position of the front of the actual marker M1 that is directly opposite to the object 501.

[0089] The front direction of the object 501 is parallel to the ground and perpendicular to the main surface. The center position O1 is a position at a reference height h from the ground and is the center position of the object 501 in the horizontal direction. Figure 3 In the example of FIG. 5 , the object 501 is a television set with its internal circuit board exposed. The reference distance D1 is an arbitrary distance such as 50 cm or 1 m. The operator 504 performs work such as assembling components on the object 501 .

[0090] In the real space 600, the relative position of the real marker M1 with respect to the object 501 is predetermined. Relative position information indicating the relative position is stored in advance in the memory 22. The relative position information includes, for example, a reference distance D1.

[0091] The initial marking information is the marking information detected by the detection device 1 when performing the spatial calibration. The spatial calibration is performed before the operator 504 performs the work. In the spatial calibration, Figure 3 As shown, the detection device 1 detects the position and posture of the actual marker M1 set at a given relative position relative to the object 501, and outputs the marker information indicating the detected position and posture to the operator terminal 2. When performing the spatial calibration, the first acquisition unit 212 acquires the marker information output from the detection device 1 as the initial marker information.

[0092] The execution unit 213 executes space calibration to associate the real space 600 with the virtual space based on the initial marking information and the relative position information stored in the memory 22. The details of the space calibration are as follows.

[0093] Figure 4 : is a diagram showing a situation of space calibration. First, the execution unit 213 sets a virtual marker M2 corresponding to the actual marker M1 in the virtual space 700 based on the initial marker information. The virtual marker M2 is a three-dimensional model obtained by simulating the actual marker M1. The virtual space 700 is a three-dimensional space represented by three axes, namely, the X-axis, the Y-axis, and the Z-axis, which are orthogonal to each other. The Z-axis is orthogonal to a reference plane 701 equivalent to the ground, and the X-axis and the Y-axis are parallel to the reference plane 701. For example, the execution unit 213 determines a position P1' on the virtual space 700 corresponding to the position P1 of the actual marker M1 included in the actual marker information, and sets the virtual marker M2 at the determined position P1'. The position P1' is, for example, the front position of the virtual marker M2. At this time, the execution unit 213 determines the posture D2' of the virtual marker M2 to be the same posture as the posture D2 of the actual marker M1 included in the actual marker information. As a result, the virtual marker M2 is set so that the posture D2' is parallel to the reference plane 701 in the virtual space 700. In addition, the coordinates of the position P1′ are determined at an arbitrary position in the coordinate system of the virtual space 700. An example of an arbitrary position is the origin of the coordinate system of the virtual space 700.

[0094] Next, the execution unit 213 sets the three-dimensional model 221 in the virtual space 700 based on the reference distance D1 included in the relative position information and with reference to the virtual marker M2 so as to maintain the relative positional relationship between the real marker M1 and the object 501 .

[0095] Figure 5 It is shown Figure 4A diagram showing a subsequent spatial calibration situation. The execution unit 213 sets the three-dimensional model 221 in the virtual space 700 so that the three-dimensional model 221 is located at a position separated from the virtual marker M2 by a reference distance D1´ in the direction of the posture D2´. Specifically, the execution unit 213 sets the three-dimensional model 221 so that the direction of the posture D2´ is parallel to the front direction of the three-dimensional model 221, the center position O1´ of the three-dimensional model 221 is located on a straight line extending from the position P1´ of the virtual marker M2 in the direction of the posture D2´, and the distance from the position P1´ of the virtual marker M2 to the center position O1´ is the reference distance D1´.

[0096] Thus, the virtual marker M2 and the three-dimensional model 221 are set in the virtual space 700 so as to maintain the positional relationship between the actual marker M1 and the object 501. In addition, the position P1' is the position of the virtual space 700 corresponding to the position P1, the reference distance D1' is the distance of the virtual space 700 corresponding to the reference distance D1, the posture D2' is the posture of the virtual space 700 corresponding to the posture D2, and the center position O1' is the position of the virtual space 700 corresponding to the center position O1.

[0097] Next, the execution unit 213 generates calibration information 222 and stores it in the memory 22. The calibration information 222 includes Figure 3 The actual reference vector V0 and Figure 5 The virtual reference vector V0' is shown. The actual reference vector V0 is a vector indicating the relative position of the actual marker M1 relative to the object 501 when performing spatial calibration. Specifically, the actual reference vector V0 is a vector starting at the position P1 and ending at the center position O1. The virtual reference vector V0' is a vector starting at the position P1' and ending at the center position O1' in the virtual space 700, and is a vector indicating the relative position of the virtual marker M2 relative to the three-dimensional model 221.

[0098] Return to reference Figure 1 . The second acquisition unit 214 acquires actual mark information indicating the position and posture of the actual mark M1 worn by the operator 504 in the actual space 600. The actual mark information is mark information indicating the position and posture of the actual mark M1 worn by the operator 504 obtained by the detection device 1 during the operation of the operator 504. The detection device 1 detects the position and posture of the actual mark M1 at a given sampling rate during the operation, and outputs the mark information indicating the detected position and posture to the operator terminal 2 at a given sampling rate. The second acquisition unit 214 acquires the mark information output from the detection device 1 during the operation of the operator 504 as the actual mark information. During the operation, the operator 504 wears the actual mark M1 on the head, for example. However, this is an example, and the actual mark M1 may also be worn at a position other than the head of the operator 504.

[0099] Based on the actual marker information, the synchronization unit 215 synchronizes the position and posture of the virtual camera in the virtual space 700 with the position and posture of the actual marker M1 indicated by the actual marker information.

[0100] Reference Figure 3 , assuming that the position of the actual mark M1 is at position P2 during the operation. In this case, the synchronization unit 215 calculates the vector from the position P2 of the actual mark M1 included in the actual mark information to the center position O1 as the actual space vector V1. The actual space vector V1 is a vector indicating the relative position of the actual mark M1 with respect to the object 501 during the operation. Next, the synchronization unit 215 calculates the difference vector ΔV between the actual space vector V1 and the actual reference vector V0.

[0101] Next, refer to Figure 5 , the synchronization unit 215 adds the difference vector ΔV to the virtual reference vector V0´ in the virtual space 700 to calculate the virtual space vector V1´. The virtual space vector V1´ is a vector that represents the relative position of the virtual marker M2 with respect to the three-dimensional model 221 during the operation. Thus, the relative position of the operator in the operation in the actual space 600 with respect to the object 501 is reproduced in the virtual space 700. Next, the synchronization unit 215 determines the starting point (position P2´) in the virtual space vector V1´ as the position of the virtual camera. Next, the synchronization unit 215 determines the posture of the actual marker M1 included in the actual marker information as the posture of the virtual camera.

[0102] Return to reference Figure 1 . The generation unit 216 generates a virtual camera image representing the three-dimensional model 221 captured by a virtual camera synchronized with the actual marker M1, and generates output information for displaying an output image, which includes the generated virtual camera image. For example, the generation unit 216 generates a virtual camera image by setting a virtual screen in the virtual space 700 and rendering the three-dimensional model 221 on the virtual screen. The virtual screen is, for example, a plane separated from the virtual camera by a given distance and the center of which passes through the line of sight of the virtual camera. The size of the virtual screen has, for example, a size equivalent to the field of view of the operator 504. Thus, a virtual camera image that reproduces the image within the field of view of the operator 504 is generated. The generation unit 216 generates the virtual camera image at a given sampling rate. Thus, the virtual camera image can reproduce the image of the field of view of the operator 504 in real time.

[0103] The output unit 217 transmits the output information generated by the generation unit 216 to the administrator terminal 3 using the communication unit 24 .

[0104] The memory 22 is a non-volatile rewritable storage device such as a flash memory or a solid state drive, for example. The memory 22 stores a three-dimensional model 221 and calibration information 222 .

[0105] The display 23 is, for example, a liquid crystal display device, etc. The communication unit 24 is a communication circuit for connecting the operator terminal 2 to the network. The communication unit 24 transmits output information to the manager terminal 3 via the network.

[0106] The administrator terminal 3 is, for example, a terminal installed at a remote location. The administrator terminal 3 is, for example, composed of a computer including a communication unit, a processor, a display, and an operation unit. The administrator terminal 3 may be composed of a desktop computer or a portable computer such as a smartphone or a tablet computer. The processor of the administrator terminal 3 causes the display to display an output image drawn based on the output information sent from the operator terminal 2.

[0107] Figure 6 3 is a diagram showing an example of a virtual camera image 900 included in an output image. The virtual camera image 900 is displayed on the display of the administrator terminal 3. The virtual camera image 900 includes an image of the three-dimensional model 221. The image of the three-dimensional model 221 is an image obtained by depicting the image of the object 501 located in the visual field of the operator 504 as viewed from the line of sight of the operator 504 using the three-dimensional model 221. Thus, the administrator 514 can confirm the image of the object 501 visually recognized by the operator 504 and can smoothly give instructions to the operator 504.

[0108] In addition, Figure 6 , the part of the circuit substrate may be the movable part 901 of the three-dimensional model 221. In this case, when the administrator 514 inputs operation information indicating an operation on the movable part 901 into the administrator terminal 3, the administrator terminal 3 sends the operation information to the operator terminal 2. The processor 21 of the operator terminal 2 that receives the operation information moves the movable part 901 based on the operation information. Specifically, the generation unit 216 of the processor 21 moves the movable part 901 according to the operation amount indicated by the operation information. Since the generation unit 216 renders the three-dimensional model 221 at a given sampling rate, the virtual camera image 900 that represents the movement of the movable part 901 in real time is displayed on the administrator terminal 3. As the movement of the movable part 901, for example, the movement of removing the movable part 901 from the main body of the three-dimensional model 221 can be listed.

[0109] The above is the configuration of the information processing system 100. Next, the processing of the information processing system 100 will be described. Figure 7 1 is a flowchart showing the overall situation of the processing of the information processing system 100 .

[0110] The construction unit 211 of the operator terminal 2 constructs the three-dimensional model 221 (step S1 ). Next, the construction unit 211 constructs the virtual space 700 having a coordinate system of three axes, namely, the X axis, the Y axis, and the Z axis, in the computer (step S2 ).

[0111] Next, the detection device 1 and the operator terminal 2 perform space calibration to establish correspondence between the actual space 600 and the virtual space 700 (step S3). Next, from the start to the end of the work performed by the operator 504 at the site, the information processing system 100 performs stabilization processing, which generates a virtual camera image 900, and causes the manager terminal 3 to display the generated virtual camera image 900 in real time (step S4).

[0112] Figure 8 is a flowchart showing the details of the processing of space calibration. In step S11, in the real space 600, the operator 504 sets the real mark M1. In this case, Figure 3 As shown, the actual mark M1 is set at a position separated from the object 501 by a reference distance D1 and at a reference height h from the ground.

[0113] In this setting, for example, in a state where the object 501 is not set, the operator 504 sets an optical tracker at a position corresponding to the center position O1. And, for example, the operator 504 positions the actual mark M1 while confirming the measured value of the optical tracker so that the actual mark M1 is set at a position separated from the center position O1 by a reference distance D1 and a reference height h. Alternatively, the operator 504 may also position the actual mark M1 while confirming the measured value of the depth sensor set in the actual space 600 so that the actual mark M1 is set at a position separated from the center position O1 by a reference distance D1 and a reference height h.

[0114] Next, in step S12, the detection device 1 detects the position P1 and the posture D2 of the actual mark M1 to detect the initial mark information. Next, in step S13, the detection device 1 outputs the initial mark information to the operator terminal 2 via the communication path C1.

[0115] Next, in step S21 , the first acquisition unit 212 of the operator terminal 2 acquires the initial marking information using the communication unit 24 .

[0116] Next, in step S22, the execution unit 213 determines a position P1' on the virtual space 700 corresponding to the position P1 of the actual marker M1 included in the initial marker information. In the first embodiment, the position P1' is an arbitrary position (eg, the origin) on the virtual space 700 and is not particularly limited.

[0117] Next, in step S23 , the execution unit 213 determines a posture D2 ′ in the virtual space 700 corresponding to the posture D2 of the actual marker M1 included in the initial marking information.

[0118] Next, in step S24, the execution unit 213 sets the virtual marker M2 in the virtual space 700 so that the virtual marker M2 takes the posture D2′ determined in step S23 at the position P1′ determined in step S22. Figure 4 As shown, a virtual marker M2 is set in the virtual space 700 .

[0119] Next, in step S25 , the execution unit 213 determines a position separated from the virtual marker M2 by the reference distance D1 in the direction of the posture D2 ′, and sets the three-dimensional model 221 so that the center position O1 ′ is located at the determined position.

[0120] Next, in step S26, the execution unit 213 calculates Figure 3 The actual reference vector V0 shown in Figure 1 is calculated Figure 5 The virtual reference vector V0′ shown in FIG. 1 generates calibration information 222 including the calculated actual reference vector V0 and the virtual reference vector V0′. The generated calibration information 222 is stored in the memory 22.

[0121] The spatial calibration is completed as described above. Next, the stabilization process will be described. Fig. 9 is a flowchart showing the details of the stabilization process. For example, the operator 504 inputs a command to notify the start of the operation to the operator terminal 2, and the operator terminal 2 obtains the input command, thereby starting the stabilization process. Alternatively, for example, the manager 514 inputs a command to notify the start of the operation to the manager terminal 3, and the operator terminal 2 obtains the input command via the manager terminal 3, thereby starting the stabilization process.

[0122] In step S41 , the detection device 1 detects the position P1 and the posture D2 of the actual marker M1 worn by the worker 504 to detect the actual marker information.

[0123] Next, in step S42 , the detection device 1 outputs the actual mark information to the operator terminal 2 .

[0124] Next, in step S51 , the second acquisition unit 414 of the operator terminal 2 acquires actual mark information.

[0125] Next, in step S52, the synchronization unit 215 performs synchronization processing. The details of the synchronization processing will be described later.

[0126] Next, in step S53 , the generation unit 216 generates a virtual camera image 900 representing the three-dimensional model 221 captured by the virtual camera synchronized with the actual marker M1 .

[0127] Next, in step S54 , the output unit 217 generates output information for displaying the virtual camera image 900 , and transmits the generated output information to the administrator terminal 3 using the communication unit 24 .

[0128] Next, in step S61, the manager terminal 3 receives the output information. In step S62, the manager terminal 3 displays an output image representing the output information.

[0129] The above process is repeatedly executed at a given sampling rate until the operator's work is completed, and the virtual camera image 900 is displayed in real time on the manager terminal 3. In addition, the operator 504 inputs a command to notify the end of the work to the operator terminal 2, and the operator terminal 2 obtains the command, thereby completing the stabilization process. Alternatively, the manager 514 inputs a command to notify the end of the work to the manager terminal 3, and the operator terminal 2 obtains the command via the manager terminal 3, thereby completing the stabilization process.

[0130] Next, the details of the synchronization process shown in step S52 will be described. Fig.10 is a flowchart showing the details of the synchronization process.

[0131] In step S521 , the synchronization unit 215 calculates a vector from the position P2 of the actual marker M1 included in the actual marker information to the center position O1 in the actual space 600 as the actual space vector V1 .

[0132] Next, in step S522 , the synchronization unit 215 calculates a difference vector ΔV between the actual space vector V1 and the actual reference vector V0 .

[0133] Next, in step S523 , the synchronization unit 215 adds the difference vector ΔV to the virtual reference vector V0 ′ in the virtual space 700 to calculate a virtual space vector V1 ′.

[0134] Next, in step S524, the synchronization unit 215 determines the starting point (position P2') in the virtual space vector V1' as the position of the virtual camera, and determines the posture of the actual marker M1 included in the actual marker information as the posture of the virtual camera. Thus, the position and posture of the virtual camera are synchronized with the position and posture of the actual marker M1 indicated by the actual marker information.

[0135] Next, a modification of the output image will be described. The output image may further include a bird's-eye view image of the three-dimensional model 221 . Fig.111 is a diagram showing an example of an overhead image 1000 .

[0136] The bird's-eye view image 1000 is an image obtained by rendering the virtual space 700 using a virtual camera, namely, a bird's-eye view virtual camera, which is different from the virtual camera that renders the virtual camera image 900. The bird's-eye view image 1000 includes the virtual camera image 900, a camera icon 1002, and a three-dimensional model 221.

[0137] The camera icon 1002 is an icon indicating the position and posture of the virtual camera that renders the virtual camera image 900. The position and posture of the camera icon 1002 change in synchronization with the position and posture of the virtual camera. The overhead virtual camera can also change its position and posture according to the instructions of the administrator. In this case, the display method of the three-dimensional model 221 in the overhead image 1000 changes according to the position and posture of the overhead virtual camera. In this way, it is possible to provide the administrator with information indicating which part of the object 501 the operator 504 is observing from an overhead view.

[0138] Furthermore, the bird's-eye view image 1000 may also be displayed on the operator terminal 2 . Fig.12 1 is a diagram showing an overhead image 1000 displayed on the manager terminal 3 and the operator terminal 2. Fig.12 In the example, the bird's-eye view image 1000 further includes a display object 1004 .

[0139] The display object 1004 is a display object indicating a position designated by the administrator in the bird's-eye view image 1000. The administrator 514 operates an operating device (not shown) to display the display object 1004 indicating the designated position on the bird's-eye view image 1000. Then, display object information for displaying the display object 1004 on the operator terminal 2 is transmitted from the administrator terminal 3 to the operator terminal 2. As a result, the operator terminal 2 displays the display object 1004 on the bird's-eye view image 1000. Fig.12 In the example of FIG. 1 , the display object 1004 is displayed at a display position of the bird's-eye view image 1000 displayed on the operator terminal 2 corresponding to the bird's-eye view image 1000 displayed on the manager terminal 3. Thus, the manager can share the designated position with the operator.

[0140] Thus, according to Embodiment 1, the manager 514 can check the virtual camera image 900 which is the image of the three-dimensional model 221 observed in the line of sight of the operator 504. As a result, the remote manager 514 can check the place observed by the operator 504 without outputting the image of the site to the outside.

[0141] Furthermore, according to Embodiment 1, initial marker information indicating the position and posture of an actual marker set at a given relative position relative to the object 501 is obtained, and based on the initial marker information and the relative position, spatial calibration is performed to establish correspondence between the actual space 600 and the virtual space 700. Therefore, the actual space 600 and the virtual space 700 can be accurately associated with each other, thereby generating a virtual camera image 900 that accurately reproduces the place observed by the operator 504 without using the actual image of the scene.

[0142] Furthermore, according to Embodiment 1, the position of the actual marker is set at an arbitrary position in the virtual space 700, and the three-dimensional model is set in the virtual space with the virtual marker as a reference, so that the relative positional relationship between the actual marker and the object is maintained. Therefore, the coordinate system of the virtual space can be set so that the position of the virtual marker is substantially the origin, and the position of the object can be determined using the coordinate system. Therefore, in an environment using a sensor that indicates the position of the actual marker by its relative position to itself, the actual marker and the virtual marker can be accurately established.

[0143] (Variation 1 of Embodiment 1)

[0144] Variation 1 of Embodiment 1 is different from Embodiment 1 in terms of spatial calibration and synchronization processing. In Variation 1 of Embodiment 1, detection device 1 is configured by a sensor that uses a three-dimensional coordinate system to represent the position of an object in real space. An example of such a sensor is an optical sensor such as an optical tracker. Figure 3 As shown in FIG. 1 , the three-dimensional coordinate system is represented by three axes, namely, the x-axis, the y-axis, and the z-axis, which are orthogonal to each other. Figure 3 In the example, the z-axis is perpendicular to the ground, and the y-axis and the x-axis are parallel to the ground. Therefore, the actual position of the marker M1 is represented by the coordinate data of x, y, and z. The actual posture of the marker M1 is represented by a three-dimensional vector.

[0145] Fig.13 FIG. 4 is a flowchart showing the processing of spatial calibration in Modification 1 of Implementation Example 1. Fig.13 In, with Figure 8 The different point is at step S26A.

[0146] In step S26A following step S25 , the execution unit 213 of the operator terminal 2 generates the coordinates of the three-dimensional model 221 in the virtual space 700 as calibration information 222 , and causes the memory 22 to store the generated calibration information 222 .

[0147] like Figure 8As described above, in the process of step S25, the coordinates of the three-dimensional model 221 in the virtual space 700 are adjusted so that the distance between the virtual marker M2 in the virtual space 700 and the three-dimensional model 221 is equal to the distance between the actual marker M1 in the real space 600 and the object 501. As a result, the coordinate system of the real space 600 coincides with the coordinate system of the virtual space 700. However, in order to generate a virtual camera image in the operation, the coordinates of the virtual marker M2 are required. For this purpose, in Modification 1, the memory 22 stores the coordinates of the three-dimensional model 221 set in the virtual space 700 as calibration information.

[0148] Reference Figure 5 For example, in Figure 5 In FIG. 1 , the coordinates of the virtual marker M2 are the position P1′, and the coordinates of the three-dimensional model 221 are the center position O1′. In this case, referring to Figure 3 , the execution unit 213 adjusts the position of the center position O1' of the three-dimensional model 221 so that the distance from the actual mark M1 to the object 501 is equal to the distance from the virtual mark M2 to the center position O1' of the three-dimensional model 221. Thus, during the operation, the positional relationship between the actual mark M1 and the object 501 can be reproduced in the virtual space 700.

[0149] Next, the synchronization process in the first modification of the first embodiment will be described. Fig.14 : is a flowchart showing an example of synchronization processing in the modification example 1 of the embodiment 1. In step S531, the synchronization unit 215 performs synchronization based on the synchronization performed in step S51 ( Fig. 9 ) to determine the position and posture of the virtual camera. Figure 3 As shown, when the position of the actual mark M1 included in the actual mark information is the position P2, the coordinates of the position P2 are plotted in the virtual space 700, and the plotted coordinates are determined as the position P2' of the virtual mark M2. In addition, the posture of the actual mark M1 included in the actual mark information is determined as the posture of the virtual mark M2. Furthermore, the synchronization unit 215 determines the position P2' of the virtual mark M2 as the position of the virtual camera, and determines the posture of the virtual mark M2 as the posture of the virtual camera. Thus, the position and posture of the actual mark M1 are synchronized with the position and posture of the virtual camera. When step S531 is completed, the processing proceeds to step S53 ( Fig. 9 ).

[0150] Thus, according to variation 1 of implementation example 1, space calibration is performed so that the coordinate system in real space 600 is consistent with the coordinate system in virtual space 700, so the position and posture of virtual marker M2 can be set using the position and posture indicated by the actual marker information as is.

[0151] Furthermore, according to variant example 1 of implementation example 1, spatial calibration is performed so that the coordinate system in the actual space is consistent with the coordinate system in the virtual space. Therefore, in an environment where the position of the actual mark is detected using a sensor that represents the position of the object using the coordinate system of the actual space, the position and posture of the virtual mark can be set while maintaining the position and posture represented by the actual mark information.

[0152] (Variation 2 of Embodiment 1)

[0153] Modification 2 of Embodiment 1 performs spatial calibration using a first actual marker and a second actual marker. Fig.15 This is a flowchart showing the processing of space calibration in Modification 2 of Implementation 1. Fig.16 , Fig.17 , Fig.18 This is an explanatory diagram of the processing in the second variation of the first embodiment.

[0154] In step S11B, if Fig.16 As shown, a first actual mark M11 and a second actual mark M12 are set in the actual space 600 by the operator 504. The first actual mark M11 is set at a position different from the object 501. The second actual mark M12 is set at the position of the object 501. Specifically, the second actual mark M12 is set near the object 501. The position of the second actual mark M12 is regarded as the position of the object 501.

[0155] Next, in step S12B, the detection device 1 detects the first initial marker information by detecting the position P11 and the posture D2 of the first actual marker M11 , and detects the second initial marker information by detecting the position P12 and the posture (not shown) of the second actual marker M12 .

[0156] Next, in step S13B, the detection device 1 outputs the first initial label information and the second initial label information to the operator terminal 2 via the communication path C1.

[0157] Next, in step S21B, the first acquisition unit 212 of the operator terminal 2 acquires the first initial marking information and the second initial marking information using the communication unit 24 .

[0158] Next, in step S22B, the execution unit 213 determines the position P11' on the virtual space 700 corresponding to the position P11 of the first actual mark M11 included in the first initial mark information, and determines the position P12' on the virtual space 700 corresponding to the position P12 of the second actual mark M12 included in the second initial mark information. For example, the execution unit 213 calculates the relative position of the second actual mark M12 with respect to the first actual mark M11 in the actual space 600, and determines the position of the second virtual mark M22 with respect to the first virtual mark M21 based on the relative position, so as to maintain the relative positional relationship between the first actual mark M11 and the second actual mark M12. In this case, the execution unit 213 calculates the vector with the position P11 as the starting point and the position P12 as the end point as the relative position of the second actual mark M12 with respect to the first actual mark M11.

[0159] Next, in step S23B, the execution unit 213 determines a posture D2´ on the virtual space 700 corresponding to the posture D2 of the first actual marker M11 included in the first initial marking information, and determines a posture (not shown) on the virtual space 700 corresponding to the posture (not shown) of the second actual marker M12 included in the second initial marking information.

[0160] Next, in step S24B, the execution unit 213 sets the first virtual marker M21 in the virtual space 700 so that at the position P11′ determined in step S22B, the first virtual marker M21 takes the posture D2′ determined in step S23. Furthermore, the execution unit 213 sets the second virtual marker M22 in the virtual space 700 so that at the position P12′ determined in step S22B, the second virtual marker M22 takes the posture (not shown) determined in step S23B. Thus, Fig.17 As shown, a virtual marker M2 is set in the virtual space 700 .

[0161] Next, in step S25B, Fig.18 As shown, the execution unit 213 sets the three-dimensional model 221 at the position P12' of the second virtual marker M22. Specifically, the execution unit 213 sets the three-dimensional model 221 so that the given position of the three-dimensional model 221 is located at the position P12' and the main surface of the three-dimensional model 221 is orthogonal to the posture D2' of the first virtual marker M21. Thus, the three-dimensional model is set in the virtual space 700 so that the relative positional relationship between the first real marker M11 and the object 501 is maintained.

[0162] In the second modification, at the stage of step S25B, the coordinate system of the real space 600 coincides with the coordinate system of the virtual space 700, and further, the position of the object 501 is updated at any time based on the second real marker M12, so there is no need to store calibration information.

[0163] Synchronous processing in variant 2 of implementation example 1 and Fig.14 The synchronization process in the variant 1 of the embodiment 1 shown in FIG. 1 is the same, so the detailed description is omitted. Fig.16 The coordinates of the position P2 of the first actual mark M11 worn by the operator 504 are determined as follows: Fig.18 The position P2' of the first virtual marker M21 in the virtual space 700 is shown. Furthermore, the posture of the first real marker M11 is determined as the posture of the first virtual marker M21. Furthermore, the determined position P2' of the first virtual marker M21 is determined as the position of the virtual camera, and the determined posture of the first virtual marker M21 is determined as the posture of the virtual camera, and the virtual camera image is captured.

[0164] As described above, according to the second modification of the first embodiment, since the spatial calibration is performed using the first actual mark M11 and the second actual mark M12, the time and effort of generating calibration information can be omitted.

[0165] (Implementation method 2)

[0166] Fig.19 This is a block diagram showing an example of the configuration of an information processing system 100A in Embodiment 2. In Embodiment 1, the operator terminal 2 is the processing subject, but in Embodiment 2, the manager terminal 3A is the processing subject. The information processing system 100A includes a detection device 1, an operator terminal 2A, and a manager terminal 3A.

[0167] In the first embodiment, the processor 21 of the operator terminal 2 includes a construction unit 211, a first acquisition unit 212, an execution unit 213, a second acquisition unit 214, a synchronization unit 215, a generation unit 216, and an output unit 217, but in the second embodiment, the processor 301 of the manager terminal 3A includes these components. In addition, in the first embodiment, the memory 22 stores the three-dimensional model 221 and the calibration information 222, but in the second embodiment, the memory 302 includes these components.

[0168] The administrator terminal 3A includes a processor 301, a memory 302, a display 303, and a communication unit 304. The processor 301 is constituted by, for example, a central processing unit. The construction unit 211, the first acquisition unit 212, the execution unit 213, the second acquisition unit 214, the synchronization unit 215, the generation unit 216, and the output unit 217 are implemented by the processor 301 executing the information processing program. However, this is an example, and these structures may also be constituted by dedicated hardware circuits.

[0169] The display 303 is, for example, a liquid crystal display device, and displays the virtual camera image 900 .

[0170] The communication unit 304 is a communication circuit for connecting the manager terminal 3A to the network. The communication unit 304 receives the initial marking information and the actual marking information transmitted from the operator terminal 2A.

[0171] Next, the processing of the information processing system 100A in Embodiment 2 is described. In Embodiment 2, the overall situation of the processing of the information processing system 100A is the same as that of Figure 7 Same, so citing Figure 7 However, in the second embodiment, the manager terminal 3A performs the process of constructing a three-dimensional model (step S1) and the process of constructing a virtual space (step S2), and the detection device 1, the operator terminal 2A, and the manager terminal 3A perform the process of space calibration (step S3).

[0172] Fig. 20 This is a flowchart showing details of the processing of spatial calibration in the second embodiment.

[0173] The processing of steps S101, S102, S103, and S202 is Figure 8 Steps S11, S12, S13, and S21 shown are the same, so detailed descriptions are omitted.

[0174] In step S202 subsequent to step S201, the operator terminal 2A transmits the initial marking information to the manager terminal 3A. In step S301, the manager terminal 3A receives the initial marking information.

[0175] Next, the processing of steps S302 to S306 is performed. The details of the processing of steps S302, S303, S304, S305, and S306 are as follows: Figure 8 Steps S22, S23, S24, S25, and S26 are the same as those shown in FIG. 1 , and therefore detailed descriptions thereof are omitted. However, in steps S302 to S306, the processing subject is not the operator terminal 2 but the administrator terminal 3A, which is different from the Figure 8 different.

[0176] Fig.21 4 is a flowchart showing the details of the stabilization process in Embodiment 2. The processes of steps S401, S402, and S501 are similar to those of Fig. 9 Steps S41, S42, and S51 shown are the same, so detailed descriptions are omitted.

[0177] In step S502 following step S501, the operator terminal 2A transmits the actual mark information to the manager terminal 3A. In step S601, the communication unit 304 of the manager terminal 3A receives the actual mark information.

[0178] Next, the processing of steps S602 to S604 is performed. The details of the processing of steps S602, S603, and S604 are as follows: Fig. 9 The steps S52, S53, and S62 shown in FIG. 1 are the same. However, in steps S602 and S603, the processing subject is not the operator terminal 2 but the manager terminal 3A, which is different from Fig. 9 Different. In addition, Fig.21 In the example, since the virtual camera image 900 is generated by the administrator terminal 3A, the Fig. 9 The processing of sending and receiving output information shown in steps S54 and S61.

[0179] As described above, according to the second embodiment, even when the processing subject is the manager terminal 3A, the same effects as those of the first embodiment can be obtained.

[0180] (Implementation method 3)

[0181] In the first embodiment, the operator terminal 2 is the processing subject, but in the third embodiment, the server is the processing subject. Fig. 22 1 is a block diagram showing an example of the configuration of an information processing system 100B in Embodiment 3. The information processing system 100B includes a detection device 1 , an operator terminal 2B, an administrator terminal 3B, and a server 4 .

[0182] The server 4 is, for example, a cloud server including one or more computers. In the first embodiment, the processor 21 of the operator terminal 2 includes a construction unit 211, a first acquisition unit 212, an execution unit 213, a second acquisition unit 214, a synchronization unit 215, a generation unit 216, and an output unit 217, but in the third embodiment, the processor 401 of the server 4 includes these structures. In addition, in the first embodiment, the memory 22 stores the three-dimensional model 221 and the calibration information 222, but in the third embodiment, the memory 402 has these structures.

[0183] The server 4 includes a processor 401, a memory 402, and a communication unit 403. The processor 401 is constituted by, for example, a central processing unit. The construction unit 211, the first acquisition unit 212, the execution unit 213, the second acquisition unit 214, the synchronization unit 215, the generation unit 216, and the output unit 217 are implemented by the processor 401 executing the information processing program. However, this is an example, and these structures may also be constituted by dedicated hardware circuits.

[0184] The communication unit 403 is a communication circuit for connecting the operator terminal 2 to the network. The communication unit 403 receives the initial marking information and the actual marking information transmitted from the operator terminal 2B.

[0185] Next, the processing of the information processing system 100B in the third embodiment is described. In the third embodiment, the overall processing of the information processing system 100B is similar to Figure 7 Same, so citing Figure 7 However, in the third embodiment, the server 4 performs the process of constructing the three-dimensional model (step S1) and the process of constructing the virtual space (step S2), and the detection device 1, the operator terminal 2B and the server 4 perform the process of space calibration (step S3) and the stabilization process (step S4).

[0186] Fig.23 This is a flowchart showing details of the processing of spatial calibration in the third embodiment.

[0187] The processing of steps S111, S112, S113, and S211 is Figure 8 Steps S11, S12, S13, and S21 shown are the same, so detailed descriptions are omitted.

[0188] In step S212 subsequent to step S211, the operator terminal 2B transmits the initial marking information to the server 4. In step S311, the server 4 receives the initial marking information.

[0189] Next, the processing of steps S312 to S316 is performed. The details of the processing of steps S312, S313, S314, S315, and S316 are as follows: Figure 8 Steps S22, S23, S24, S25, and S26 are the same as those shown in FIG. 1 , and therefore detailed descriptions thereof are omitted. However, in steps S312 to S316, the processing subject is not the operator terminal 2 but the server 4, which is different from the Figure 8 different.

[0190] Fig.24 is a flowchart showing the details of the stabilization process in Embodiment 3. The processes of Steps S411, S412, and S511 are similar to those of Steps S411, S412, and S511. Fig. 9Steps S41, S42, and S51 shown are the same, so detailed descriptions are omitted.

[0191] In step S512 subsequent to step S511, the operator terminal 2B transmits the actual mark information to the server 4. In step S701, the first acquisition unit 212 of the server 4 acquires the actual mark information using the communication unit 403.

[0192] Next, the processing of steps S702 to S704 is performed. The details of the processing of steps S702 and S703 are as follows: Fig. 9 The steps S52 and S53 shown are the same.

[0193] In step S704 subsequent to step S703 , the output unit 217 of the server 4 transmits output information including the virtual camera image to the manager terminal 3B.

[0194] Next, in step S611, the manager terminal 3B receives the output information. Next, in step S612, the manager terminal 3B displays the output information.

[0195] As described above, according to the third embodiment, even when the processing subject is set to the server 4, the same effect as that of the first embodiment can be obtained.

[0196] The present disclosure can adopt the following modified examples.

[0197] (1) In Embodiment 2 and Embodiment 3, the spatial calibration process may also be performed in accordance with Embodiment 1. Fig.13 The processing of the modified example of embodiment 1 is shown.

[0198] (2) In the above-mentioned Embodiments 1 to 3, real markers and virtual markers are used as an example of a method of detecting the position and posture of the user, but the present disclosure is not limited thereto. For example, in the present disclosure, a method of directly sensing the position and posture of the user using the detection device 1 may be adopted.

[0199] Industrial Applicability

[0200] According to the present disclosure, the present invention is useful in a remote assistance system for assisting a worker's work from a remote location.

Claims

1. An information processing method is an information processing method in an information processing system for assisting an operator's work, wherein: constructing in the virtual space a three-dimensional model of an object that the operator works on in the real space, acquiring position information and posture information of the operator in the actual space, Based on the position information and the posture information, the position and posture of the virtual camera in the virtual space are synchronized with the position and posture of the operator. generating a virtual camera image, wherein the virtual camera image represents the three-dimensional model captured by the synchronized virtual camera, The output is used to display output information of an output image including the virtual camera image.

2. The information processing method according to claim 1, wherein: The position information and the posture information are actual marker information indicating the position and posture of an actual marker worn by the operator. In the information processing method, Acquire initial marking information, wherein the initial marking information indicates the position and posture of an actual marker set in the actual space, Based on the initial marking information and the relative positions of the object and the actual marking, performing space calibration to establish correspondence between the actual space and the virtual space, acquiring actual marker information indicating a position and a posture of the actual marker worn by the operator in the actual space, Based on the actual marker information, the position and posture of the virtual camera are synchronized with the position and posture of the actual marker indicated by the actual marker information.

3. The information processing method according to claim 2, wherein: In the spatial calibration, Based on the initial marking information, a virtual marker corresponding to the actual marker is set at an arbitrary position in the virtual space, Based on the relative position, the three-dimensional model is set in the virtual space with the virtual marker as a reference so that the relative positional relationship between the actual marker and the object is maintained. Calibration information is generated, the calibration information including an actual reference vector indicating a relative position of the actual marker with respect to the object when the spatial calibration is performed, and a virtual reference vector indicating a relative position of the virtual marker with respect to the three-dimensional model.

4. The information processing method according to claim 3, wherein: In the synchronization, Based on the actual mark information, a real space vector representing the relative position of the actual mark with respect to the object is calculated, Calculate the difference vector between the actual space vector and the actual reference vector, determining the position of the virtual marker relative to the three-dimensional model based on the virtual reference vector and the difference vector, and determining the posture of the virtual marker based on the posture of the actual marker included in the actual marker information, The position of the virtual marker is determined as the position of the virtual camera, The posture of the virtual marker is set as the posture of the virtual camera.

5. The information processing method according to claim 3, wherein: The initial marking information and the actual marking information are information obtained by sensing the surrounding environment to detect the relative position of the device.

6. The information processing method according to claim 2, wherein: In the spatial calibration, Based on the initial marking information, a virtual marker corresponding to the actual marker is set in the virtual space, Based on the relative position, the three-dimensional model is set in the virtual space so that the relative position relationship between the actual mark and the object is maintained, The coordinates of the set three-dimensional model are generated as calibration information.

7. The information processing method according to claim 6, wherein: In the synchronization, The position of the actual mark is determined as the position of the virtual camera, The posture of the actual marker is determined as the posture of the virtual camera.

8. The information processing method according to claim 6, wherein: The initial marking information and the actual marking information are information acquired using a sensor that indicates a position of an object using a coordinate system of the actual space.

9. The information processing method according to claim 2, wherein: The initial marking information includes first initial marking information indicating the position and posture of a first actual marking set at a position different from the object, and second initial marking information indicating the position and posture of a second actual marking set at the position of the object. In the spatial calibration, Based on the first initial marker information, a first virtual marker corresponding to the first real marker is set in the virtual space, Based on the second initial marker information, a second virtual marker corresponding to the second real marker is set in the virtual space so that the relative positional relationship between the first real marker and the second real marker is maintained. The three-dimensional model is set at the position of the second actual mark.

10. The information processing method according to claim 9, wherein: In the synchronization, The position of the first actual marker is determined as the position of the virtual camera, The posture of the first actual marker is determined as the posture of the virtual camera.

11. The information processing method according to claim 1, wherein: Furthermore, the virtual camera image changes synchronously with changes in the position and posture of the virtual camera.

12. The information processing method according to claim 1, wherein: Furthermore, the output image includes a bird's-eye view image of the three-dimensional model and a camera icon displayed in correspondence with the bird's-eye view image. The display mode of the camera icon changes synchronously with the position and posture of the virtual camera.

13. The information processing method according to claim 12, wherein: The output image is displayed on the operator terminal of the operator and the manager terminal of the manager, In the information processing method, Further, display object information is obtained from the administrator terminal, wherein the display object information is used to display a display object indicating a position specified by the administrator in the three-dimensional model shown in the bird's-eye view image or the virtual camera image. Furthermore, based on the display object information, the display object is displayed overlappingly on the bird's-eye view image or the virtual camera image.

14. The information processing method according to claim 1, wherein: The three-dimensional model includes a main body and a movable part installed on the main body. In the information processing method, Furthermore, operation information is acquired from the administrator terminal, wherein the operation information indicates an operation from the administrator on the movable part. The image representing the movable part included in the virtual camera image changes based on the operation information.

15. The information processing method according to claim 2, wherein: The initial marker information and the actual marker information are information acquired from a sensor capable of detecting the position and posture of the actual marker.

16. An information processing device for assisting an operator in performing a task. The information processing device comprises: A construction unit that constructs, in a virtual space, a three-dimensional model of an object that the operator works on in a real space; A first acquisition unit that acquires position information and posture information of the operator in the actual space; a synchronization unit that synchronizes the position and posture of the virtual camera in the virtual space with the position and posture of the operator based on the position information and the posture information; A generating unit, generating a virtual camera image, wherein the virtual camera image represents the three-dimensional model photographed by the synchronized virtual camera; and An output unit outputs output information for displaying an output image including the virtual camera image.

17. An information processing program for causing a computer to function as an information processing system for assisting an operator in performing work. The information processing program causes the computer to execute the following processing: constructing in the virtual space a three-dimensional model of an object that the operator works on in the real space, acquiring position information and posture information of the operator in the actual space, Based on the position information and the posture information, the position and posture of the virtual camera in the virtual space are synchronized with the position and posture of the operator. generating a virtual camera image, wherein the virtual camera image represents the three-dimensional model captured by the synchronized virtual camera, The output is used to display output information of an output image including the virtual camera image.

Images