Information processing method, information processing device, and information processing program

Abstract

The information processing apparatus acquires a three-dimensional target in which an actual object including a dynamic region is reproduced in a virtual space, specifies a first dynamic region indicating the dynamic region in the three-dimensional target, acquires an actual image of the actual object captured by an imaging apparatus, detects a second dynamic region indicating the dynamic region from the actual image, and transmits the detected second dynamic region to the virtual space. An image of the second dynamic region is embedded in real time as a texture image of the first dynamic region, and a display image of the three-dimensional target in which the image of the second dynamic region is embedded is output.

Description

Technical Field

[0001] The present disclosure relates to a technology for updating a three-dimensional object. Background Art

[0002] Patent Document 1 discloses a technology in which a 3D model generation device generates a 3D model of a subject by visual volume intersection based on camera images of a close camera and a distant camera, and generates a 3D model by assuming that the subject exists outside the viewing angle range of the close camera.

[0003] However, Patent Document 1 does not consider the case where a three-dimensional object includes a dynamic region, and therefore cannot update the dynamic region of the three-dimensional object in real time.

[0004] Prior Art Literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2022-29730 Summary of the invention

[0007] The present disclosure has been made to solve such a problem, and an object of the present disclosure is to provide a technology capable of updating a dynamic region included in a three-dimensional object in real time.

[0008] An information processing method in one embodiment of the present invention is an information processing method in a computer, the method comprising: obtaining a three-dimensional target obtained by reproducing an actual object including a dynamic area in a virtual space; determining a first dynamic area representing the dynamic area in the three-dimensional target; obtaining an actual image of the actual object captured by a camera; detecting a second dynamic area representing the dynamic area from the actual image; embedding the image of the second dynamic area as a texture image of the first dynamic area in real time; and outputting a display image of the three-dimensional target in which the image of the second dynamic area is embedded.

[0009] According to the present disclosure, a dynamic area included in a three-dimensional object can be updated in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is an overall configuration diagram of the information processing system in Implementation Example 1.

[0011] Figure 2 This is a diagram showing an example of a target device.

[0012] Figure 3 This is a flowchart showing an example of a process of generating a three-dimensional object in Embodiment 1 of the present disclosure.

[0013] Figure 4This is a flowchart showing an example of the update process in the first embodiment.

[0014] Figure 5 It is shown Figure 4 A flowchart showing details of the detection process of the second dynamic area shown in step S13.

[0015] Figure 6 It is shown Figure 4 This is a flowchart showing details of the updating process of the first dynamic area shown in step S14.

[0016] Figure 7 This is a diagram showing an example of an initial texture image including a second dynamic area captured when a three-dimensional object is generated.

[0017] Figure 8 FIG. 1 is a diagram showing an example of an actual image captured by a camera and including a second dynamic range.

[0018] Fig. 9 This is a diagram showing how the image of the second dynamic area embedded in the first dynamic area is embedded.

[0019] Fig.10 This is a flowchart showing an example of a process for generating a three-dimensional object in the second embodiment.

[0020] Fig.11 This is a flowchart showing details of the update process of the first dynamic area in the second embodiment.

[0021] Fig.12 is a diagram showing an example of an initial texture image.

[0022] Fig.13 This is a diagram showing how the image of the second dynamic range is embedded in the first dynamic range in the second embodiment. DETAILED DESCRIPTION

[0023] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the following embodiment is an example of embodying the present invention and does not limit the technical scope of the present invention.

[0024] (Knowledge that forms the basis of this disclosure)

[0025] When an operator is working on an object at a work site, the operator sometimes performs the work while confirming instructions from a remote operator outside the work site. In this case, if the remote operator can confirm which part of the object the operator is observing, the remote operator can smoothly instruct the operator. To achieve this, for example, if a shooting device such as a sports camera or smart glasses is worn on the operator's head, and the image of the work site captured by the shooting device is sent in real time to a remote terminal owned by the remote operator, the remote operator can confirm which part of the object the operator is observing.

[0026] However, there are work sites where it is prohibited to output images to the outside from the viewpoint of safety. In this case, there is a problem that the remote operator cannot confirm the part that the operator is observing.

[0027] To this end, the inventors have studied the following technology: by synchronizing the position and posture of the on-site operator with the position and posture of the virtual camera, a virtual camera image is generated that reproduces the field of view of the on-site operator in a virtual space, and the generated virtual camera image is displayed on the remote terminal of the remote operator.

[0028] The object sometimes includes object elements whose status changes dynamically, such as on a monitor. Hereinafter, the area representing the object elements is referred to as a dynamic area. When the object is reproduced in a three-dimensional virtual space, the remote operator can more accurately grasp the situation on site by updating the dynamic area in real time. A three-dimensional object is generated by embedding an image representing the surface of the object in a three-dimensional model representing the three-dimensional shape of the object.

[0029] To this end, the inventors have come to the following realization, which has led to the present disclosure: if the dynamic area is determined based on a three-dimensional target obtained by reproducing an actual object in a virtual space, and the dynamic area is detected from an actual image of the object captured by a camera in real time, and the image of the dynamic area detected from the actual image is embedded in the dynamic area of ​​the three-dimensional target, then a three-dimensional target that reproduces the dynamic area in real time can be obtained.

[0030] (1) An information processing method in one aspect of the present disclosure is an information processing method in a computer, the information processing method comprising: obtaining a three-dimensional target obtained by reproducing an actual object including a dynamic area in a virtual space; determining a first dynamic area representing the dynamic area in the three-dimensional target; obtaining an actual image of the actual object captured by a camera; detecting a second dynamic area representing the dynamic area from the actual image; embedding the image of the second dynamic area as a texture image of the first dynamic area in real time; and outputting a display image of the three-dimensional target in which the image of the second dynamic area is embedded.

[0031] According to this configuration, the image of the second dynamic area detected from the actual image captured by the imaging device is embedded in the first dynamic area of ​​the three-dimensional object in real time, so that the dynamic area included in the three-dimensional object can be updated in real time.

[0032] (2) In the information processing method described in (1) above, it may be that a mark representing the dynamic area is attached to the actual object, and the determination of the first dynamic area includes: detecting the first dynamic area based on the mark in the three-dimensional target, and the detection of the second dynamic area includes: detecting the second dynamic area based on an image representing the mark contained in the actual image.

[0033] According to this configuration, since the marker added to the actual object is added, the first dynamic area and the second dynamic area can be accurately detected based on the marker.

[0034] (3) In the information processing method described in (1) above, it may also be that the dynamic area corresponds to the object elements constituting the actual object, and the determination of the first dynamic area includes: detecting the area of ​​the object elements from the three-dimensional target through object recognition processing, and the detection of the second dynamic area includes: detecting the area of ​​the object elements from the actual image through the object recognition processing.

[0035] According to this configuration, since the first dynamic region and the second dynamic region are detected by the object recognition process, the first dynamic region and the second dynamic region can be accurately detected.

[0036] (4) In the information processing method described in any one of (1) to (3) above, the actual image may be repeatedly acquired, and the detection of the second dynamic area, the embedding, and the output may be performed each time the actual image is acquired.

[0037] According to this configuration, every time an actual image is acquired, the first dynamic range can be updated with the image of the second dynamic range.

[0038] (5) In the information processing method described in any one of (1) to (4) above, embedding the image of the second dynamic area may also include: obtaining distortion parameters of the shooting device; using the distortion parameters to correct the distortion of the image of the second dynamic area; and embedding the corrected image of the second dynamic area into the first dynamic area.

[0039] According to this configuration, since the distortion-corrected image of the second dynamic range is embedded in the first dynamic range, the image of the second dynamic range can be embedded in the first dynamic range without any sense of incongruity even when the actual image is captured at an arbitrary position and posture.

[0040] (6) In the information processing method described in (5) above, the distortion parameter may be estimated by camera calibration, wherein the camera calibration uses the current position and posture information of the shooting device, the initial position and posture information of the shooting device when shooting the initial texture image embedded in the first dynamic area, the two-dimensional position information of the dynamic area in the initial texture image, the two-dimensional position information of the second dynamic area in the actual image, and the three-dimensional position information of the first dynamic area.

[0041] According to this configuration, since the distortion parameter is estimated using the camera calibration, the distortion of the image in the second dynamic range can be accurately corrected using the distortion parameter.

[0042] (7) In the information processing method described in (6) above, current position and posture information of the imaging device may be estimated by applying a self-position estimation algorithm to the initial texture image and the actual image.

[0043] According to this configuration, the current position and posture information of the imaging device can be calculated without using an external sensor.

[0044] (8) In the information processing method described in any one of (1) to (4) above, embedding the image of the second dynamic area may include: editing the image of the second dynamic area so as to match the shape of the first dynamic area based on the two-dimensional position information of the vertices of the second dynamic area and the two-dimensional position information of the vertices of the first dynamic area; and embedding the edited image of the second dynamic area into the first dynamic area.

[0045] According to this configuration, the image of the second dynamic region can be embedded in the first dynamic region without causing any discomfort, without using three-dimensional position information.

[0046] (9) In the information processing method described in any one of (1) to (8) above, the dynamic area may be an area corresponding to a display area of ​​a monitor included in the real object.

[0047] According to this configuration, the image in the first dynamic range can be changed in conjunction with changes in the image displayed on the monitor provided in the real object.

[0048] (10) In another embodiment of the present invention, an information processing device is an information processing device having a processor, wherein the processor performs the following processing: obtaining a three-dimensional target obtained by reproducing an actual object including a dynamic area in a virtual space; determining a first dynamic area representing the dynamic area in the three-dimensional target; obtaining an actual image of the actual object captured by a camera; detecting a second dynamic area representing the dynamic area from the actual image; embedding the image of the second dynamic area as a texture image of the first dynamic area in real time; and outputting a display image of the three-dimensional target in which the image of the second dynamic area is embedded.

[0049] According to this configuration, it is possible to provide an information processing device that updates a dynamic area included in a three-dimensional object in real time.

[0050] (11) In another embodiment of the present invention, an information processing program causes a computer to perform the following processing: obtaining a three-dimensional target obtained by reproducing an actual object including a dynamic area in a virtual space; determining a first dynamic area representing the dynamic area in the three-dimensional target; obtaining an actual image of the actual object captured by a camera; detecting a second dynamic area representing the dynamic area from the actual image; embedding the image of the second dynamic area as a texture image of the first dynamic area in real time; and outputting a display image of the three-dimensional target in which the image of the second dynamic area is embedded.

[0051] According to this configuration, it is possible to provide an information processing program that updates a dynamic area included in a three-dimensional object in real time.

[0052] The present disclosure can also be implemented as such an information processing program or an information processing system that operates according to the information processing program. In addition, it is self-evident that such a computer program can be circulated via a computer-readable non-transient recording medium such as a CD-ROM or a communication network such as the Internet.

[0053] In addition, the embodiments described below all represent a specific example of the present disclosure. The numerical values, shapes, structural elements, steps, the order of steps, etc. shown in the following embodiments are an example and are not intended to limit the present disclosure. In addition, for the structural elements in the following embodiments that are not recorded in the independent claims representing the highest concept, they are described as arbitrary structural elements. In addition, it is also possible to combine the various contents in all embodiments.

[0054] (Implementation method 1)

[0055] Figure 1It is an overall structural diagram of the information processing system in embodiment 1. The information processing system includes an information processing device 1, a three-dimensional scanner 2, a camera 3, and a remote terminal 4. The information processing device 1 is composed of a computer. The information processing device 1 and the remote terminal 4 are connected to each other via a network NT so as to be able to communicate with each other. The network NT is, for example, the Internet. The three-dimensional scanner 2 and the camera 3 are connected to each other via a wireless or wired communication path so as to be able to communicate with each other. An example of a communication path is a wireless LAN, a wired LAN, Bluetooth (registered trademark), etc. The three-dimensional scanner 2 and the camera 3 exist at a site where an object device (an example of an actual object) on which an operator performs operations is set. The information processing device 1 can be set on the edge side including the site, or on the cloud side. When the information processing device 1 is set on the cloud side, the communication path is formed by the network NT.

[0056] The site is, for example, an actual space where the target equipment is set up. An example of a site is a factory, an experimental site, a test site, a chemical plant, etc. A factory can be a factory that manufactures electrical products such as televisions and washing machines, or a factory that manufactures cars, iron, etc. These sites are examples, and can be any place as long as it is a place where operators perform operations on the target equipment. For example, the site can also be a place where maintenance of machines or equipment is performed. An example of the target equipment is a manufacturing production line in a factory, etc. Here, the equipment is illustrated as an example of an actual object, but the actual object can also be a manufactured product manufactured by a manufacturing production line, etc. An example of a manufactured product is an electronic product, a car, etc. The target equipment includes a dynamic area. The dynamic area is an area where the state changes over time, such as the display area of ​​a monitor.

[0057] The three-dimensional scanner 2 acquires three-dimensional data of the object device by scanning the object device. The three-dimensional scanner 2 is, for example, a laser beam type three-dimensional scanner that acquires three-dimensional point group data of the object by irradiating a line laser to the object and receiving reflected light from the object. The three-dimensional scanner 2 outputs the point group data obtained by scanning the object device to the information processing device 1. In addition, the three-dimensional scanner 2 has a camera for capturing a texture image of the object, and outputs the texture image of the object device captured by the camera to the information processing device 1. The three-dimensional scanner 2 and the camera 3 are examples of a photographing device. Here, the camera of the three-dimensional scanner 2 captures the object device from multiple directions. The three-dimensional scanner 2 uses a portable three-dimensional scanner carried by the measurer, but this is an example, and a set-type three-dimensional scanner may also be used.

[0058] The camera 3 is composed of, for example, a portable camera equipped by an operator who performs operations on the target device. However, this is just an example, and the camera 3 may be a camera provided in a portable terminal carried by the operator, or may be a set-up camera. The camera 3 captures the target device at a given frame rate, and transmits the obtained actual image of the target device to the information processing device 1.

[0059] The remote terminal 4 is a terminal of a remote operator installed at a remote location. The remote terminal 4 is composed of a computer equipped with a display and a communication circuit. The remote terminal 4 may be a stationary computer or a portable computer.

[0060] The information processing device 1 includes a processor 10 , a memory 20 , and a communication unit 30 . The processor 10 is, for example, a central processing unit (CPU). The processor 10 includes an acquisition unit 11 , a determination unit 12 , a detection unit 13 , an embedding unit 14 , and an output unit 15 .

[0061] The acquisition unit 11 acquires a three-dimensional object obtained by reproducing the object device including the dynamic area in the virtual space. Here, the acquisition unit 11 acquires the point group data of the object device sent from the three-dimensional scanner 2 using the communication unit 30, and generates the three-dimensional object of the object device based on the acquired point group data. The three-dimensional object is three-dimensional data generated by embedding the texture image captured by the camera of the three-dimensional scanner 2 in the three-dimensional model representing the three-dimensional shape of the object device. The so-called virtual space refers to an imaginary three-dimensional space constructed in a computer.

[0062] In this embodiment, in the target device, a plurality of marks indicating that the target area is a dynamic area are arranged at the boundary of the dynamic area. Therefore, in the three-dimensional object, a plurality of marks are also arranged at the boundary of the dynamic area.

[0063] The determination unit 12 determines a first dynamic area representing a dynamic area in the three-dimensional object acquired by the acquisition unit 11. Here, the determination unit 12 determines the first dynamic area by detecting a plurality of markers displayed in the three-dimensional object. Alternatively, the determination unit 12 may determine the first dynamic area by reading from the memory 20 the three-dimensional position information of the plurality of markers detected when the three-dimensional object is generated.

[0064] The detection unit 13 uses the communication unit 30 to acquire the actual image of the target device captured by the camera 3, and detects the second dynamic area representing the dynamic area from the acquired actual image. Here, the detection unit 13 detects the second dynamic area by detecting a plurality of markers captured in the actual image. In addition, the detection unit 13 detects the second dynamic area every time the acquisition unit 11 acquires the actual image, that is, detects the second dynamic area in real time.

[0065] The embedding unit 14 embeds the image of the second dynamic area in real time as the texture image of the first dynamic area determined by the determination unit 12. The so-called real-time embedding, for example, means embedding the image of the second dynamic area in the first dynamic area in synchronization with the frame rate of the camera 3. However, this is an example, and the so-called real-time embedding may also mean embedding the image of the second dynamic area in the first dynamic area at a certain time interval in which the frame rate is sparsely distributed.

[0066] The output unit 15 outputs a display image of the three-dimensional target in which the image of the second dynamic area is embedded. Here, the output unit 15 only needs to generate the display image by rendering the three-dimensional target photographed by a virtual camera set in the virtual space in real time. The output unit 15 generates display data for displaying the generated display image on the remote terminal 4, and outputs the generated display data to the remote terminal 4 using the communication unit 30. Thus, the display image of the three-dimensional target is displayed on the display of the remote terminal 4. Therefore, the information processing device 1 can enable the remote operator to confirm the status of the object device in real time without sending the actual image of the object device. The position and posture of the virtual camera can be arbitrarily changed according to the instructions sent from the remote terminal 4. Thus, the remote operator can confirm the status of the object device from any direction through the three-dimensional target.

[0067] The memory 20 is composed of a non-volatile rewritable storage device such as a solid state drive and a hard disk drive. The memory 20 stores the three-dimensional object generated by the acquisition unit 11. The memory 20 stores the three-dimensional position information and posture information of the camera of the three-dimensional scanner 2 that captured the texture image when the three-dimensional object was generated.

[0068] The communication unit 30 is composed of a communication circuit that connects the information processing device 1 to the network NT. The communication unit 30 sends the display data of the three-dimensional object to the remote terminal 4. The communication unit 30 receives the point group data and the texture image from the three-dimensional scanner 2. The communication unit 30 receives the actual image captured by the camera 3 in real time. The communication unit 30 receives the instruction for setting the position and posture of the virtual camera from the remote terminal 4.

[0069] Figure 2 5 is a diagram showing an example of an object device 500. The object device 500 is a device having a frame including a frame. A monitor 501 is mounted on the surface of the object device 500. The monitor 501 displays a monitor screen 511 showing information related to the object device 500, such as the state of the object device 500. Figure 2 As shown in the left figure to the right figure, the monitor screen 511 changes over time. Figure 2 In the case where a three-dimensional object is generated at the time point of the left figure, even if the monitor screen 511 is Figure 2If the content of the monitor screen 511 is not reflected in the three-dimensional object, the remote operator cannot confirm the status of the monitor screen 511 of the target device 500 in real time. Therefore, in this embodiment, the image of the monitor screen 511 is cut out from the actual image captured by the camera 3, and the cut out image is embedded in the area of ​​the monitor 501 of the three-dimensional object in real time. As a result, the remote operator can confirm the content of the monitor screen 511 in real time.

[0070] Figure 3 This is a flowchart showing an example of a process of generating a three-dimensional object in Embodiment 1 of the present disclosure.

[0071] In step S1 , the three-dimensional scanner 2 scans the target device 500 . Thus, the acquisition unit 11 acquires point cloud data and a plurality of texture images of the target device 500 from the three-dimensional scanner 2 .

[0072] Next, in step S2 , the acquisition unit 11 generates a three-dimensional model from the point cloud data, and performs a meshing process for representing the surface of the generated three-dimensional model with a plurality of meshes.

[0073] Next, in step S3, the acquisition unit 11 generates a three-dimensional object by performing texture mapping processing to embed the texture image into the meshed three-dimensional model. For example, the acquisition unit 11 only needs to acquire the three-dimensional position information and posture information of the three-dimensional scanner 2 relative to the object device 500 when the texture image is photographed from the three-dimensional scanner 2, and embed the multiple texture images into the three-dimensional model using the acquired three-dimensional position information and posture information.

[0074] The acquisition unit 11 only needs to determine the texture image that best satisfies a given condition among the plurality of texture images captured in the first dynamic region and embed the texture image in the first dynamic region. Hereinafter, the texture image embedded when generating a three-dimensional object is referred to as an initial texture image.

[0075] As given conditions, conditions related to the shooting distance and conditions related to the shooting direction can be used. The shooting distance refers to the distance from the three-dimensional scanner 2 to the plurality of markers. For the conditions related to the shooting distance, the shorter the shooting distance, the higher the satisfaction of the condition. The shooting direction refers to the shooting direction of the three-dimensional scanner 2. For the conditions related to the shooting direction, the closer the shooting direction is to the front direction of the marker, the higher the satisfaction of the condition.

[0076] The acquisition unit 11 only needs to determine the texture image that has the largest sum of the satisfaction degree of the condition related to the shooting distance and the satisfaction degree of the condition related to the shooting direction from among the multiple texture images that have been shot in the dynamic area as the image that satisfies the given condition. In this way, the texture image that has shot the mark with the highest accuracy is embedded in the first dynamic area as the initial texture image. In addition, the given condition may be any one of the condition related to the shooting distance and the condition related to the shooting direction.

[0077] Next, in step S4, the acquisition unit 11 stores the camera position information indicating the position of the three-dimensional scanner 2 when the initial texture image embedded in the first dynamic area is photographed, the posture information indicating the posture, the three-dimensional position information of each of the plurality of markers, and the two-dimensional position information indicating the position of each of the plurality of markers in the initial texture image in the memory 20. Hereinafter, the camera position information and the posture information when the initial texture image is photographed are referred to as initial camera position information and initial posture information.

[0078] Next, in step S5 , the acquisition unit 11 outputs the generated three-dimensional object. Here, the acquisition unit 11 only needs to output the generated three-dimensional object to the memory 20 .

[0079] Figure 4 2 is a flowchart showing an example of the updating process in Embodiment 1. In step S11 , the acquisition unit 11 acquires the three-dimensional object from the memory 20 .

[0080] Next, in step S12, the determination unit 12 determines the first dynamic area. Here, the determination unit 12 may detect a plurality of markers arranged in the three-dimensional object by applying pattern matching to the three-dimensional object, and determine the area surrounded by the detected plurality of markers as the first dynamic area.

[0081] For example, in the case where a plurality of markers are arranged at the positions of the four vertices of a quadrilateral monitor, the determination unit 12 only needs to determine the quadrilateral area surrounded by the four markers as the first dynamic area. In addition, the number of markers arranged in the object device 500 can be either three or two. In the case where the number of markers is three, the quadrilateral area delimited by the three markers is determined as the first dynamic area. In the case where the number of markers is two, the two markers are arranged at the two opposite vertices of the quadrilateral dynamic area. Therefore, in the case where the number of markers is two, the quadrilateral area delimited by the two markers is determined as the first dynamic area. Here, the first dynamic area is set as a quadrilateral, but this is an example, and it can also be a polygon such as a triangle, a pentagon, or a hexagon. In this case, markers of a number corresponding to the shape of the first dynamic area are arranged.

[0082] In addition, in step S12, the determination unit 12 may also use Figure 3 The first dynamic area is determined based on the three-dimensional position information of the marker stored in the memory 20 in step S4.

[0083] Next, in step S13, the detection unit 13 performs a process of detecting the second dynamic area from the actual image captured by the camera 3. The details of this process will be described in detail. Figure 5 And then described.

[0084] Next, in step S14, the embedding unit 14 performs an update process of updating the first dynamic area with the image of the second dynamic area. The details of this process will be described in detail. Figure 6 The processing of steps S13 and S14 is repeatedly performed. Thus, the image of the second dynamic area is embedded in the first dynamic area in real time.

[0085] Figure 5 It is shown Figure 4 Flowchart showing details of the second dynamic area detection process shown in step S13 of FIG. 1 . In step S21 , the detection unit 13 acquires the actual image captured by the camera 3 using the communication unit 30 .

[0086] Next, in step S22 , the detection unit 13 detects a plurality of markers from the actual image by performing pattern matching using the image of the marker as a template.

[0087] Next, in step S23, the detection unit 13 detects the region surrounded by the detected plurality of markers as the second dynamic region. The details of this process are the same as the process of detecting the first dynamic region from the plurality of markers by the identification unit 12, and thus the detailed description is omitted.

[0088] Figure 6 It is shown Figure 4 This is a flowchart showing details of the updating process of the first dynamic area shown in step S14.

[0089] In step S31, the embedding unit 14 performs editing processing of the actual image based on the three-dimensional position information of each of the plurality of markers. The details of this processing will be described later.

[0090] Next, in step S32 , the embedding unit 14 embeds the edited image of the second dynamic region in the first dynamic region.

[0091] Next, in step S33, the output unit 15 generates a display image of the three-dimensional object with the first dynamic area updated, and transmits display data of the generated display image to the remote terminal 4 using the communication unit 30. Thus, the display image of the three-dimensional object is displayed on the display of the remote terminal 4.

[0092] Below, use Figure 7-Figure 8 Come to Figure 6 The details of the actual image editing process shown in step S31 are described below. Figure 7 : is a diagram showing an example of an initial texture image 100 including a dynamic area captured when a three-dimensional object is generated. The initial texture image 100 includes four markers M1 to M4. The markers M1 to M4 are two-dimensional markers with a given dot pattern drawn on the surface. The same dot pattern is drawn in the markers M1 to M4 respectively. The quadrilateral area surrounded by the markers M1 to M4 is the dynamic area 700. P1 to P4 are three-dimensional position information indicating the three-dimensional positions of the markers M1 to M4. p1 to p4 are two-dimensional position information indicating the positions of the markers M1 to M4 in the initial texture image. The three-dimensional position information P is represented by three coordinate components X, Y, and Z. For example, in the three-dimensional position information P1, the X, Y, and Z components are represented by the coordinate components X1, Y1, and Z1, respectively. The two-dimensional position information p is represented by two coordinate components x and y. For example, the two-dimensional position information p1 is represented by two coordinate components x1 and y1.

[0093] When generating a three-dimensional object, the three-dimensional position information P1 to P4 and the two-dimensional position information p1 to p4 are acquired. The initial camera position information Pc indicates the three-dimensional position of the three-dimensional scanner 2 when the initial texture image is captured. The initial posture information Qc indicates the posture of the three-dimensional scanner 2 when the initial texture image is captured. The initial posture information Qc is represented by three components: the sideslip angle R, the pitch angle P, and the yaw angle Y.

[0094] Figure 8 2 is a diagram showing an example of an actual image 200 captured by the camera 3 and including the second dynamic area. The two-dimensional position information p1' to p4' is two-dimensional position information indicating the positions of the markers M1 to M4 in the actual image 200. The two-dimensional position information p1' to p4' is detected when the markers M1 to M4 are detected by the detection unit 13. Since the target device 500 is fixed in the actual space, the three-dimensional position information P1 to P4 is Figure 7 The three-dimensional position information P1 to P4 are the same. The camera position information Pc' indicates the current three-dimensional position of the camera when the actual image 200 is captured. The posture information Qc' indicates the posture of the camera when the actual image 200 is captured. The camera position information Pc' and the posture information Qc' are unknown.

[0095] In the editing process of the actual image, first, the embedding unit 14 estimates the camera position information Pc´ and the posture information Qc´ by applying the self-position estimation algorithm to the initial texture image 100 and the actual image 200. An example of the self-position estimation algorithm is SLAM (Simultaneous Localization and Mapping). In detail, the embedding unit 14 detects feature points from the initial texture image 100 and the actual image 200 respectively, and matches the detected feature points with each other. For the detection of feature points, algorithms such as SIFT (Scale-Invariant Feature Transform) are used. In addition, the embedding unit 14 only needs to calculate the movement and rotation of the camera based on the matching results, and use the calculated movement, rotation, and the initial camera position information Pc and initial posture information Qc to estimate the camera position information Pc´ and posture information Qc´.

[0096] Next, the embedding unit 14 performs camera correction (camera calibration) to estimate the distortion parameters of the camera 3. The camera correction uses the estimated camera position information Pc´ and posture information Qc´, the initial camera position information Pc and initial posture information Qc, the three-dimensional position information P1~P4 of each marker M1~M4, the two-dimensional position information p1~p4 of each marker M1~M4, and the two-dimensional position information p1´~p4´ of each marker M1~M4.

[0097] As camera calibration, camera calibration using a camera model can be used. A camera model is a model that uses camera parameters to represent the relationship between two-dimensional coordinates on an image and three-dimensional coordinates in a real space corresponding to the two-dimensional coordinates. The camera parameters include external parameters, internal parameters, and distortion parameters.

[0098] Next, the embedding unit 14 uses the estimated distortion parameters to perform distortion correction to remove distortion from the actual image 200. Thus, the distortion is removed, and the actual image 200 edited so that the size matches the size of the first dynamic range is obtained.

[0099] Next, the embedding unit 14 cuts out the image of the dynamic region 700 from the distortion-corrected real image 200. This image becomes the image of the second dynamic region. Furthermore, the embedding unit 14 embeds the image of the second dynamic region into the first dynamic region of the three-dimensional object.

[0100] Fig. 9 This is a diagram showing how the image 800 of the second dynamic range is embedded in the first dynamic range 900 . Fig. 9The upper part of FIG. 1 is a diagram showing the actual image 200 after distortion correction. The embedding unit 14 cuts out the image of the dynamic region 700 from the actual image 200 after distortion correction as the image 800 of the second dynamic region. Fig. 9 As shown in the lower part of , the embedding unit 14 embeds the cut-out image 800 of the second dynamic region into the first dynamic region 900 of the three-dimensional object. Thus, the image 800 of the second dynamic region included in the actual image 200 captured at an arbitrary position and posture different from the initial texture image 100 can be accurately embedded into the first dynamic region 900.

[0101] Thus, according to Embodiment 1, the image of the second dynamic area detected from the actual image 200 captured by the camera 3 is embedded in real time in the first dynamic area determined from the three-dimensional object. Therefore, the texture image of the dynamic area included in the three-dimensional object can be updated in real time.

[0102] (Implementation method 2)

[0103] Embodiment 2 embeds the image 800 of the second dynamic region into the first dynamic region 900 without using the three-dimensional position information of each of the markers M1 to M4. In Embodiment 2, the same reference numerals are given to the same components as those in Embodiment 1, and the description thereof is omitted. Figure 1 In addition, the method of the second embodiment does not use the three-dimensional position information P1 to P4 of the markers M1 to M4 but uses the two-dimensional position information p1 to p4, p1' to p4' to embed the image 800 of the second dynamic area. Therefore, it is assumed that the initial texture image 100 and the actual image 200 are images obtained by photographing the dynamic area 700 of the target device from the front to some extent.

[0104] Fig.10 FIG. 1 is a flowchart showing an example of a process for generating a three-dimensional object in Embodiment 2. Fig.10 In, with Figure 3 The difference lies in step S4'. In step S4', the acquisition unit 11 stores the image name of the initial texture image 100 and the two-dimensional position information p1 to p4 of the markers M1 to M4 in the memory 20. Here, the three-dimensional object is configured so that the image name of the initial texture image 100 can be used as a key to determine in which region the initial texture image 100 is embedded.

[0105] In implementation mode 2, the update process is Figure 4 The first dynamic region is basically the same as the first dynamic region, but the process of determining the first dynamic region shown in step S12 and the process of updating the first dynamic region shown in step S14 are different, so these processes are described below. Figure 4The detection process of the second dynamic area shown in step S13 is the same as Figure 5 The detection process of the second dynamic range in the first embodiment shown is the same, so the description is omitted.

[0106] exist Figure 4 In step S12 shown in FIG. Fig.10 In step S4, the image name of the initial texture image stored in the memory 20 is used as a key to determine the initial texture image 100, and the two-dimensional position information p1-p4 of the markers M1-M4 is detected based on the initial texture image 100 to determine the first dynamic area. Alternatively, the determination unit 12 may also obtain the image name of the initial texture image stored in the memory 20 by Fig.10 In step S4', the first dynamic area is determined by using the two-dimensional position information p1 to p4 of the markers M1 to M4 stored in the memory 20.

[0107] Fig.11 FIG. 4 is a flowchart showing the details of the update process of the first dynamic area in the second embodiment. Fig.11 In, with Figure 6 The difference lies in step S31 ′. In step S31 ′, the embedding unit 14 performs editing processing of the actual image based on the two-dimensional position information p1 to p4 of each of the markers M1 to M4.

[0108] Below, use Fig.12 , Fig.13 The editing process of the actual image based on the two-dimensional position information is described below. Fig.12 is a diagram showing an example of an initial texture image 100. Fig.12 In, with Figure 7 The difference is as follows: Since the three-dimensional position information P1 to P4, the initial camera position information Pc, and the initial posture information Qc are not used in the second embodiment, their illustration is omitted. In the second embodiment, the two-dimensional position information p1 to p4 of the markers M1 to M4 is used, so Fig.12 The two-dimensional position information p1 to p4 is shown in FIG. The two-dimensional position information p1 to p4 is acquired when the three-dimensional object is generated.

[0109] Fig.13 This is a diagram showing how the image 800 of the second dynamic range is embedded in the first dynamic range 900 in the second embodiment. Fig.13 The left side of shows the actual image 200 including the second dynamic area captured by the camera 3. The detection unit 13 detects the markers M1 to M4 by applying pattern matching to the actual image 200, and detects the position information of the detected markers M1 to M4 as two-dimensional position information p1' to p4'.

[0110] Next, the embedding unit 14 cuts out the image 800 of the second dynamic region surrounded by the two-dimensional position information p1' to p4' from the actual image 200. Fig.13 As shown in the middle of , an image 800 of the second dynamic area is cut out from the actual image 200 .

[0111] Next, the embedding unit 14 reads the initial texture image 100 and the two-dimensional position information p1 to p4 of the markers M1 to M4 from the memory 20 using the image name of the initial texture image.

[0112] Next, the embedding unit 14 edits the image 800 of the second dynamic region so that the two-dimensional position information p1′ to p4′ located at the vertices of the second dynamic region are aligned with the image 800 of the second dynamic region. Fig.13 The two-dimensional position information p1 to p4 in the initial texture image 100 shown on the right side of . The embedding unit 14 may edit the image 800 of the second dynamic region by, for example, affine transformation.

[0113] Next, the embedding unit 14 embeds the edited second dynamic region image 800 into the first dynamic region 900 of the three-dimensional object. In this case, the embedding unit 14 only needs to determine the region of the three-dimensional object into which the initial texture image 100 is embedded based on the image name of the initial texture image 100, and determine the first dynamic region in the three-dimensional object based on the determined region and the two-dimensional position information p1 to p4 in the initial texture image 100.

[0114] As described above, according to the second embodiment, the image 800 of the second dynamic region can be embedded in the first dynamic region 900 without using the three-dimensional position information P1 to P4, so that the processing load of the processor consumed by the embedding process can be reduced.

[0115] The present disclosure can adopt the following modified examples.

[0116] (1) Markers M1 to M4 are used in embodiments 1 and 2, but the present disclosure is not limited to this. For example, the first dynamic area 900 and the second dynamic area can also be determined by object recognition processing. For example, in a case where the dynamic area is composed of a display area of ​​a monitor (an example of an object element), the determination unit 12 only needs to determine the display area of ​​the monitor by applying object recognition processing to a three-dimensional target, and determine the determined display area of ​​the monitor as the first dynamic area. In addition, the detection unit 13 only needs to determine the display area of ​​the monitor by applying object recognition processing to an actual image, and detect the determined display area of ​​the monitor as the second dynamic area. As an object recognition process, a method using an object recognizer that has been pre-machined can be used. As an object recognizer, an object recognizer that outputs a boundary box representing the display area of ​​the monitor superimposed with a three-dimensional target or an actual image can be used.

[0117] (2) When generating a three-dimensional object, instead of using an image captured by the camera of the three-dimensional scanner 2, an actual image captured by the camera 3 may be used as a texture image embedded in the three-dimensional object. In this case, the acquisition unit 11 only needs to use the mark captured in the actual image as a marker and embed the actual image captured by the camera 3 as a texture image in the three-dimensional model.

[0118] (3) In the above-described embodiment, the distortion parameters are acquired through camera calibration, but known distortion parameters may also be acquired from the memory 20 .

[0119] Industrial Applicability

[0120] The present disclosure is useful in the technical field of assisting a field from a remote location.

Claims

1. An information processing method is an information processing method in a computer. The information processing method comprises: Acquire a three-dimensional object obtained by reproducing a real object including a dynamic area in a virtual space; Determining a first dynamic region in the three-dimensional object that represents the dynamic region; Acquiring an actual image of the actual object captured by a camera; detecting a second dynamic region representing the dynamic region from the actual image; embedding the image of the second dynamic area as a texture image of the first dynamic area in real time; A display image of the three-dimensional object in which the image of the second dynamic area is embedded is output.

2. The information processing method according to claim 1, wherein: A mark indicating the dynamic area is attached to the actual object. The determining of the first dynamic area includes: detecting the first dynamic area based on the marker in the three-dimensional object, The detecting of the second dynamic area includes detecting the second dynamic area based on an image representing the marker included in the actual image.

3. The information processing method according to claim 1, wherein: The dynamic area corresponds to the object elements constituting the real object, The determining of the first dynamic area includes: determining the area of ​​the object element from the three-dimensional object through object recognition processing, The detection of the second dynamic area includes detecting the area of ​​the object element from the actual image through the object recognition processing.

4. The information processing method according to claim 1, wherein: The actual image is repeatedly acquired, The detection of the second dynamic area, the embedding, and the output are performed every time the actual image is acquired.

5. The information processing method according to claim 1, wherein: The embedding of the image of the second dynamic area comprises: Obtaining distortion parameters of the photographing device; Using the distortion parameter, correcting the distortion of the image in the second dynamic range; and The corrected image of the second dynamic range is embedded in the first dynamic range.

6. The information processing method according to claim 5, wherein: The distortion parameters are estimated through camera calibration, which uses the current position and posture information of the shooting device, the initial position and posture information of the shooting device when shooting the initial texture image embedded in the first dynamic area, the two-dimensional position information of the dynamic area in the initial texture image, the two-dimensional position information of the second dynamic area in the actual image, and the three-dimensional position information of the first dynamic area.

7. The information processing method according to claim 6, wherein: The current position and posture information of the camera are estimated by applying a self-position estimation algorithm to the initial texture image and the actual image.

8. The information processing method according to claim 1, wherein: The embedding of the image of the second dynamic area comprises: Based on the two-dimensional position information of the vertices of the second dynamic region and the two-dimensional position information of the vertices of the first dynamic region, editing the image of the second dynamic region to make it conform to the shape of the first dynamic region; and The edited image of the second dynamic area is embedded in the first dynamic area.

9. The information processing method according to claim 1, wherein: The dynamic area is an area corresponding to a display area of ​​a monitor included in the real object.

10. An information processing device comprising a processor, The processor performs the following processing: Acquire a three-dimensional object obtained by reproducing a real object including a dynamic area in a virtual space; Determining a first dynamic region in the three-dimensional object that represents the dynamic region; Acquiring an actual image of the actual object captured by a camera; detecting a second dynamic region representing the dynamic region from the actual image; embedding the image of the second dynamic area as a texture image of the first dynamic area in real time; A display image of the three-dimensional object in which the image of the second dynamic area is embedded is output.

11. An information processing program that causes a computer to execute the following processing: Acquire a three-dimensional object obtained by reproducing a real object including a dynamic area in a virtual space; Determining a first dynamic region in the three-dimensional object that represents the dynamic region; Acquiring an actual image of the actual object captured by a camera; detecting a second dynamic region representing the dynamic region from the actual image; embedding the image of the second dynamic area as a texture image of the first dynamic area in real time; A display image of the three-dimensional object in which the image of the second dynamic area is embedded is output.

Images