Mapping method, information processing device, and program

Abstract

A mapping method, an information processing device, and a program for mapping a characteristic portion of a detected object onto a three-dimensional model are provided. [Solution] An information processing device acquires a first image of an object photographed by a camera, acquires a first detection image of the object photographed simultaneously by a detection camera arranged parallel to the camera at a predetermined distance, determines an estimated shooting position and a first estimated angle of view of the camera when the first image was photographed based on the first image and a three-dimensional model, determines an estimated shooting position of the detection camera when the first detection image was photographed based on the estimated shooting position of the camera when the first image was photographed and the predetermined distance, detects a first characteristic part from the first detection image, and determines the position of the first characteristic part on the three-dimensional model in a virtual space based on the intersection of a straight line from the estimated shooting position of the detection camera when the first detection image was photographed, directed toward the first characteristic part arranged within the first estimated angle of view, and the three-dimensional model.

Description

[Technical Field]

[0001] The present disclosure relates to a method for mapping a captured image onto a three-dimensional model, an information processing device, and a program. [Background technology]

[0002] Conventionally, a technology has been disclosed in which a flying device equipped with a camera generates a three-dimensional model of an object to be inspected based on multiple images of the object, and displays the three-dimensional model by mapping the positions of abnormalities in the object to be inspected in the images (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-211257 Summary of the Invention [Problem to be solved by the invention]

[0004] However, Patent Document 1 assumes that one camera captures multiple images directly below in the vertical direction in one inspection, and does not fully consider, for example, when a flying device is equipped with multiple cameras, how to improve the efficiency of estimating the camera position and orientation, or when multiple inspections are performed. As such, there is room for improvement in the technology for mapping captured images onto a 3D model.

[0005] In view of the above circumstances, an object of the present disclosure is to improve the technology for mapping captured images onto a three-dimensional model. [Means for solving the problem]

[0006] (1) A mapping method according to an embodiment of the present disclosure includes: A mapping method executed by an information processing device, Placing a three-dimensional model of the object in a virtual space; acquiring a first image of the object captured by a camera; acquiring a first detection image of the object photographed simultaneously with the first image by a detection camera arranged in parallel with the camera at a predetermined distance; determining an estimated photographing position and a first estimated angle of view of the camera when photographing the first image based on the first image and the three-dimensional model; determining an estimated photographing position of the detection camera when capturing the first detection image based on the estimated photographing position of the camera when capturing the first image and the predetermined distance; Detecting a first feature portion from the first detection image; determining, in the virtual space, a position of the first characteristic portion on the three-dimensional model based on an intersection point between a straight line extending from an estimated shooting position of the detection camera at the time of capturing the first detection image to the first characteristic portion disposed within the first estimated angle of view and the three-dimensional model; Includes.

[0007] (2) A mapping method according to an embodiment of the present disclosure is the mapping method according to (1), further comprising: acquiring a second image of the object photographed by the camera a predetermined period after the first detection image is photographed; acquiring a second detection image of the object captured simultaneously with the second image by the detection camera; determining an estimated shooting position and a second estimated angle of view of the camera when capturing the second image based on the second image and the three-dimensional model; determining an estimated photographing position of the detection camera when capturing the second detection image based on the estimated photographing position of the camera when capturing the second image and the predetermined distance; Detecting a second feature portion from the second detection image; determining, in the virtual space, a position of the second feature on the three-dimensional model based on an intersection of a straight line extending from a capturing position of the detection camera at the time of capturing the second detection image toward the second feature located within the second estimated angle of view and the three-dimensional model; plotting the first feature and the second feature on the three-dimensional model; Includes.

[0008] (3) A mapping method according to an embodiment of the present disclosure is the mapping method according to (2), If the position of the first feature portion on the three-dimensional model is the same as the position of the second feature portion on the three-dimensional model, a difference based on the first feature portion and the second feature portion is plotted on the three-dimensional model.

[0009] (4) A mapping method according to an embodiment of the present disclosure is the mapping method according to any one of (1) to (3), The detection camera includes at least one of a visible light camera, an infrared light camera, a multispectral camera, and a hyperspectral camera.

[0010] (5) A mapping method according to an embodiment of the present disclosure is the mapping method according to (2) or (3), further comprising: The method includes detecting the first feature portion and the second feature portion based on the first detection image and the second detection image and a learning model.

[0011] (6) A mapping method according to an embodiment of the present disclosure is the mapping method according to any one of (1) to (5), The first characteristic portion is a portion where a predetermined object or a predetermined defect included in the first detection image is photographed.

[0012] (7) A mapping method according to an embodiment of the present disclosure is the mapping method according to (6), The predetermined object includes at least one of water, a predetermined mineral, and a predetermined plant.

[0013] (8) A mapping method according to an embodiment of the present disclosure is the mapping method according to (6), The predetermined defect includes a defect or flaw in the structure.

[0014] (9) A mapping method according to an embodiment of the present disclosure is the mapping method according to any one of (1) to (8), The first detection image is an image taken by the detection camera mounted on a moving body.

[0015] (10) An information processing device according to an embodiment of the present disclosure includes: An information processing device including a control unit, The control unit A 3D model of the object is placed in the virtual space, acquiring a first image of the object photographed by a camera; a detection camera arranged in parallel with the camera at a predetermined distance from the camera acquires a first detection image of the object photographed simultaneously with the first image; determining an estimated photographing position and a first estimated angle of view of the camera when photographing the first image based on the first image and the three-dimensional model; determining an estimated photographing position of the detection camera when the first detection image was captured based on the estimated photographing position of the camera when the first image was captured and the predetermined distance; Detecting a first feature portion from the first detection image; In the virtual space, the position of the first characteristic portion on the three-dimensional model is determined based on the intersection of a straight line extending from the estimated shooting position of the detection camera at the time of shooting the first detection image to the first characteristic portion located within the first estimated angle of view and the three-dimensional model.

[0016] (11) A program according to an embodiment of the present disclosure includes: On the computer, Placing a three-dimensional model of the object in a virtual space; acquiring a first image of the object captured by a camera; acquiring a first detection image of the object photographed simultaneously with the first image by a detection camera arranged in parallel with the camera at a predetermined distance; determining an estimated photographing position and a first estimated angle of view of the camera when photographing the first image based on the first image and the three-dimensional model; determining an estimated photographing position of the detection camera when capturing the first detection image based on the estimated photographing position of the camera when capturing the first image and the predetermined distance; Detecting a first feature portion from the first detection image; determining, in the virtual space, a position of the first characteristic portion on the three-dimensional model based on an intersection point between a straight line extending from an estimated shooting position of the detection camera at the time of capturing the first detection image to the first characteristic portion disposed within the first estimated angle of view and the three-dimensional model; Execute the following. [Effects of the Invention]

[0017] According to one embodiment of the present disclosure, a technique for mapping captured images onto a three-dimensional model is improved. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a block diagram illustrating a schematic configuration of a system according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a block diagram showing a schematic configuration of an information processing device. [Figure 3] FIG. 1 is a block diagram showing a schematic configuration of a moving body. [Figure 4] FIG. 2 is a schematic diagram showing the positional relationship between a camera of a moving body and a detection camera. [Figure 5] 10 is a flowchart showing a first operation example of the information processing device. [Figure 6]FIG. 1 is a schematic diagram showing an example of a display of a three-dimensional model. [Figure 7A] 10 is a flowchart showing a second operation example of the information processing device. [Figure 7B] 10 is a flowchart showing a second operation example of the information processing device. [Figure 8] FIG. 1 is a schematic diagram showing an example of a display of a three-dimensional model. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, embodiments of the present disclosure will be described.

[0020] (Outline of the embodiment) The outline and configuration of a system 1 according to this embodiment will be described with reference to FIG.

[0021] The system 1 according to this embodiment includes an information processing device 10 and a mobile object 20. The information processing device 10 and the mobile object 20 are communicably connected to a network 30 including, for example, a mobile communication network and the Internet.

[0022] The information processing device 10 is any device used by a user (e.g., a facility manager, an infrastructure inspector, a vegetation management implementation investigator, a farm manager, etc.). For example, a general-purpose electronic device such as a personal computer, a smartphone, a tablet terminal, or a wearable terminal, or a dedicated electronic device, can be adopted as the information processing device 10. Furthermore, for example, the information processing device 10 may be a server device installed in a data center or the like. The information processing device 10 is capable of communicating with a mobile object 20 via a network 30. Note that while FIG. 1 shows an example in which the system 1 is equipped with one information processing device 10, this is not limiting. The system 1 may be equipped with two or more information processing devices 10.

[0023] The mobile object 20 is a mobile object equipped with a camera for capturing images or video (hereinafter collectively referred to as an image) and a detection camera, and may be, for example, an unmanned aerial vehicle. Unmanned aerial vehicles include drones. However, the mobile object 20 is not limited to this, and may also be an automobile, ship, robot, etc. equipped with a camera and a detection camera.

[0024] First, an overview of this embodiment will be described, and details will be described later. The mapping method according to this embodiment is executed by an information processing device 10. The information processing device 10 first places a 3D model of an object in a virtual space. The information processing device 10 also acquires a first image of the object captured by a camera. The information processing device 10 also acquires a first detection image of the object captured simultaneously with the first image by a detection camera arranged parallel to the camera at a predetermined distance. The information processing device 10 also determines an estimated shooting position and a first estimated angle of view of the camera when the first image was captured, based on the first image and the 3D model. The information processing device 10 also determines an estimated shooting position of the detection camera when the first detection image was captured, based on the estimated shooting position of the camera when the first image was captured and a predetermined distance. The information processing device 10 also detects a first feature portion from the first detection image. Furthermore, the information processing device 10 determines the position of the first feature on the three-dimensional model in the virtual space based on the intersection of a straight line from the estimated shooting position of the detection camera at the time of shooting the first detection image to the first feature located within the first estimated angle of view and the three-dimensional model.

[0025] As described above, according to this embodiment, the estimated shooting position and the first estimated angle of view of the camera are determined based on the first image and the three-dimensional model. Because the camera position and orientation estimation is performed by the information processing device 10, rather than the moving body 20, the processing load on the moving body 20 can be reduced. Therefore, for example, if the moving body 20 is a drone, the power consumption of the moving body 20 can be reduced, ensuring a longer flight time. Furthermore, because the camera position and orientation estimation is performed by the information processing device 10, real-time processing is also possible. Furthermore, according to this embodiment, the estimated shooting position of the detection camera is determined based on the estimated shooting position of the camera. Therefore, even if the moving body 20 is equipped with multiple cameras, the estimated shooting position can be appropriately determined. As described above, according to this embodiment, the mapping technique for a captured image on a three-dimensional model is improved.

[0026] Next, each component of the system 1 will be described in detail.

[0027] (Configuration of information processing device) As shown in FIG. 2, the information processing device 10 includes a control unit 11, a storage unit 12, an input unit 13, an output unit 14, and a communication unit 15.

[0028] The control unit 11 includes at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a central processing unit (CPU) or a graphics processing unit (GPU), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The control unit 11 executes processes related to the operation of the information processing device 10 while controlling each unit of the information processing device 10.

[0029] The storage unit 12 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a random access memory (RAM) or a read only memory (ROM). The RAM is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The ROM is, for example, an electrically erasable programmable read only memory (EEPROM). The storage unit 12 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 12 stores data used in the operation of the information processing device 10 and data obtained by the operation of the information processing device 10.

[0030] The input unit 13 includes at least one input interface. The input interface is, for example, a physical key, a capacitance key, a pointing device, or a touch screen integrated with a display. The input interface may also be, for example, a sound sensor that accepts voice input, or a camera that accepts gesture input. The input unit 13 accepts an operation to input data used for the operation of the information processing device 10. The input unit 13 may be connected to the information processing device 10 as an external input device instead of being provided in the information processing device 10. Any connection method may be used, for example, a Universal Serial Bus (USB), a High-Definition Multimedia Interface (HDMI) (registered trademark), or Bluetooth (registered trademark).

[0031] The output unit 14 includes at least one output interface. The output interface is, for example, a display that outputs information as an image, or a speaker that outputs information as sound. The display is, for example, an LCD (liquid crystal display) or an organic EL (electro luminescence) display. The output unit 14 displays and outputs data obtained by the operation of the information processing device 10. The output unit 14 may be connected to the information processing device 10 as an external output device instead of being provided in the information processing device 10. Any connection method can be used, for example, USB, HDMI (registered trademark), or Bluetooth (registered trademark).

[0032] The communication unit 15 includes at least one external communication interface. The communication interface may be either a wired communication interface or a wireless communication interface. In the case of wired communication, the communication interface is, for example, a LAN (Local Area Network) interface or a USB (Universal Serial Bus). In the case of wireless communication, the communication interface is, for example, an interface compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th generation), or 5G (5th generation), or an interface compatible with short-range wireless communication such as Bluetooth (registered trademark). The communication unit 15 receives data used in the operation of the information processing device 10 and transmits data obtained by the operation of the information processing device 10.

[0033] The functions of the information processing device 10 are realized by executing a program according to this embodiment on a processor corresponding to the control unit 11. That is, the functions of the information processing device 10 are realized by software. The program causes a computer to execute the operations of the information processing device 10, thereby causing the computer to function as the information processing device 10. That is, the computer functions as the information processing device 10 by executing the operations of the information processing device 10 in accordance with the program.

[0034] In this embodiment, the program can be recorded on a computer-readable recording medium. The computer-readable recording medium includes non-transitory computer-readable media, such as a magnetic recording device, an optical disc, a magneto-optical recording medium, or a semiconductor memory. The program can be distributed, for example, by selling, transferring, or lending a portable recording medium, such as a DVD (digital versatile disc) or a CD-ROM (compact disc read only memory), on which the program is recorded. The program can also be distributed by storing the program in the storage of an external server and transmitting the program from the external server to another computer. The program can also be provided as a program product.

[0035] Some or all of the functions of the information processing device 10 may be implemented by a dedicated circuit equivalent to the control unit 11. In other words, some or all of the functions of the information processing device 10 may be implemented by hardware.

[0036] (Mobile configuration) As shown in FIG. 3, the moving object 20 includes a control unit 21, a storage unit 22, a camera 23, a detection camera 24, and a communication unit 25.

[0037] The control unit 21 includes at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a central processing unit (CPU) or a graphics processing unit (GPU), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The control unit 21 controls each part of the mobile object 20 and executes processes related to the operation of the mobile object 20.

[0038] The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a random access memory (RAM) or a read only memory (ROM). The RAM is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The ROM is, for example, an electrically erasable programmable read only memory (EEPROM). The storage unit 22 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 22 stores data used in the operation of the mobile body 20 and data obtained by the operation of the mobile body 20.

[0039] The camera 23 is an imaging device that captures images (including moving images, as mentioned above). For example, the camera 23 may be an RGB camera that captures images using the three basic colors of red (R), green (G), and blue (B). In this case, the camera 23 acquires information for each color through a sensor and a filter, and combines this information using image processing technology to generate an image. The sensor may be, for example, a CMOS (Complementary Metal-Oxide-Semiconductor) or a CCD (Charge-Coupled Device) sensor. When a CMOS sensor is used, high-speed processing can be achieved with low power consumption. When a CCD sensor is used, high image quality and low noise can be achieved.

[0040] The detection camera 24 is an imaging device that captures images (including moving images, as described above, hereinafter referred to as detection images) for detecting characteristic portions. The detection camera 24 includes at least one of a visible light camera, an infrared camera, a multispectral camera, and a hyperspectral camera. A visible light camera is an imaging device that detects visible light with wavelengths of approximately 380 nm to 750 nm. An infrared camera is an imaging device that detects infrared light (750 nm or longer) with wavelengths longer than visible light. An infrared camera can be particularly used for imaging in dark places and sensing the temperature of objects. Either a cooled or uncooled infrared camera may be used. A multispectral camera is a camera that can simultaneously capture light of multiple different wavelengths. A multispectral camera can also capture wavelengths other than visible light, such as near-infrared and ultraviolet light. A hyperspectral camera is an imaging device that subdivides light across a wide wavelength range and acquires information in multiple spectral bands. Detailed spectral data from each pixel enables detection of differences and abnormalities in materials that cannot be captured by ordinary cameras. The detection camera 24 is arranged in parallel with the camera 23 at a predetermined distance. FIG. 4 is a schematic diagram showing the positional relationship between the camera 23 and the detection camera 24 mounted on the moving body 20. As shown in FIG. 4, the camera 23 and the detection camera 24 are arranged in parallel with each other at a predetermined distance d, for example, at the bottom of the moving body 20. In other words, the camera 23 and the detection camera 24 are arranged at a predetermined distance d and offset in the planar direction. On the other hand, the orientations of the camera 23 and the detection camera 24 are the same. In other words, the angle of view of the camera 23 and the angle of view of the detection camera 24 are the same. Note that the orientations of the camera 23 and the detection camera 24 may be different. In other words, the angle of view of the camera 23 and the angle of view of the detection camera 24 may be different. In this case, correction may be performed based on the angle of view of the camera 23 and the angle of view of the detection camera 24.

[0041] The communication unit 25 includes at least one external communication interface. The communication interface may be either a wired or wireless communication interface. In the case of wired communication, the communication interface may be, for example, a LAN (Local Area Network) interface or a USB (Universal Serial Bus). In the case of wireless communication, the communication interface may be, for example, an interface compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th generation), or 5G (5th generation), or an interface compatible with short-range wireless communication such as Bluetooth (registered trademark). The communication unit 25 receives data used in the operation of the mobile object 20 and transmits data obtained by the operation of the mobile object 20. Note that the processing in this embodiment may or may not be real-time processing. Therefore, for example, if the mobile object 20 is an unmanned aerial vehicle and wired communication is performed using a communication interface, a communication connection may be established after the unmanned aerial vehicle has flown, and various processing may be performed.

[0042] The functions of the moving body 20 are realized by executing a program according to this embodiment on a processor corresponding to the control unit 21. That is, the functions of the moving body 20 are realized by software. The program causes a computer to execute the operations of the moving body 20, thereby causing the computer to function as the moving body 20. That is, the computer functions as the moving body 20 by executing the operations of the moving body 20 in accordance with the program.

[0043] Some or all of the functions of the moving object 20 may be realized by a dedicated circuit equivalent to the control unit 21. In other words, some or all of the functions of the moving object 20 may be realized by hardware.

[0044] (First operation example of information processing device) A first operation example of the information processing device 10 according to this embodiment will be described with reference to FIG.

[0045] Step S1: The control unit 11 of the information processing device 10 places a 3D model of an object in a virtual space. The 3D model of the object may be created by any method. For example, the 3D model of the object may be created by photogrammetry or the like.

[0046] Step S2: The control unit 11 acquires a first image of the object photographed by the camera 23. Specifically, the control unit 11 acquires the first image of the object photographed by the camera 23 via the communication unit 25 of the moving object 20.

[0047] Step S3: The control unit 11 acquires a first detection image of the object photographed simultaneously with the first image by the detection camera 24. Specifically, the control unit 11 acquires the first detection image of the object photographed by the detection camera 24 via the communication unit 25 of the moving object 20.

[0048] Step S4: Based on the first image and the three-dimensional model, the control unit 11 determines an estimated shooting position and a first estimated angle of view (posture) of the camera 23 when the first image was captured. Any method may be used to determine the estimated shooting position and the first estimated angle of view. For example, the control unit 11 may determine the estimated shooting position and the first estimated angle of view using a Visual Positioning System (VPS). The VPS is a technology that extracts feature points from a two-dimensional image and matches them with corresponding points on a three-dimensional model.

[0049] Step S5: The control unit 11 determines the estimated shooting position of the detection camera 24 when the first detection image was captured, based on the estimated shooting position of the camera 23 when the first image was captured and the predetermined distance d. Here, as described above, the camera 23 and the detection camera 24 are provided in parallel below the moving body 20, separated by the predetermined distance d. Therefore, the estimated shooting position of the detection camera 24 can be determined from the estimated shooting position of the camera 23 and the predetermined distance d. Furthermore, the angle of view of the detection camera 24 when capturing an image is the same as the first estimated angle of view. In other words, the estimated shooting position and estimated angle of view of the detection camera 24 can be determined based on the estimated shooting position and estimated angle of view of the camera 23 and the positional relationship between the camera 23 and the detection camera 24 on the moving body 20.

[0050] Step S6: The control unit 11 detects a first characteristic portion from the first detection image. Any method can be used for detecting the first characteristic portion. For example, the control unit 11 may detect the first characteristic portion based on the first detection image and a trained model. Specifically, the control unit 11 may input data related to the first detection image into a trained model for image recognition, thereby acquiring data related to the first characteristic portion as the output of the trained model. The first characteristic portion may be a portion of the first detection image where a predetermined object or a predetermined defect is captured. The predetermined object may include water, a predetermined mineral, a predetermined plant, etc. The predetermined defect may be a defect or a scratch on a structure. The defect or a scratch on a structure may be peeling, cracks, holes, corrosion, distortion, or breakage of tiles. These defects or scratches can have a significant impact on the durability, safety, etc. of a building, and therefore require early detection and appropriate repair. For example, characteristic portions can be detected by processing such as analyzing the spectrum of information obtained by a hyperspectral camera and extracting characteristic wavelength ranges. More specifically, detection is possible using NDVI, which is known as an index showing the distribution and activity of vegetation.

[0051] Here, the trained model is a model created by machine learning using a machine learning algorithm. The trained model may be, for example, a machine learning model built based on a decision tree. Examples of machine learning models built based on a decision tree include, but are not limited to, Light GBM and XGBoost. Alternatively, the trained model may be a model generated based on a machine learning algorithm such as a convolutional neural network (CNN), a recurrent neural network (RNN), or other deep learning.

[0052] Step S7: The control unit 11 determines the position of the first feature on the three-dimensional model based on the intersection of a straight line (hereinafter also referred to as a ray) from the estimated shooting position of the detection camera 24 when the first detection image was captured to the first feature located within the first estimated angle of view in the virtual space with the three-dimensional model. If there are multiple intersections between the ray and the three-dimensional model, the control unit 11 determines the position of the first feature on the three-dimensional model based on the intersection closest to the estimated shooting position (in other words, the first intersection where the ray intersects with the three-dimensional model).

[0053] The control unit 11 may further plot the first characteristic portion on the three-dimensional model. That is, the control unit 11 may store the position coordinates of the first characteristic portion in the data of the three-dimensional model. The control unit 11 may display the three-dimensional model on which the first characteristic portion is plotted. Any method may be employed for displaying the three-dimensional model. For example, the control unit 11 may display the three-dimensional model by a user interface via the output unit 14 of the information processing device 10.

[0054] FIG. 6 is a schematic diagram showing a display example of a three-dimensional model. Here, the description will be given assuming that the object is a planter and the characteristic part is water. In FIG. 6, a first detection image 102 is superimposed on the three-dimensional model 101. Also in FIG. 6, an estimated shooting position 103 of the detection camera 24 is superimposed. Furthermore, in FIG. 6, auxiliary lines 104 to 107 indicating the angle of view of the detection camera 24 are superimposed. The auxiliary lines 104 to 107 in FIG. 6 are line segments connecting the estimated shooting position of the detection camera 24 and four vertices of the first detection image 102. As shown in FIG. 6, a characteristic part 108 in the first detection image 102 is mapped as a plot group 109 on the three-dimensional model 101 based on the above-mentioned intersections.

[0055] As described above, according to this embodiment, the estimated shooting position and the first estimated angle of view of the camera are determined based on the first image and the three-dimensional model. Because the camera position and orientation estimation is performed by the information processing device 10, rather than the moving body 20, the processing load on the moving body 20 can be reduced. Therefore, for example, if the moving body 20 is a drone, the power consumption of the moving body 20 can be reduced and a longer flight time can be ensured. Furthermore, because the camera position and orientation estimation is performed by the information processing device 10, real-time processing is also possible. Furthermore, according to this embodiment, the estimated shooting position of the detection camera 24 is determined based on the estimated shooting position of the camera 23. Therefore, even if the moving body 20 is equipped with multiple cameras, the estimated shooting position can be appropriately determined. As described above, according to this embodiment, the mapping technique for a captured image on a three-dimensional model is improved.

[0056] (Second operation example of information processing device) 7A and 7B, a second operation example of the information processing device 10 according to this embodiment will be described. Operations that are the same as those in the first operation example will be assigned the same reference numerals, and descriptions thereof will be omitted.

[0057] Step S8: Following step S7 described above, the control unit 11 acquires a second image of the object photographed by the camera 23 a predetermined period after the first detection image is photographed. Specifically, the control unit 11 acquires the second image of the object photographed by the camera 23 via the communication unit 25 of the mobile object 20. The predetermined period may be any period. For example, the predetermined period may be one hour, one day, etc.

[0058] Step S9: The control unit 11 acquires a second detection image of the object photographed simultaneously with the second image by the detection camera 24. Specifically, the control unit 11 acquires the second detection image of the object photographed by the detection camera 24 via the communication unit 25 of the moving object 20.

[0059] Step S10: Based on the second image and the three-dimensional model, the control unit 11 determines an estimated shooting position and a second estimated angle of view (posture) of the camera 23 when the second image was captured. Any method may be used to determine the estimated shooting position and the second estimated angle of view. For example, the control unit 11 may determine the estimated shooting position and the second estimated angle of view using a VPS.

[0060] Step S11: The control unit 11 determines the estimated shooting position of the detection camera 24 when capturing the second detection image, based on the estimated shooting position of the camera 23 when capturing the second image and the predetermined distance d. As described above, the estimated shooting position of the detection camera 24 can be determined from the estimated shooting position of the camera 23 and the predetermined distance d. The angle of view of the detection camera 24 when capturing the image is the same as the second estimated angle of view.

[0061] Step S12: The control unit 11 detects a second feature portion from the second detection image. For example, the control unit 11 may detect the second feature portion based on the second detection image and a trained model. Specifically, for example, the control unit 11 may input data related to the second detection image into a trained model for image recognition, thereby acquiring data related to the second feature portion as the output of the trained model. The second feature portion may be a portion where a predetermined object or a predetermined defect included in the second detection image is photographed. The predetermined object may include water, a predetermined mineral, a predetermined plant, etc. The predetermined defect may be a defect or scratch on a structure.

[0062] Step S13: The control unit 11 determines the position of the second feature on the three-dimensional model based on the intersection of the three-dimensional model and a straight line (ray) directed from the estimated shooting position of the detection camera 24 at the time of capturing the second detection image to the second feature located within the second estimated angle of view in the virtual space. If there are multiple intersections between the ray and the three-dimensional model, the control unit 11 determines the position of the second feature on the three-dimensional model based on the intersection closest to the estimated shooting position (in other words, the first intersection where the ray intersects with the three-dimensional model).

[0063] Step S14: The control unit 11 plots the first characteristic portion and the second characteristic portion on the three-dimensional model. That is, the control unit 11 stores the position coordinates of the first characteristic portion and the second characteristic portion in the data of the three-dimensional model. The control unit 11 may display the three-dimensional model on which the first characteristic portion and the second characteristic portion are plotted.

[0064] FIG. 8 is a schematic diagram showing a display example of a three-dimensional model. The same components as those in FIG. 6 are denoted by the same reference numerals, and a description thereof will be omitted. As in FIG. 6, the description will be given assuming that the object is a planter and the characteristic feature is water. In FIG. 8, a second detection image 112 is superimposed on the three-dimensional model 101. Also in FIG. 8, an estimated shooting position 113 of the detection camera 24 is superimposed. Furthermore, in FIG. 8, auxiliary lines 114 to 117 indicating the angle of view of the detection camera 24 are superimposed. The auxiliary lines 114 to 117 in FIG. 8 are line segments connecting the estimated shooting position of the detection camera 24 and the four vertices of the second detection image 112. As shown in FIG. 8, a characteristic feature 118 in the second detection image 112 is mapped as a plot group 119 on the three-dimensional model 101 based on the above-mentioned intersections.

[0065] Here, if the position of a first feature portion on the 3D model is the same as the position of a second feature portion on the 3D model, a difference based on the first feature portion and the second feature portion may be plotted on the 3D model. This allows for understanding the progression of an abnormality, deterioration, etc. in the same location, such as whether it is expanding or remaining constant. The display manner of the plot may also be changed depending on changes in the same location. For example, the color, brightness, etc. of the plot may be changed depending on the amount of change, difference, etc. A gradation may also be added to the plot. This makes it possible to visually easily grasp the amount of change. The gradient of the change may also be determined based on the amount of change, difference, etc. Future conditions may also be predicted based on the gradient of the amount of change, difference, etc. The environment at the time of shooting (weather, temperature, humidity, shadows at the time of shooting, etc.) may also be taken into consideration. For example, feature portions may be plotted on the 3D model based on multiple images, taking into account the influence of the environment at the time of shooting, noise, etc.

[0066] In the second operation example, the first detection image and the second detection image were images taken at two different points in time, but the number of detection images is not limited to this, and detection images taken at three or more different points in time may also be taken.

[0067] Although the present disclosure has been described based on the drawings and examples, it should be noted that those skilled in the art may make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to be logically inconsistent, and multiple components or steps can be combined or divided into one.

[0068] For example, in the above-described embodiment, the configuration and operation of the information processing device 10 may be distributed among multiple computers that can communicate with each other. Also, for example, an embodiment in which some or all of the components of the information processing device 10 are provided in a mobile object 20 is possible.

[0069] In the present embodiment, an example has been shown in which the estimated shooting position and angle of view are determined using the camera 23 and the VPS, but the present invention is not limited to this. Any method may be used to determine the estimated shooting position and angle of view, such as a method using multiple GPSs or SLAM technology.

[0070] (Industrial Applicability) The mapping technology according to this embodiment can be widely used for geological surveys, mineral type classification, plant type classification, understanding plant growth conditions, and inspecting defects and abnormalities in structures. In addition, it can also be applied to security and disaster prevention fields, such as monitoring topographical changes during disasters and maintaining infrastructure facilities. Furthermore, it can also be used for geological surveys on planets other than Earth, and for inspecting structures such as artificial satellites and space stations in outer space, and is a technology that will play a major role in space projects and exploration missions. Thus, this mapping technology is expected to be used in a variety of industrial fields, from the global environment to outer space. [Explanation of symbols]

[0071] 1 System 10. Information processing equipment 11 Control section 12 Storage section 13 Input section 14 Output section 15 Communications Department 20 Mobile 21 Control section 22 Memory section 23 Camera 24 Detection camera 25 Communications Department 30 Network 101 3D Models 102 First detection image 103 Estimated shooting location 104, 105, 106, 107 auxiliary lines 108 Characteristic parts 109 Plot Group 112 Second detection image 113 Estimated shooting location 114, 115, 116, 117 auxiliary lines 118 Characteristic parts 119 Plot Group

Claims

1. A mapping method executed by an information processing device, Placing a three-dimensional model of the object in a virtual space; acquiring a first image of the object captured by a camera; acquiring a first detection image of the object photographed simultaneously with the first image by a detection camera arranged in parallel with the camera at a predetermined distance; determining an estimated shooting position and a first estimated angle of view of the camera when capturing the first image based on the first image and the three-dimensional model; determining an estimated photographing position of the detection camera when capturing the first detection image based on the estimated photographing position of the camera when capturing the first image and the predetermined distance; Detecting a first feature portion from the first detection image; determining, in the virtual space, a position of the first characteristic portion on the three-dimensional model based on an intersection point between a straight line extending from an estimated shooting position of the detection camera at the time of capturing the first detection image to the first characteristic portion disposed within the first estimated angle of view and the three-dimensional model; A mapping method including:

2. 2. The mapping method of claim 1, further comprising: acquiring a second image of the object photographed by the camera a predetermined period after the first detection image is photographed; acquiring a second detection image of the object captured simultaneously with the second image by the detection camera; determining an estimated shooting position and a second estimated angle of view of the camera when capturing the second image based on the second image and the three-dimensional model; determining an estimated photographing position of the detection camera when capturing the second detection image based on the estimated photographing position of the camera when capturing the second image and the predetermined distance; Detecting a second feature portion from the second detection image; determining, in the virtual space, a position of the second characteristic portion on the three-dimensional model based on an intersection point between a straight line extending from a capturing position of the detection camera at the time of capturing the second detection image and the second characteristic portion disposed within the second estimated angle of view, and the three-dimensional model; plotting the first feature and the second feature on the three-dimensional model; A mapping method including:

3. 3. The mapping method of claim 2, further comprising: A mapping method in which, when the position of the first feature portion on the three-dimensional model is the same as the position of the second feature portion on the three-dimensional model, a difference based on the first feature portion and the second feature portion is plotted on the three-dimensional model.

4. 2. The mapping method of claim 1, comprising: The mapping method, wherein the detection camera includes at least one of a visible light camera, an infrared light camera, a multispectral camera, and a hyperspectral camera.

5. 3. The mapping method of claim 2, further comprising: A mapping method including detecting the first feature portion and the second feature portion based on the first detection image and the second detection image and a learning model.

6. 2. The mapping method of claim 1, comprising: A mapping method, wherein the first feature portion is a portion where a predetermined object or a predetermined defect included in the first detection image is photographed.

7. 7. A mapping method according to claim 6, comprising: A mapping method, wherein the predetermined object includes at least one of water, a predetermined mineral, and a predetermined plant.

8. 7. A mapping method according to claim 6, comprising: The mapping method, wherein the predetermined defect includes a defect or flaw in a structure.

9. 2. The mapping method of claim 1, comprising: A mapping method, wherein the first detection image is an image taken by the detection camera mounted on a moving body.

10. An information processing device including a control unit, The control unit A three-dimensional model of the object is placed in the virtual space, acquiring a first image of the object photographed by a camera; a detection camera arranged in parallel with the camera at a predetermined distance from the camera acquires a first detection image of the object photographed simultaneously with the first image; determining an estimated shooting position and a first estimated angle of view of the camera when capturing the first image based on the first image and the three-dimensional model; determining an estimated photographing position of the detection camera when the first detection image was captured based on the estimated photographing position of the camera when the first image was captured and the predetermined distance; Detecting a first feature portion from the first detection image; an information processing device that determines, in the virtual space, the position of the first feature on the three-dimensional model based on the intersection of a straight line from the estimated shooting position of the detection camera at the time of shooting the first detection image toward the first feature located within the first estimated angle of view and the three-dimensional model.

11. On the computer, Placing a three-dimensional model of the object in a virtual space; acquiring a first image of the object captured by a camera; acquiring a first detection image of the object photographed simultaneously with the first image by a detection camera arranged in parallel with the camera at a predetermined distance; determining an estimated shooting position and a first estimated angle of view of the camera when capturing the first image based on the first image and the three-dimensional model; determining an estimated photographing position of the detection camera when capturing the first detection image based on the estimated photographing position of the camera when capturing the first image and the predetermined distance; Detecting a first feature portion from the first detection image; determining, in the virtual space, a position of the first characteristic portion on the three-dimensional model based on an intersection point between a straight line extending from an estimated shooting position of the detection camera at the time of capturing the first detection image to the first characteristic portion disposed within the first estimated angle of view and the three-dimensional model; A program that executes the following.

Images