Mapping method, information processing device, and program

Abstract

The present invention provides a mapping method, an information processing device, and a program for mapping the characteristic features of detected objects onto a three-dimensional model. [Solution] The information processing device acquires a first image of an object captured by a camera, acquires a first detection image of an object simultaneously captured by a detection camera arranged in parallel at a predetermined distance from the camera, determines the estimated shooting position of the camera and the first estimated field of view at the time the first image was captured based on the first image and a 3D model, determines the estimated shooting position of the detection camera at the time the first detection image was captured based on the estimated shooting position of the camera and a predetermined distance, detects a first feature portion from the first detection image, and determines the position of the first feature portion on the 3D model in a virtual space based on the intersection of a line from the estimated shooting position of the detection camera at the time the first detection image was captured to the first feature portion located within the first estimated field of view and the 3D model.

Description

Technical Field

[0001] The present disclosure relates to a mapping method on a three-dimensional model of a photographed image, an information processing apparatus, and a program.

Background Art

[0002] Conventionally, a technique has been disclosed in which an aircraft equipped with a camera generates a three-dimensional model of an inspection object based on a plurality of images of the inspection object taken, and displays a three-dimensional model in which abnormal positions of the inspection object in the images are mapped (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in Patent Document 1, it is premised that in one inspection, one camera takes a plurality of images directly below in the vertical direction. For example, when the aircraft has a plurality of cameras, the efficiency of estimating the position and orientation of the cameras and the case where the inspection is carried out a plurality of times are not sufficiently considered. Thus, there is room for improvement in the mapping technology on the three-dimensional model of the photographed image.

[0005] In view of such circumstances, an object of the present disclosure is to improve the mapping technology on the three-dimensional model of the photographed image.

Means for Solving the Problems

[0006] (1) The mapping method according to an embodiment of the present disclosure is a mapping method executed by an information processing apparatus, placing a three-dimensional model of an object in a virtual space, and To acquire a first image of the object captured by the camera, A first detection image of the object captured simultaneously with the first image is acquired by a detection camera arranged in parallel at a predetermined distance from the aforementioned camera, Based on the first image and the three-dimensional model, the estimated shooting position of the camera and the first estimated field of view at the time the first image was taken are determined. Based on the estimated shooting position of the camera at the time of capturing the first image and the predetermined distance, the estimated shooting position of the detection camera at the time of capturing the first detection image is determined. To detect a first feature portion from the first detection image, In the virtual space, the position of the first feature portion on the 3D model is determined based on the intersection of a straight line from the estimated shooting position of the detection camera at the time of capturing the first detection image, to the first feature portion located within the first estimated field of view, and the 3D model. Includes.

[0007] (2) A mapping method according to one embodiment of the present disclosure is the mapping method described in (1), further, After a predetermined period following the capture of the first detection image, a second image of the object captured by the camera is acquired. The detection camera acquires a second detection image of the object captured simultaneously with the second image, Based on the second image and the three-dimensional model, the estimated shooting position of the camera and the second estimated field of view at the time the second image was taken are determined. Based on the estimated shooting position of the camera at the time of capturing the second image and the predetermined distance, the estimated shooting position of the detection camera at the time of capturing the second detection image is determined. To detect a second feature portion from the second detection image, In the virtual space, the position of the second feature portion on the 3D model is determined based on the intersection of a straight line from the shooting position of the detection camera at the time of capturing the second detection image, to the second feature portion located within the second estimated field of view, and the 3D model. Plotting the first feature portion and the second feature portion on the three-dimensional model, Includes.

[0008] (3) A mapping method according to one embodiment of the present disclosure is the mapping method described in (2), If the position of the first feature portion on the 3D model is the same as the position of the second feature portion on the 3D model, the difference between the first feature portion and the second feature portion is plotted on the 3D model.

[0009] (4) A mapping method according to one embodiment of the present disclosure is a mapping method according to any one of (1) to (3), The detection camera includes at least one of a visible light camera, an infrared light camera, a multispectral camera, and a hyperspectral camera.

[0010] (5) A mapping method according to one embodiment of the present disclosure is a mapping method described in (2) or (3), further, This includes detecting the first feature portion and the second feature portion based on the first detection image, the second detection image, and a learning model.

[0011] (6) A mapping method according to one embodiment of the present disclosure is a mapping method according to any one of (1) to (5), The first characteristic portion is the portion in which a predetermined object or predetermined defect is captured in the first detection image.

[0012] (7) A mapping method according to one embodiment of the present disclosure is the mapping method described in (6), The predetermined object includes at least one of water, a predetermined mineral, and a predetermined plant.

[0013] (8) The mapping method according to an embodiment of the present disclosure is the mapping method described in (6), wherein the predetermined defect includes a defect or damage related to a structure.

[0014] (9) The mapping method according to an embodiment of the present disclosure is the mapping method according to any one of (1) to (8), wherein the first detection image is an image captured by the detection camera mounted on the moving body.

[0015] (10) An information processing apparatus according to an embodiment of the present disclosure is an information processing apparatus including a control unit, wherein the control unit disposes a three-dimensional model of an object in a virtual space, acquires a first image related to the object captured by a camera, acquires a first detection image related to the object captured simultaneously with the first image by a detection camera arranged in parallel at a predetermined distance from the camera, determines an estimated shooting position and a first estimated shooting angle of the camera at the time of shooting the first image based on the first image and the three-dimensional model, determines an estimated shooting position of the detection camera at the time of shooting the first detection image based on the estimated shooting position of the camera at the time of shooting the first image and the predetermined distance, detects a first feature portion from the first detection image, and determines a position of the first feature portion on the three-dimensional model based on an intersection of a straight line directed from the estimated shooting position of the detection camera at the time of shooting the first detection image within the first estimated shooting angle to the first feature portion and the three-dimensional model in the virtual space.

[0016] (11) A program according to an embodiment of the present disclosure causes a computer to Placing a 3D model of an object in a virtual space, To acquire a first image of the object captured by the camera, A first detection image of the object captured simultaneously with the first image is acquired by a detection camera arranged in parallel at a predetermined distance from the aforementioned camera, Based on the first image and the three-dimensional model, the estimated shooting position of the camera and the first estimated field of view at the time the first image was taken are determined. Based on the estimated shooting position of the camera at the time of capturing the first image and the predetermined distance, the estimated shooting position of the detection camera at the time of capturing the first detection image is determined. To detect a first feature portion from the first detection image, In the virtual space, the position of the first feature portion on the 3D model is determined based on the intersection of a straight line from the estimated shooting position of the detection camera at the time of capturing the first detection image, to the first feature portion located within the first estimated field of view, and the 3D model. Make it run. [Effects of the Invention]

[0017] According to one embodiment of this disclosure, the mapping technique for captured images onto a three-dimensional model is improved. [Brief explanation of the drawing]

[0018] [Figure 1] This block diagram shows a schematic configuration of a system according to one embodiment of the present disclosure. [Figure 2] This is a block diagram illustrating the schematic configuration of an information processing device. [Figure 3] This is a block diagram showing the schematic configuration of a mobile unit. [Figure 4] This is a schematic diagram showing the relative positions of the camera on the moving object and the detection camera. [Figure 5] This is a flowchart illustrating the first operational example of the information processing device. [Figure 6]This is a schematic diagram showing an example of a 3D model display. [Figure 7A] This flowchart shows a second example of the operation of the information processing device. [Figure 7B] This flowchart shows a second example of the operation of the information processing device. [Figure 8] This is a schematic diagram showing an example of a 3D model display. [Modes for carrying out the invention]

[0019] The embodiments of this disclosure will be described below.

[0020] (Summary of the embodiment) Referring to Figure 1, the overview and configuration of System 1 according to this embodiment will be described.

[0021] The system 1 according to this embodiment comprises an information processing device 10 and a mobile device 20. The information processing device 10 and the mobile device 20 are communicated with a network 30, which includes, for example, a mobile communication network and the Internet.

[0022] The information processing device 10 is any device used by a user (e.g., a facility manager, infrastructure inspector, vegetation management surveyor, farm manager, etc.). For example, a general-purpose electronic device such as a personal computer, smartphone, tablet terminal, or wearable device, or a dedicated electronic device, can be used as the information processing device 10. Alternatively, the information processing device 10 may be a server device installed in a data center or the like. The information processing device 10 can communicate with the mobile device 20 via the network 30. Although Figure 1 shows an example where system 1 has one information processing device 10, it is not limited to this. System 1 may have two or more information processing devices 10.

[0023] The mobile body 20 is a mobile body equipped with a camera for capturing images or video (hereinafter collectively referred to as images) and a detection camera, and may be, for example, an unmanned aerial vehicle. An unmanned aerial vehicle includes a drone. However, the mobile body 20 is not limited to this and may be a car, ship, robot, etc., equipped with a camera and a detection camera.

[0024] First, an overview of this embodiment will be described, and details will be described later. The mapping method according to this embodiment is performed by an information processing device 10. The information processing device 10 first places a 3D model of the object in a virtual space. The information processing device 10 also acquires a first image of the object captured by a camera. The information processing device 10 also acquires a first detection image of the object captured simultaneously with the first image by a detection camera placed in parallel at a predetermined distance from the camera. The information processing device 10 also determines the estimated shooting position of the camera and the first estimated field of view at the time the first image was captured, based on the first image and the 3D model. The information processing device 10 also determines the estimated shooting position of the detection camera at the time the first detection image was captured, based on the estimated shooting position of the camera at the time the first image was captured and a predetermined distance. The information processing device 10 also detects a first feature portion from the first detection image. Furthermore, the information processing device 10 determines the position of the first feature portion on the 3D model in the virtual space, based on the intersection of a straight line from the estimated shooting position of the detection camera at the time of capturing the first detection image to the first feature portion located within the first estimated field of view, and the 3D model.

[0025] As described above, according to this embodiment, the estimated shooting position of the camera and the first estimated field of view are determined based on the first image and the 3D model. Since the camera's position and orientation estimation is performed by the information processing device 10 rather than the mobile body 20, the processing load on the mobile body 20 can be reduced. Therefore, for example, if the mobile body 20 is a drone, the power consumption of the mobile body 20 can be suppressed, and a longer flight time can be secured. Also, since the camera's position and orientation estimation is performed by the information processing device 10, real-time processing is possible. Furthermore, according to this embodiment, the estimated shooting position of the detection camera is determined based on the estimated shooting position of the camera. Therefore, even if the mobile body 20 is equipped with multiple cameras, the estimated shooting positions can be appropriately determined. Thus, according to this embodiment, the mapping technology of the captured image on the 3D model is improved.

[0026] Next, we will describe each component of System 1 in detail.

[0027] (Configuration of information processing device) As shown in Figure 2, the information processing device 10 comprises a control unit 11, a storage unit 12, an input unit 13, an output unit 14, and a communication unit 15.

[0028] The control unit 11 includes at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU (central processing unit) or GPU (graphics processing unit), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). The control unit 11 controls each part of the information processing device 10 and executes processes related to the operation of the information processing device 10.

[0029] The storage unit 12 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or at least two combinations thereof. The semiconductor memory is, for example, RAM (random access memory) or ROM (read-only memory). The RAM is, for example, SRAM (static random access memory) or DRAM (dynamic random access memory). The ROM is, for example, EEPROM (electrically erasable programmable read-only memory). The storage unit 12 functions, for example, as a main memory, auxiliary memory, or cache memory. The storage unit 12 stores data used for the operation of the information processing device 10 and data obtained by the operation of the information processing device 10.

[0030] The input unit 13 includes at least one input interface. The input interface may be, for example, a physical key, a capacitive key, a pointing device, or a touchscreen integrated with a display. Alternatively, the input interface may be, for example, a sound sensor that accepts voice input, or a camera that accepts gesture input. The input unit 13 accepts operations to input data used for the operation of the information processing device 10. Instead of being integrated into the information processing device 10, the input unit 13 may be connected to the information processing device 10 as an external input device. Any connection method can be used, for example, USB (Universal Serial Bus), HDMI (registered trademark) (High-Definition Multimedia Interface), or Bluetooth (registered trademark).

[0031] The output unit 14 includes at least one output interface. The output interface is, for example, a display that outputs information as an image, or a speaker that outputs information as sound. The display is, for example, an LCD (liquid crystal display) or an organic EL (electroluminescence) display. The output unit 14 displays and outputs data obtained by the operation of the information processing device 10. Instead of being provided in the information processing device 10, the output unit 14 may be connected to the information processing device 10 as an external output device. Any connection method can be used, for example, USB, HDMI (registered trademark), or Bluetooth (registered trademark).

[0032] The communication unit 15 includes at least one external communication interface. The communication interface may be either a wired or wireless communication interface. In the case of wired communication, the communication interface may be, for example, a LAN (Local Area Network) interface or a USB (Universal Serial Bus) interface. In the case of wireless communication, the communication interface may be, for example, an interface compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th generation), or 5G (5th generation), or an interface compatible with short-range wireless communication such as Bluetooth (registered trademark). The communication unit 15 receives data used for the operation of the information processing device 10 and transmits data obtained by the operation of the information processing device 10.

[0033] The functions of the information processing device 10 are realized by executing the program according to this embodiment on a processor corresponding to the control unit 11. In other words, the functions of the information processing device 10 are realized by software. The program causes the computer to perform the operations of the information processing device 10, thereby causing the computer to function as the information processing device 10. That is, the computer functions as the information processing device 10 by performing the operations of the information processing device 10 according to the program.

[0034] In this embodiment, the program can be recorded on a computer-readable recording medium. The computer-readable recording medium includes non-temporary computer-readable media, such as magnetic recording devices, optical discs, magneto-optical recording media, or semiconductor memory. The program can be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs (digital versatile discs) or CD-ROMs (compact disc read-only memory) on which the program is recorded. Alternatively, the program may be distributed by storing it on the storage of an external server and transmitting it from the external server to other computers. The program may also be provided as a program product.

[0035] Some or all of the functions of the information processing device 10 may be implemented by a dedicated circuit corresponding to the control unit 11. In other words, some or all of the functions of the information processing device 10 may be implemented by hardware.

[0036] (Configuration of the mobile unit) As shown in Figure 3, the mobile unit 20 comprises a control unit 21, a storage unit 22, a camera 23, a detection camera 24, and a communication unit 25.

[0037] The control unit 21 includes at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU (central processing unit) or GPU (graphics processing unit), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). The control unit 21 controls each part of the mobile body 20 and executes processes related to the operation of the mobile body 20.

[0038] The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or at least two combinations thereof. The semiconductor memory is, for example, RAM (random access memory) or ROM (read-only memory). The RAM is, for example, SRAM (static random access memory) or DRAM (dynamic random access memory). The ROM is, for example, EEPROM (electrically erasable programmable read-only memory). The storage unit 22 functions, for example, as a main memory, auxiliary memory, or cache memory. The storage unit 22 stores data used for the operation of the mobile body 20 and data obtained by the operation of the mobile body 20.

[0039] Camera 23 is an imaging device that captures images (including moving images as described above). For example, camera 23 may be an RGB camera that captures images using three basic colors: red (R), green (G), and blue (B). In this case, camera 23 acquires information on each color through sensors and filters, and generates an image by integrating this information using image processing technology. For example, CMOS (Complementary Metal-Oxide-Semiconductor) and CCD (Charge-Coupled Device) sensors can be used. When using a CMOS sensor, low power consumption and high-speed processing can be achieved. When using a CCD sensor, high image quality and low noise can be achieved.

[0040] The detection camera 24 is an imaging device that captures images (including moving images as described above, hereinafter also referred to as detection images) for detecting feature parts. The detection camera 24 includes at least one of a visible light camera, an infrared camera, a multispectral camera, and a hyperspectral camera. A visible light camera is an imaging device that detects visible light with wavelengths from approximately 380 nm to 750 nm. An infrared camera is an imaging device that detects infrared light (750 nm or longer) with wavelengths longer than visible light. Infrared cameras can be used in particular for shooting in dark places and for sensing the temperature of objects. Either a cooled or uncooled infrared camera may be used. A multispectral camera is a camera that can simultaneously capture light of multiple different wavelengths. Multispectral cameras can also capture wavelengths other than visible light, such as near-infrared and ultraviolet light. A hyperspectral camera is an imaging device that subdivides light over a wide wavelength range and acquires information from a large number of spectral bands. Detailed spectral data from each pixel makes it possible to detect differences in materials, anomalies, etc., that cannot be captured by ordinary cameras. The detection camera 24 is positioned parallel to the camera at a predetermined distance apart. Figure 4 is a schematic diagram showing the positional relationship between the camera 23 mounted on the mobile body 20 and the detection camera 24. As shown in Figure 4, the camera 23 and the detection camera 24 are positioned parallel to each other, for example, at the bottom of the mobile body 20, at a predetermined distance d apart. In other words, the camera 23 and the detection camera 24 are positioned offset in the plane direction at a predetermined distance d apart. On the other hand, the orientations of the camera 23 and the detection camera 24 are the same. In other words, the field of view of the camera 23 and the field of view of the detection camera 24 are the same. However, the orientations of the camera 23 and the detection camera 24 may be different. In other words, the field of view of the camera 23 and the field of view of the detection camera 24 may be different. In this case, correction can be made based on the field of view of the camera 23 and the field of view of the detection camera 24.

[0041] The communication unit 25 includes at least one external communication interface. The communication interface may be either a wired or wireless communication interface. In the case of wired communication, the communication interface may be, for example, a LAN (Local Area Network) interface or a USB (Universal Serial Bus) interface. In the case of wireless communication, the communication interface may be, for example, an interface compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th generation), or 5G (5th generation), or an interface compatible with short-range wireless communication such as Bluetooth (registered trademark). The communication unit 25 receives data used for the operation of the mobile body 20 and transmits data obtained by the operation of the mobile body 20. The processing in this embodiment may or may not be real-time processing. Therefore, for example, if the mobile body 20 is an unmanned aerial vehicle and wired communication is performed via the communication interface, the communication connection may be established after the flight of the unmanned aerial vehicle and various processes may be executed.

[0042] The functions of the mobile unit 20 are realized by executing the program according to this embodiment on a processor corresponding to the control unit 21. In other words, the functions of the mobile unit 20 are realized by software. The program causes the computer to perform the actions of the mobile unit 20, thereby causing the computer to function as the mobile unit 20. That is, the computer functions as the mobile unit 20 by performing the actions of the mobile unit 20 according to the program.

[0043] Some or all of the functions of the mobile unit 20 may be implemented by a dedicated circuit corresponding to the control unit 21. In other words, some or all of the functions of the mobile unit 20 may be implemented by hardware.

[0044] (First example of operation of the information processing device) Referring to Figure 5, a first example of operation of the information processing device 10 according to this embodiment will be described.

[0045] Step S1: The control unit 11 of the information processing device 10 places a 3D model of the object in the virtual space. The 3D model of the object may be created by any method. For example, the 3D model of the object may be created by photogrammetry or the like.

[0046] Step S2: The control unit 11 acquires a first image of the object captured by the camera 23. Specifically, the control unit 11 acquires a first image of the object captured by the camera 23 via the communication unit 25 of the mobile unit 20.

[0047] Step S3: The control unit 11 acquires a first detection image of the object captured simultaneously with the first image by the detection camera 24. Specifically, the control unit 11 acquires the first detection image of the object captured by the detection camera 24 via the communication unit 25 of the mobile unit 20.

[0048] Step S4: The control unit 11 determines the estimated shooting position and first estimated field of view (orientation) of the camera 23 at the time of capturing the first image, based on the first image and the 3D model. Any method may be used to determine the estimated shooting position and first estimated field of view. For example, the control unit 11 may determine the above estimated shooting position and first estimated field of view using a Visual Positioning System (VPS). A VPS is a technique that extracts feature points from a 2D image and matches them with corresponding points on a 3D model.

[0049] Step S5: The control unit 11 determines the estimated shooting position of the detection camera 24 when capturing the first detection image, based on the estimated shooting position of the camera 23 when capturing the first image and a predetermined distance d. Here, as described above, the camera 23 and the detection camera 24 are installed in parallel at the bottom of the moving body 20, separated by a predetermined distance d. Therefore, the estimated shooting position of the detection camera 24 can be determined from the estimated shooting position of the camera 23 and the predetermined distance d. Also, the field of view when the detection camera 24 is shooting is the same as the first estimated field of view. In other words, the estimated shooting position and estimated field of view of the detection camera 24 can be determined based on the estimated shooting position and estimated field of view of the camera 23 and the positional relationship between the camera 23 and the detection camera 24 on the moving body 20.

[0050] Step S6: The control unit 11 detects a first feature portion from the first detection image. Any method can be used for the process of detecting the first feature portion. For example, the control unit 11 may detect the first feature portion based on the first detection image and a trained model. Specifically, for example, the control unit 11 may input data related to the first detection image into a trained model for image recognition, thereby obtaining data related to the first feature portion as the output of the trained model. The first feature portion may be a portion of the first detection image in which a predetermined object or a predetermined defect is captured. The predetermined object may include water, a predetermined mineral, a predetermined plant, etc. The predetermined defect may be a defect or damage related to a structure. The defect or damage related to a structure may be tile peeling, cracks, holes, corrosion, distortion, or breakage. These defects or damages can have a significant impact on the durability and safety of a building, and early detection and appropriate repair are necessary. For example, it is possible to detect the feature portion by analyzing the spectrum from information obtained with a hyperspectral camera and extracting characteristic wavelength ranges. More specifically, detection is possible using NDVI, which is known as an indicator of vegetation distribution and activity levels.

[0051] Here, a trained model is a model created by machine learning using a machine learning algorithm. A trained model may be, for example, a machine learning model built on a decision tree. Examples of machine learning models built on a decision tree include, but are not limited to, Light GBM and XGBoost. Alternatively, a trained model may be a model generated based on machine learning algorithms such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), or other deep learning methods.

[0052] Step S7: In the virtual space, the control unit 11 determines the position of the first feature portion on the 3D model based on the intersection of a straight line (hereinafter also referred to as a Ray) from the estimated shooting position of the detection camera 24 at the time of capturing the first detection image to the first feature portion located within the first estimated field of view, and the 3D model. If there are multiple intersection points between the Ray and the 3D model, the control unit 11 determines the position of the first feature portion on the 3D model based on the intersection point closest to the estimated shooting position (in other words, the first intersection point where the Ray intersects the 3D model).

[0053] The control unit 11 may further plot the first feature portion on the 3D model. That is, the control unit 11 may store the position coordinates of the first feature portion in the data of the 3D model. The control unit 11 may display the 3D model on which the first feature portion has been plotted. Any method can be used for displaying the 3D model. For example, the control unit 11 may display the 3D model through a user interface via the output unit 14 of the information processing device 10.

[0054] Figure 6 is a schematic diagram showing an example of a 3D model display. Here, for example, the object is a planter and the feature part is water. In Figure 6, the first detection image 102 is superimposed on the 3D model 101. Also in Figure 6, the estimated shooting position 103 of the detection camera 24 is superimposed. Furthermore, auxiliary lines 104 to 107 indicating the field of view of the detection camera 24 are superimposed in Figure 6. Note that the auxiliary lines 104 to 107 in Figure 6 are line segments connecting the estimated shooting position of the detection camera 24 and the four vertices of the first detection image 102. As shown in Figure 6, the feature part 108 in the first detection image 102 is mapped as a group of plots 109 on the 3D model 101 based on the intersection points mentioned above.

[0055] As described above, according to this embodiment, the estimated shooting position of the camera and the first estimated field of view are determined based on the first image and the 3D model. Since the camera's position and orientation estimation is performed by the information processing device 10 rather than the mobile body 20, the processing load on the mobile body 20 can be reduced. Therefore, for example, if the mobile body 20 is a drone, the power consumption of the mobile body 20 can be suppressed, and a longer flight time can be secured. Also, since the camera's position and orientation estimation is performed by the information processing device 10, real-time processing is possible. Furthermore, according to this embodiment, the estimated shooting position of the detection camera 24 is determined based on the estimated shooting position of the camera 23. Therefore, even if the mobile body 20 is equipped with multiple cameras, the estimated shooting positions can be appropriately determined. In this way, according to this embodiment, the mapping technology of captured images on the 3D model is improved.

[0056] (Second example of operation of the information processing device) A second example of operation of the information processing device 10 according to this embodiment will be described with reference to Figures 7A and 7B. The same reference numerals are used for operations that are the same as in the first example of operation, and their descriptions are omitted.

[0057] Step S8: Following step S7 described above, the control unit 11 acquires a second image of the object captured by the camera 23 after a predetermined period has elapsed since the first detection image was captured. Specifically, the control unit 11 acquires the second image of the object captured by the camera 23 via the communication unit 25 of the mobile unit 20. The predetermined period may be any period. For example, the predetermined period may be one hour, one day, etc.

[0058] Step S9: The control unit 11 acquires a second detection image of the object captured simultaneously with the second image by the detection camera 24. Specifically, the control unit 11 acquires the second detection image of the object captured by the detection camera 24 via the communication unit 25 of the mobile unit 20.

[0059] Step S10: The control unit 11 determines the estimated shooting position and second estimated field of view (orientation) of the camera 23 at the time of capturing the second image, based on the second image and the 3D model. Any method may be used to determine the estimated shooting position and second estimated field of view. For example, the control unit 11 may determine the above estimated shooting position and second estimated field of view using VPS.

[0060] Step S11: The control unit 11 determines the estimated shooting position of the detection camera 24 when capturing the second detection image, based on the estimated shooting position of the camera 23 when capturing the second image and a predetermined distance d. As described above, the estimated shooting position of the detection camera 24 can be determined from the estimated shooting position of the camera 23 and a predetermined distance d. Also, the field of view when the detection camera 24 is shooting is the same as the second estimated field of view.

[0061] Step S12: The control unit 11 detects a second feature portion from the second detection image. For example, the control unit 11 may detect a second feature portion based on the second detection image and a trained model. Specifically, for example, the control unit 11 may input data related to the second detection image into a trained model for image recognition, thereby obtaining data related to the second feature portion as the output of the trained model. The second feature portion may be a portion of the second detection image in which a predetermined object or a predetermined defect is captured. The predetermined object may include water, a predetermined mineral, a predetermined plant, etc. The predetermined defect may be a defect or damage related to a structure.

[0062] Step S13: In the virtual space, the control unit 11 determines the position of the second feature portion on the 3D model based on the intersection of a straight line (Ray) from the estimated shooting position of the detection camera 24 at the time of capturing the second detection image, to the second feature portion located within the second estimated field of view, and the 3D model. If there are multiple intersection points between the Ray and the 3D model, the control unit 11 determines the position of the second feature portion on the 3D model based on the intersection point closest to the estimated shooting position (in other words, the first intersection point where the Ray intersects the 3D model).

[0063] Step S14: The control unit 11 plots the first feature portion and the second feature portion on the 3D model. That is, the control unit 11 stores the position coordinates of the first feature portion and the second feature portion in the data of the 3D model. The control unit 11 may display the 3D model on which the first feature portion and the second feature portion have been plotted.

[0064] Figure 8 is a schematic diagram showing an example of a 3D model display. Components identical to those in Figure 6 are denoted by the same reference numerals and their explanations are omitted. As with Figure 6, here we will explain assuming, for example, that the object is a planter and the feature is water. In Figure 8, the second detection image 112 is superimposed on the 3D model 101. Also in Figure 8, the estimated shooting position 113 of the detection camera 24 is superimposed. Furthermore, in Figure 8, auxiliary lines 114 to 117 indicating the field of view of the detection camera 24 are superimposed. Note that the auxiliary lines 114 to 117 in Figure 8 are line segments connecting the estimated shooting position of the detection camera 24 and the four vertices of the second detection image 112. As shown in Figure 8, the feature portion 118 in the second detection image 112 is mapped as a group of plots 119 on the 3D model 101 based on the intersection points mentioned above.

[0065] Here, if the position of the first feature part on the 3D model is the same as the position of the second feature part on the 3D model, the difference based on the first and second feature parts may be plotted on the 3D model. This makes it possible to grasp the progression of anomalies, deterioration, etc., at the same location, such as whether they are expanding or being maintained. The display method of the plot may also be changed according to the changes at the same location. For example, the color, brightness, etc. of the plot may be changed according to the amount of change, difference, etc. A gradient may also be applied to the plot. In this way, the amount of change can be easily grasped visually. The gradient of change may also be determined based on the amount of change, difference, etc. The future state may be predicted based on the amount of change, difference, etc. The environment at the time of shooting (weather, temperature, humidity, shadows at the time of shooting, etc.) may also be taken into consideration. For example, based on multiple shots, the feature parts may be plotted on the 3D model taking into account the influence of the environment at the time of shooting, noise, etc.

[0066] In the second example of operation, images from two different time points were used as the detection image (first detection image) and the detection image (second detection image). However, the number of detection images is not limited to this, and detection images from three or more different time points may be used.

[0067] While this disclosure has been described based on the drawings and embodiments, it should be noted that those skilled in the art may make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of this disclosure. For example, the functions, etc., included in each component or step can be rearranged in a logically consistent manner, and multiple components or steps can be combined into one or divided into two.

[0068] For example, in the embodiment described above, it is also possible to distribute the configuration and operation of the information processing device 10 to multiple computers that can communicate with each other. Furthermore, for example, it is also possible to provide some or all of the components of the information processing device 10 on the mobile device 20.

[0069] In this embodiment, an example was shown in which the estimated shooting position and field of view are determined by using camera 23 and VPS, but this is not the only example. Any method may be used to determine the estimated shooting position and field of view, such as a method using multiple GPS devices or SLAM technology.

[0070] (Industrial applicability) The mapping technology according to this embodiment can be widely used for geological surveys, mineral classification, plant classification, understanding plant growth conditions, and inspection of structural defects and abnormalities. In addition, it can be applied to security and disaster prevention fields, such as monitoring topographic changes during disasters and maintaining infrastructure. Furthermore, it can be used for geological surveys on planets other than Earth, and for inspecting structures such as artificial satellites and space stations in outer space, making it a technology that will play a significant role in space projects and exploration missions. Thus, this mapping technology is expected to be utilized in a wide range of industrial fields, from the Earth's environment to outer space. [Explanation of Symbols]

[0071] 1 System 10 Information Processing Devices 11 Control Unit 12 Storage section 13 Input section 14 Output section 15 Communications Department 20 Mobile Units 21 Control Unit 22 Memory section 23 Cameras 24 detection cameras 25 Communications Department 30 Networks 101 3D Models 102 First detection image 103 Estimated shooting location 104, 105, 106, 107 Auxiliary lines 108 Feature section 109 plots 112 Second detection image 113 Estimated shooting location 114, 115, 116, 117 Auxiliary lines 118 Feature section 119 plots

Claims

1. A mapping method performed by an information processing device, Placing a 3D model of an object in a virtual space, To acquire a first image of the object captured by the camera, A first detection image of the object captured simultaneously with the first image is acquired by a detection camera arranged in parallel at a predetermined distance from the aforementioned camera, Based on the first image and the three-dimensional model, the estimated shooting position of the camera and the first estimated field of view at the time the first image was taken are determined. Based on the estimated shooting position of the camera at the time of capturing the first image and the predetermined distance, the estimated shooting position of the detection camera at the time of capturing the first detection image is determined. To detect a first feature portion from the first detection image, In the virtual space, the position of the first feature portion on the three-dimensional model is determined based on the intersection of a straight line from the estimated shooting position of the detection camera at the time of capturing the first detection image, to the first feature portion located within the first estimated field of view, and the three-dimensional model. A mapping method that includes this.

2. The mapping method according to claim 1, further, After a predetermined period following the capture of the first detection image, a second image relating to the object captured by the camera is acquired. The detection camera acquires a second detection image of the object captured simultaneously with the second image, Based on the second image and the three-dimensional model, the estimated shooting position of the camera and the second estimated field of view at the time the second image was taken are determined. Based on the estimated shooting position of the camera at the time of capturing the second image and the predetermined distance, the estimated shooting position of the detection camera at the time of capturing the second detection image is determined. To detect a second feature portion from the second detection image, In the virtual space, the position of the second feature portion on the three-dimensional model is determined based on the intersection of a straight line from the shooting position of the detection camera at the time of capturing the second detection image, to the second feature portion located within the second estimated field of view, and the three-dimensional model. The first feature portion and the second feature portion are plotted on the three-dimensional model, A mapping method that includes this.

3. A mapping method according to claim 2, A mapping method in which, if the position of the first feature portion on the three-dimensional model is the same as the position of the second feature portion on the three-dimensional model, the difference based on the first feature portion and the second feature portion is plotted on the three-dimensional model.

4. A mapping method according to claim 1, A mapping method wherein the detection camera includes at least one of a visible light camera, an infrared light camera, a multispectral camera, and a hyperspectral camera.

5. The mapping method according to claim 2, further A mapping method comprising detecting the first feature portion and the second feature portion based on the first detection image and the second detection image and a learning model.

6. A mapping method according to claim 1, A mapping method in which the first characteristic portion is a portion of the first detection image in which a predetermined object or predetermined defect is captured.

7. A mapping method according to claim 6, A mapping method wherein the predetermined object includes at least one of water, a predetermined mineral, and a predetermined plant.

8. A mapping method according to claim 6, The aforementioned specified defect includes defects or damage to a structure, and is a mapping method.

9. A mapping method according to claim 1, A mapping method wherein the first detection image is an image captured by the detection camera mounted on a moving object.

10. An information processing device comprising a control unit, The control unit, By placing a 3D model of the object in a virtual space, A first image of the object captured by the camera is acquired, A detection camera, positioned parallel to the aforementioned camera at a predetermined distance, acquires a first detection image of the object captured simultaneously with the first image. Based on the first image and the three-dimensional model, the estimated shooting position of the camera and the first estimated field of view at the time the first image was taken are determined. Based on the estimated shooting position of the camera at the time of capturing the first image and the predetermined distance, the estimated shooting position of the detection camera at the time of capturing the first detection image is determined. A first feature portion is detected from the first detection image. An information processing device that determines the position of the first feature portion on the three-dimensional model in the virtual space, based on the intersection of a straight line from the estimated shooting position of the detection camera at the time of capturing the first detection image to the first feature portion located within the first estimated field of view, and the three-dimensional model.

11. On the computer, Placing a 3D model of an object in a virtual space, To acquire a first image relating to the aforementioned object captured by the camera, A first detection image of the object captured simultaneously with the first image is acquired by a detection camera arranged in parallel at a predetermined distance from the aforementioned camera, Based on the first image and the three-dimensional model, the estimated shooting position of the camera and the first estimated field of view at the time the first image was taken are determined. Based on the estimated shooting position of the camera at the time of capturing the first image and the predetermined distance, the estimated shooting position of the detection camera at the time of capturing the first detection image is determined. To detect a first feature portion from the first detection image, In the virtual space, the position of the first feature portion on the three-dimensional model is determined based on the intersection of a straight line from the estimated shooting position of the detection camera at the time of capturing the first detection image, to the first feature portion located within the first estimated field of view, and the three-dimensional model. A program that executes the command.

Images