Information processing device

Abstract

A training unit 13 performs training using, as teacher data, image data of a captured image of a sample subject and depth information (depth information indicating a depth calculated or measured in advance) indicating the depth in the captured image, and generates a depth estimation trained model for estimating the depth in the image from the image data. Furthermore, the training unit 13 performs training using, as teacher data, depth information obtained by inputting the image data to the depth estimation trained model and depth information indicating a depth calculated or measured in advance (that is, depth information corresponding to a ground truth value), and generates a confidence estimation trained model for estimating the confidence of the estimated depth information. An estimation unit 14 estimates the imaging position and the imaging direction of an imaging device (that is, the orientation of the imaging device) from a plurality of pieces of image data indicating a subject image captured by the imaging device. A generation unit 15 inputs image data indicating a subject image captured by the imaging device to a trained model stored in the storage unit 12, and generates depth information and confidence that correspond to the image data. A correction unit 16 corrects the imaging position and the imaging direction that are estimated by the estimation unit 14, on the basis of the depth information and the confidence that have been generated by the generation unit 15.

Description

Information processing device

[0004]

[0001] The present invention relates to a technique for estimating an imaging position and an imaging direction by an imaging device.

[0002] As a method for estimating the imaging position and the imaging direction of an imaging device based on a plurality of image data captured by the imaging device, techniques such as Simultaneous Localization and Mapping (SLAM) and Structure from Motion (SfM) are known. The imaging position and the imaging direction of the imaging device estimated for each image data are used, for example, to generate a three-dimensional image from two-dimensional images by a technique called Gaussian Splatting.

[0003] Patent Document 1 discloses a first management unit that manages a first database storing a plurality of imaging information each including image data, position data representing an imaging position where the image data was captured, and depth data representing a distance from the imaging position represented by the position data to an object, an acquisition unit that acquires target image data, a position estimation unit that estimates target position data representing an imaging position where the target image data was captured based on the target image data, a depth estimation unit that estimates target depth data representing a distance from the imaging position represented by the target position data to an object based on the target image data, a registration unit that registers new imaging information including the target image data, the target position data, and the target depth data in the first database, and a display control unit that causes a display device to display map data representing a map of an environment where the image data was captured. The display control unit causes at least one of the image data and the depth data included in designated imaging information among the plurality of imaging information stored in the first database to be displayed in correspondence with a pixel position corresponding to the position data included in the designated imaging information in the map data. An information processing device is disclosed.

[0004] Japanese Patent Application Laid-Open No. 2022-105442

[0005] When estimating the imaging position and the imaging direction of an imaging device using only image data, the accuracy of the estimation is poor.

[0006] Therefore, the present invention aims to improve the accuracy of estimating the imaging position and imaging direction by an imaging device, even when using image data.

[0007] To solve the above problems, the present invention provides an information processing device characterized by comprising: an estimation unit that estimates the imaging position and imaging direction by an imaging device from image data showing an image captured by the imaging device; a generation unit that generates depth information in three-dimensional space and a reliability level of said depth information from the image data; and a correction unit that corrects the estimated imaging position and imaging direction based on the generated depth information and the reliability level.

[0008] According to the present invention, even when using image data, the accuracy of estimating the relative imaging position and imaging direction by the imaging device can be improved.

[0009] This is a block diagram showing an example of the hardware configuration of an information processing device 10 according to one embodiment of the present invention. This is a block diagram showing an example of the hardware configuration of an imaging device 20 according to the same embodiment. This is a block diagram showing an example of the functional configuration of the information processing device 10. This is a flowchart showing an example of the operation of the information processing device 10.

[0010] [Embodiment] [Configuration] An embodiment of the present invention will be described below. Figure 1 is a diagram showing the hardware configuration of an information processing device 10 according to an embodiment of the present invention. Physically, the information processing device 10 is configured as a computer including a processor 1001, memory 1002, storage 1003, communication device 1004, and a bus connecting these. Each of these devices operates on power supplied from a battery (not shown). In the following description, the word "device" can be read as a circuit, device, unit, etc. The hardware configuration of the information processing device 10 may be configured to include one or more of the devices shown in Figure 1, or it may be configured to omit some of the devices. Alternatively, multiple devices with different housings may be connected to each other via communication to constitute the information processing device 10.

[0011] Each function in the information processing device 10 is realized by loading predetermined software (programs) onto hardware such as the processor 1001 and memory 1002, which allows the processor 1001 to perform calculations, control communication by the communication device 1004, and control at least one of data reading and writing in the memory 1002 and storage 1003.

[0012] The processor 1001 controls the entire computer, for example, by running an operating system. The processor 1001 may consist of a central processing unit (CPU) that includes interfaces with peripheral devices, control units, arithmetic units, registers, and so on.

[0013] The processor 1001 reads programs (program code), software modules, data, etc., from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes accordingly. The program used is one that causes the computer to execute at least a part of the operations described later. Functional blocks of the information processing device 10 may be stored in the memory 1002 and implemented by control programs that run on the processor 1001. Various processes may be executed by one processor 1001, but may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The program may also be transmitted to the information processing device 10 via a telecommunications line.

[0014] The memory 1002 is a computer-readable recording medium and may consist of at least one of the following: ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. The memory 1002 may also be called a register, cache, main memory, etc. The memory 1002 can store executable programs (program code), software modules, etc., for carrying out the method according to this embodiment.

[0015] The storage 1003 is a computer-readable recording medium and may consist of at least one of the following: an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disc, a digital multipurpose disc, a Blu-ray® disc), a smart card, flash memory (e.g., a card, a stick, a key drive), a floppy® disk, a magnetic strip, etc. The storage 1003 may also be called an auxiliary storage device.

[0016] The communication device 1004 is hardware (transmitting / receiving device) for communication between computers, and is also called a network device, network controller, network card, or communication module. Image data showing a two-dimensional image captured from multiple imaging positions in a certain imaging direction by an imaging device described later is input to the information processing device 10 via the communication device 1004.

[0017] Each device, such as the processor 1001 and the memory 1002, is connected by a bus for communicating information. The bus may be configured using a single bus, or different buses may be used for each device.

[0018] The information processing device 10 may include hardware such as a microprocessor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array), and some or all of each functional block may be realized by this hardware. For example, the processor 1001 may be implemented using at least one of these hardware components.

[0019] Figure 2 shows an example of the electrical configuration of the imaging device 20. The imaging device can be any device that captures images of a subject at multiple imaging positions and in any imaging direction, but below we will describe a self-propelled imaging device 20.

[0020] Physically, the imaging device 20 is configured as a computer device including a processor 2001, memory 2002, storage 2003, communication device 2004, user interface device 2005, imaging device 2006, wheel control mechanism 2008, and a bus connecting these. In the following description, the term "device" can be read as a circuit, device, unit, etc. The electrical configuration of the imaging device 20 may include one or more of the devices shown in the figure, or it may be configured without some of the devices. The processor 2001, memory 2002, storage 2003, and communication device 2004 are the same hardware as the processor 1001, memory 1002, storage 1003, and communication device 1004 described above. The communication device 2004 transmits image data showing a two-dimensional image captured by the imaging device 2006 from multiple imaging positions in an arbitrary imaging direction to the communication device 1004 of the information processing device 10.

[0021] The user interface device 2005 includes an input device (e.g., keys, microphone, switch, button, etc.) that receives input from the user and an output device (e.g., display, speaker, LED lamp, etc.) that provides output to the user. The input device and the output device may be configured as an integrated unit (e.g., a touchscreen).

[0022] The imaging device 2006 is a device that generates two-dimensional image data representing the captured image.

[0023] The wheel control mechanism 2007 is a mechanism for controlling the wheels and includes a motor mechanism for rotating the wheels and a steering mechanism for changing the orientation of the wheel axles.

[0024] Next, Figure 3 is a block diagram showing the functional configuration of the information processing device 10. In the information processing device 10, the processor 1001 reads programs and the like from the storage 1003 into the memory 1002 and executes them, thereby realizing the functions of the acquisition unit 11, the storage unit 12, the learning unit 13, the estimation unit 14, the generation unit 15, the correction unit 16, and the output unit 17.

[0025] The imaging device 20 captures a predetermined subject from multiple different imaging positions and in different imaging directions. The acquisition unit 11 acquires image data showing the captured images taken by the imaging device 20.

[0026] The learning unit 13 uses image data of a sample subject and depth information indicating the depth in that image (depth information indicating the depth determined by prior calculation or measurement) as training data to perform machine learning and generate a depth estimation learning model that estimates the depth of each pixel in three-dimensional space from the image data. When arbitrary image data is input to this depth estimation learning model, the depth information of the image data is output.

[0027] Furthermore, the learning unit 13 uses the depth information obtained by inputting image data to this depth estimation learning model, along with the depth information indicating the depth identified by prior calculation or measurement (i.e., depth information corresponding to the correct value), as training data to perform machine learning and generate a confidence estimation learning model that estimates the confidence level of the estimated depth information. This confidence level is estimated, for example, using the reciprocal of the difference between the depth information estimated by the depth estimation trained model and the depth information that is the correct value. In other words, the confidence level of the depth information = 1 / {(estimated depth information) - (depth information that is the correct value)}. This confidence level is generated for each pixel that makes up the image data.

[0028] The depth estimation learning model and the confidence estimation learning model described above may be generated as a single integrated learning model, or they may be generated as separate learning models. The generated depth estimation learning model and confidence estimation learning model, that is, the trained models generated through training to generate depth information in three-dimensional space and the confidence level of that depth information, are stored in the memory unit 12.

[0029] The estimation unit 14 estimates the imaging position and imaging direction (i.e., the orientation of the imaging device) of the imaging device 20 from a plurality of image data showing images of the subject captured by the imaging device. Known methods for estimating the imaging position and imaging direction of the imaging device from a plurality of image data captured by the imaging device include, for example, Simultaneous Localization and Mapping (SLAM) and Structure from Motion (SfM). Therefore, the estimation unit 14 estimates the imaging position and imaging direction of the imaging device using these well-known techniques such as SLAM or SfM.

[0030] The generation unit 15 generates depth information in three-dimensional space and the confidence level of that depth information from image data showing an image of a subject captured by the imaging device 20. Specifically, the generation unit 15 inputs image data showing an image of a subject captured by the imaging device 20 into a learning model stored in the storage unit 12, and generates depth information and a confidence level corresponding to that image data.

[0031] The correction unit 16 corrects the imaging position and imaging direction estimated by the estimation unit 14 based on the depth information and reliability generated by the generation unit 15. The correction unit 16 defines a set of pixels whose reliability, generated by the generation unit 15 using a learning model, is equal to or greater than a threshold (first threshold) as high-reliability pixels, and corrects the estimated imaging position and imaging direction using the depth information corresponding to each of these high-reliability pixels. More specifically, the correction unit 16 calculates scale information in three-dimensional space for each pixel based on the depth information corresponding to the high-reliability pixels, and applies the average value of the scale information calculated for each pixel as the scale information for each pixel in the entire image area including the high-reliability pixels. Then, the correction unit 16 corrects the imaging position and imaging direction in three-dimensional space corresponding to the entire image area according to this scale information.

[0032] The output unit 17 outputs the imaging position and imaging direction of the imaging device 20, which have been corrected by the correction unit 16 for each image data. The imaging position and imaging direction of the imaging device 20 in each image data are used, for example, to generate a 3D image from 2D images taken from multiple viewpoints using Gaussian Splatting. Gaussian Splatting is a technique that represents a 3D space using a Gaussian-like spreading point cloud, and is realized by learning the position, color, diffusion direction, scale information, and transparency of the point cloud using 2D image data as training data.

[0033] [Operation] Next, the operation of this embodiment will be described. Figure 4 is a flowchart showing an example of the operation of the information processing device 10. It is assumed that, before the processing shown in Figure 4 is started, the learning model (depth estimation learning model and confidence estimation learning model) generated by the learning unit 13 performing the learning described above is stored in the storage unit 12.

[0034] In Figure 4, the acquisition unit 11 acquires multiple image data showing images of the subject captured by the imaging device 20 (step S11).

[0035] The estimation unit 14 estimates the imaging position and imaging direction (i.e., the orientation of the imaging device 20) of the imaging device 20 from a plurality of image data showing images of the subject captured by the imaging device 20 (step S12).

[0036] The generation unit 15 generates depth information in three-dimensional space and the reliability of that depth information from image data showing an image of the subject captured by the imaging device 20 (step S13).

[0037] The correction unit 16 corrects the imaging position and imaging direction estimated by the estimation unit 14 based on the depth information and reliability generated by the generation unit 15 (step S14).

[0038] The output unit 17 outputs the imaging position and imaging direction of the imaging device, which have been corrected by the correction unit 16 for the image data, in correspondence with the image data (step S15).

[0039] According to the embodiments described above, by correcting the imaging position and imaging direction of the imaging device using highly reliable depth information in the captured image, it is possible to improve the accuracy of estimating the imaging position and imaging direction of the imaging device, even when using only image data.

[0040] [Modifications] The present invention is not limited to the embodiments described above. The embodiments described above may be modified as follows. Furthermore, two or more of the following modifications may be combined and implemented.

[0041] [Modification 1] A region in which pixels with high reliability of depth information are locally concentrated may be defined as a high-reliability region, and correction may be performed using the depth information of pixels belonging to such a high-reliability region. Specifically, the correction unit 16 identifies pixels such that the distance between pixels whose depth information reliability corresponds to a first threshold or higher is within a second threshold. The set of pixels identified in this way corresponds to a high-reliability region in which pixels with high reliability of depth information are locally concentrated. The correction unit 16 then corrects the estimated imaging position and imaging direction based on the depth information corresponding to the pixels in the high-reliability region, that is, the depth information corresponding to pixels whose depth information reliability corresponds to a first threshold or higher is within a second threshold. Specifically, the correction unit 16 calculates scale information in three-dimensional space for each pixel based on the depth information corresponding to each pixel constituting the high-reliability region, applies the average value of the scale information calculated for each pixel as the scale information for each pixel in the entire image region including the high-reliability region, and corrects the imaging position and imaging direction in three-dimensional space corresponding to the entire image region according to this scale information.

[0042] [Modification 2] In the above embodiment, the correction unit 16 applied the average value of the scale information corresponding to the high-reliability pixels as the scale information for each pixel in the entire image area to correct the imaging position and imaging direction of the imaging device.

[0043] Instead of using such high-reliability pixels, the imaging position and imaging direction of the imaging device may be corrected by weighting the depth information of each pixel. Specifically, the correction unit 16 calculates scale information s based on the following equation (1), where di is the depth information of each pixel, Ci is the reliability of each pixel, and N is the total number of pixels. The correction unit 16 then applies the calculated scale information s to the entire image area, including the high-reliability pixels. In other words, the correction unit 16 corrects the imaging position and imaging direction of the imaging device in the three-dimensional space corresponding to the entire image area based on the scale information s.

[0044] [Other Modifications] The block diagrams used in the description of the above embodiments show functional unit blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. Also, the realization method of each functional block is not particularly limited. That is, each functional block may be realized using one physically or logically combined device, or two or more physically or logically separated devices may be directly or indirectly (e.g., using wired, wireless, etc.) connected and realized using these multiple devices. The functional block may be realized by combining software with the above one device or the above multiple devices.

[0045] Functions include, but are not limited to, judgment, decision, determination, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, solution, selection, selection, establishment, comparison, assumption, expectation, regarded as, notification (broadcasting), notification (notifying), communication (communicating), forwarding, configuration (configuring), reconfiguration (reconfiguring), allocation (allocating, mapping), assignment (assigning), etc. For example, a functional block (component) that functions to transmit is referred to as a transmission control unit (transmitting unit) or a transmitter. In any case, as described above, the realization method is not particularly limited.

[0046] For example, the information processing apparatus 10 in an embodiment of the present disclosure may function as a computer that performs the processing of the present disclosure.

[0047] Each aspect / embodiment described in the present disclosure may be applied to at least one of systems using LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), FRA (Future Radio Access), NR (new Radio), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (ULtra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark), and other suitable systems, and next-generation systems extended based on these. Also, multiple systems may be combined (for example, a combination of at least one of LTE and LTE-A and 5G, etc.) and applied.

[0048] The processing procedures, sequences, flowcharts, etc. of each aspect / embodiment described in the present disclosure may be reordered as long as there is no contradiction. For example, regarding the methods described in the present disclosure, the elements of various steps are presented using an exemplary order and are not limited to the specific order presented.

[0049] The input / output information, etc. may be stored in a specific location (for example, memory) or may be managed using a management table. The input / output information, etc. may be overwritten, updated, or appended. The output information, etc. may be deleted. The input information, etc. may be transmitted to other devices.

[0050] The determination may be made based on a value represented by 1 bit (0 or 1), a boolean value (true or false), or a numerical comparison (for example, comparison with a predetermined value).

[0051] Although the present disclosure has been described in detail above, it will be clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure may be implemented in modified and altered forms without departing from the intent and scope of the present disclosure as defined by the claims. Accordingly, the descriptions in the present disclosure are for illustrative purposes only and are not intended to be restrictive in any way.

[0052] Software should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., whether they are called software, firmware, middleware, microcode, hardware description languages, or by any other name. Furthermore, software, instructions, information, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technologies (such as coaxial cable, fiber optic cable, twisted pair, or digital subscriber line (DSL)) and wireless technologies (such as infrared or microwave), at least one of these wired and wireless technologies is included in the definition of a transmission medium.

[0053] The information, signals, etc., described herein may be represented using any of the following different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc., which may be referred to throughout the above description, may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof. Terms used herein and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meaning.

[0054] Furthermore, the information, parameters, etc., described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0055] In this disclosure, the phrase "based on" does not mean "based solely on" unless otherwise specified. In other words, the phrase "based on" means both "based solely on" and "based at least on."

[0056] Any reference to elements using the designations “First,” “Second,” etc., as used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Accordingly, references to the First and Second elements do not imply that only two elements may be employed, or that the First element must precede the Second element in any way.

[0057] In the above-described configuration of each device, the term "part" may be replaced with "means," "circuit," "device," etc.

[0058] Where the terms “include,” “including,” and variations thereof are used in this disclosure, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used in this disclosure is not intended to mean exclusive OR.

[0059] In this disclosure, if articles are added through translation, such as a, an, and the in English, this disclosure may include the fact that the noun following these articles is plural.

[0060] In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combine" may be interpreted similarly to "different."

[0061] 10: Information processing device, 11: Acquisition unit, 12: Storage unit, 13: Learning unit, 14: Estimation unit, 15: Generation unit, 16: Correction unit, 17: Output unit, 1001: Processor, 1002: Memory, 1003: Storage, 1004: Communication device.

Claims

1. An information processing device comprising: an estimation unit that estimates the imaging position and imaging direction by an imaging device from image data showing an image captured by the imaging device; a generation unit that generates depth information in three-dimensional space and a reliability level of said depth information from the image data; and a correction unit that corrects the estimated imaging position and imaging direction based on the generated depth information and the reliability level.

2. The information processing apparatus according to claim 1, characterized in that the generation unit generates the depth information and the confidence level using the image data and a trained model generated through training for generating depth information in three-dimensional space and the confidence level of said depth information.

3. The information processing apparatus according to claim 2, characterized in that the generation unit generates the confidence score using the reciprocal of the difference between the depth information generated using the trained model and its correct value.

4. The information processing apparatus according to claim 1, characterized in that the correction unit corrects the estimated imaging position and imaging direction using the depth information whose reliability corresponds to or exceeds a first threshold.

5. The information processing apparatus according to claim 4, characterized in that the correction unit corrects the estimated imaging position and imaging direction based on depth information corresponding to each pixel constituting the image data.

6. The information processing apparatus according to claim 4, characterized in that the correction unit corrects the estimated imaging position and imaging direction based on depth information corresponding to pixels such that the distance between pixels corresponding to a reliability of 1 or more is within 2 thresholds.

7. The information processing apparatus according to claim 1, characterized in that the correction unit calculates scale information s based on the following formula, where di is the depth information of each pixel, Ci is the confidence level, and N is the number of pixels, and corrects the estimated imaging position and imaging direction based on the calculated scale information s.

Images