Information processing device, information processing method, and program

Abstract

The present disclosure pertains to an information processing device, an information processing method, and a program which make it possible to correct a to-be-corrected-pixel arising in the field of view of a sensor. This information processing device comprises a processing unit that: performs processing, using a learned model trained by machine learning, on at least some of a first image which is acquired by a first sensor and which expresses an object by depth information, a second image which is acquired by a second sensor and which represents an object by depth information, and a third image acquired from the first image and the second image; and corrects a to-be-corrected-pixel contained in one of the images. The present disclosure can be applied, for example, to a device that has a plurality of depth sensors.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program, and particularly to an information processing apparatus, an information processing method, and a program capable of correcting correction target pixels caused by the field of view of a sensor.

Background Art

[0002] There is a technique of concatenating depth images obtained by a plurality of depth sensors to generate a wider-angle depth image.

[0003] Patent Document 1 discloses a technique of detecting defective pixels in depth measurement data, defining depth correction of the detected defective pixels, and applying the depth correction to the depth measurement data of the detected defective pixels in order to improve the quality of a depth map.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When generating various depth images by arranging a plurality of depth sensors so that their fields of view have a predetermined relationship, the depth image to be processed may include correction target pixels such as defective pixels due to the field of view of the sensor, and it is required to correct the correction target pixels caused by the field of view of the sensor.

[0006] The present disclosure has been made in view of such a situation, and enables correction of correction target pixels caused by the field of view of a sensor.

Means for Solving the Problems

[0007] The information processing device in the first aspect of this disclosure is an information processing device comprising a processing unit that processes at least a portion of a first image showing an object acquired by a first sensor with depth information, a second image showing an object acquired by a second sensor with depth information, and a third image obtained from the first and second images using a trained model learned by machine learning, and corrects a pixel to be corrected included in any of the images.

[0008] The information processing method and program of the first aspect of this disclosure are the information processing method and program corresponding to the information processing device of the first aspect of this disclosure described above.

[0009] In the information processing apparatus, information processing method, and program of the first aspect of this disclosure, a first image showing an object acquired by a first sensor with depth information, a second image showing an object acquired by a second sensor with depth information, and at least a portion of a third image obtained from the first image and the second image are processed using a trained model learned by machine learning, and a pixel to be corrected included in any of the images is corrected.

[0010] The information processing device in the second aspect of the present disclosure includes a processing unit that processes at least a portion of an image showing an object acquired by a sensor having a first light source and a second light source using a trained model learned by machine learning, wherein the first light source and the second light source are arranged so that their light irradiation areas overlap, and the processing unit identifies a position corresponding to the overlapping portion of the irradiation area in the image as a pixel correction position, and corrects the depth information of the pixel correction position using the trained model.

[0011] The information processing method and program of the second aspect of this disclosure are the information processing method and program corresponding to the information processing device of the second aspect of this disclosure described above.

[0012] In the information processing apparatus, information processing method, and program of the second aspect of this disclosure, processing is performed using a trained model learned by machine learning on at least a portion of an image showing an object in depth information obtained by a sensor having a first light source and a second light source arranged so that the light irradiation areas overlap. The position corresponding to the overlapping portion of the irradiation areas in the image is identified as a pixel correction position, and the depth information of the pixel correction position is corrected using the trained model.

[0013] The information processing devices in the first and second aspects of this disclosure may be independent devices or internal blocks constituting a single device. [Brief explanation of the drawing]

[0014] [Figure 1] This figure shows an example configuration of a distance measuring device to which this disclosure applies. [Figure 2] This diagram illustrates the correction of the overlapping field of view of two depth sensors. [Figure 3] This figure shows an example configuration of a learning device and inference unit when using supervised learning. [Figure 4] This is a flowchart explaining the correction process. [Figure 5] This diagram illustrates the correction of the gap in the field of view of the two depth sensors. [Figure 6] This diagram illustrates the correction of the overlapping field of view of two depth sensors with different fields of view. [Figure 7] This diagram illustrates the correction of overlapping illumination areas of multiple light sources within a single depth sensor. [Figure 8] This figure shows a first example of another configuration of a ranging device to which the present disclosure applies. [Figure 9] This figure shows a second example of another configuration of a ranging device to which the present disclosure applies. [Figure 10] This figure shows a third example of another configuration of a ranging device to which the present disclosure applies. [Figure 11]A diagram showing a fourth example of another configuration of the distance measurement device to which the present disclosure is applied. [Figure 12] A diagram showing a configuration example of a system including a device that performs AI processing. [Figure 13] A block diagram showing a configuration example of an electronic device. [Figure 14] A block diagram showing a configuration example of an edge server or a cloud server. [Figure 15] A block diagram showing a configuration example of an optical sensor. [Figure 16] A block diagram showing a configuration example of a processing unit. [Figure 17] A diagram showing the flow of data between a plurality of devices.

Embodiments for Carrying Out the Invention

[0015] <1. Embodiments of the Present Disclosure>

[0016] (Configuration Example of Device) FIG. 1 is a diagram showing a configuration example of a distance measurement device to which the present disclosure is applied.

[0017] In FIG. 1, the distance measurement device 10 includes a depth sensor 11-1, a depth sensor 11-2, a distance measurement calculation unit 12-1, a distance measurement calculation unit 12-2, a stitching processing unit 13, and a distance correction unit 14. In the distance measurement device 10, the depth sensor 11-1 and the depth sensor 11-2 are arranged so as to have an overlap in the field of view (FoV: Fields of View).

[0018] The depth sensor 11-1 is a distance measurement sensor such as a ToF sensor. For example, in a ToF sensor, the time from when light emitted from a light source is reflected by an object until it is received by a light receiving element (light receiving unit) is measured, and the distance is calculated based on the known speed of light. The ToF sensor may be either a dToF (direct Time of Flight) method or an iToF (indirect Time of Flight) method.

[0019] The depth sensor 11-1 may be a structure light type sensor, a LiDAR (Light Detection and Ranging) type sensor, a stereo camera, or the like. The depth sensor 11-1 measures the target object and supplies the resulting RAW data to the distance measurement calculation unit 12-1.

[0020] The distance measurement calculation unit 12-1 performs distance calculations using the RAW data supplied from the depth sensor 11-1 based on the distance measurement calibration parameters, and supplies the resulting distance data to the object to the stitching processing unit 13. The distance measurement calculation unit 12-1 may be located within the depth sensor 11-1.

[0021] The depth sensor 11-2 is a distance measuring sensor such as a ToF sensor. The depth sensor 11-2 measures the object and supplies the resulting RAW data to the distance calculation unit 12-2.

[0022] The distance measurement calculation unit 12-2 performs distance calculations using the RAW data supplied from the depth sensor 11-2 based on the distance measurement calibration parameters, and supplies the resulting distance data to the object to the stitching processing unit 13. The distance measurement calculation unit 12-2 may be located within the depth sensor 11-2.

[0023] The distance measurement calibration parameters include, for example, parameters related to offset, cyclic error, distortion, and temperature compensation. The distance measurement calculation unit 12-1 and the distance measurement calculation unit 12-2 can appropriately use the necessary parameters from among the distance measurement calibration parameters.

[0024] The stitching processing unit 13 is supplied with distance data from the distance measurement calculation unit 12-1 and distance data from the distance measurement calculation unit 12-2. Based on internal and external calibration parameters, the stitching processing unit 13 performs a merging process to combine the distance data, and supplies the resulting merged distance data to the distance correction unit 14.

[0025] For example, in the stitching processing unit 13, depth images obtained from the signals of each depth sensor are supplied as distance data, and a process is performed to concatenate these depth images, thereby generating a wide-angle depth image by concatenating two depth images as the concatenated distance data. A depth image is an image that shows an object using depth information. A wide-angle depth image is a depth image that is wider in angle than a single depth image, achieved by concatenating multiple depth images.

[0026] The internal and external calibration parameters include parameters related to lens distortion used for internal calibration and parameters related to camera orientation used for external calibration. The stitching processing unit 13 can appropriately use the necessary parameters from among the internal and external calibration parameters.

[0027] The distance correction unit 14 performs a correction process to correct the combined distance data supplied from the stitching processing unit 13 based on the overlapping area information, and outputs the corrected combined distance data obtained as a result. The overlapping area information includes information such as a map of overlapping pixels. The distance correction unit 14 can appropriately use the necessary information from the information included in the overlapping area information. The corrected combined distance data outputs data such as a point cloud.

[0028] The distance correction unit 14 corrects the combined distance data by processing at least a portion of the combined distance data using a trained model (learned model) learned by machine learning. For example, the distance correction unit 14 generates a wide-angle depth image with defects corrected by processing a wide-angle depth image with defects using a trained model (learned model) learned by machine learning.

[0029] In the distance measuring device 10 configured as described above, two depth sensors, depth sensor 11-1 and depth sensor 11-2, are arranged so as to overlap in the field of view (FoV), and by combining the two depth images obtained by measuring an object, a wider-angle depth image can be generated.

[0030] At this time, the overlapping area of ​​the fields of view of depth sensor 11-1 and depth sensor 11-2 has the following effects, for example.

[0031] In other words, light from one depth sensor's light source can interfere with light from another depth sensor's light source, potentially causing adverse effects. This is due to the increased number of multipath patterns resulting from having multiple light sources. Furthermore, the areas of each depth image corresponding to the overlapping field of view, i.e., the image edges, are highly likely to be degraded due to lens distortion or insufficient correction of that distortion.

[0032] Since light is emitted from the light sources of depth sensor 11-1 and depth sensor 11-2, the depth value becomes abnormal in overlapping areas of the field of view due to double exposure. Furthermore, if light is emitted from the light source of one depth sensor, but then light is also emitted from the light source of the other depth sensor, the confidence value will also become abnormal. The light-receiving element (light-receiving part) within the depth sensor has a range in which it can receive light, but if light is emitted from depth sensor 11-1 and depth sensor 11-2 simultaneously, there is a risk that the range will be exceeded and saturate.

[0033] Figure 2 shows the overlapping portion of the field of view of depth sensor 11-2 with the field of view of depth sensor 11-1. In Figure 2, the field of view FoV1 of depth sensor 11-1 is shown by a dashed line, the field of view FoV2 of depth sensor 11-2 is shown by a double-dashed line, and the overlapping portion O of field of view FoV2 with field of view FoV1 is shown by a dot pattern.

[0034] As mentioned earlier, in this overlapping area O, light is emitted from both the light sources of the two depth sensors, depth sensor 11-1 and depth sensor 11-2, and the edges of the depth images obtained from each depth sensor deteriorate, resulting in various effects on the depth image.

[0035] Therefore, the technology disclosed herein corrects wide-angle depth images containing defects due to such overlapping fields of view, thereby obtaining defect-free wide-angle depth images. When correcting a defective wide-angle depth image, processing is performed on at least a portion of the wide-angle depth image using a trained model (learned model) trained by machine learning.

[0036] (Processing using a learned model) In the distance correction unit 14 shown in Figure 1, processing is performed on at least a portion of the wide-angle depth image containing defects using a trained model. Figure 3 shows an example configuration of a learning device that performs processing during training and an inference unit that performs processing during inference when using supervised learning.

[0037] In Figure 3, the learning device 200, which performs processing during learning, is shown in the upper section, and the inference unit 111, which performs processing during inference, is shown in the lower section. The inference unit 111 is included in the distance correction unit 14 in Figure 1.

[0038] In Figure 3, the learning device 200 has a learning model 221. The learning model 221 is a machine learning model that uses a neural network (NN) to take a wide-angle depth image with defects in the depth value and pixel position information (defective pixel position information) indicating the location of the defective pixels as input, and outputs a wide-angle depth image.

[0039] For example, in the learning model 221, a wide-angle depth image with a defect in the depth value and pixel position information indicating the location of the defective pixel (defective pixel position information) are used as training data, and by repeatedly training with information on correcting the defective pixel position (or the region containing it) as the training data, it becomes possible to output a wide-angle depth image with the defects corrected as the output. For example, a wide-angle depth image with defects corrected is a wide-angle depth image in which defects caused by the overlapping field of view of the two depth sensors have been corrected. As a neural network, for example, a DNN (Deep Neural Network) or an autoencoder can be used.

[0040] In this way, the trained model 221, which was trained using machine learning during the training phase, can be used as a pre-trained model during inference.

[0041] In Figure 3, the inference unit 111 has a learning model 121. The learning model 121 corresponds to the learning model 221, which has been trained by machine learning during the training process.

[0042] The learning model 121 performs inference using a wide-angle depth image with a defect in the depth value and information about the location of the defect pixel as input, and outputs a wide-angle depth image with the defect corrected. Here, the wide-angle depth image with a defect in the depth value is a wide-angle depth image generated by concatenating two depth images as measurement data from two depth sensors, depth sensor 11-1 and depth sensor 11-2. In other words, the defect in the wide-angle depth image is due to the overlapping portion of the fields of view of the two depth sensors. The defect pixel location information is information about the location of the defect pixel identified from the wide-angle depth image.

[0043] Furthermore, other machine learning methods may be used as supervised learning. For example, during training, the learning model 221 may be trained to output information about pixel positions with defects corrected, so that during inference, the learning model 121 performs inference using a wide-angle depth image with defects in the depth value and the defective pixel position information as input, and outputs information about pixel positions with defects corrected.

[0044] Alternatively, a learning model can be generated using unsupervised learning. For example, a learning model that uses a neural network to perform machine learning with a wide-angle depth image without defects as input can be used. By repeatedly performing unsupervised learning on this model without knowing about the wide-angle depth image with defects, it will eventually output a wide-angle depth image with the defects removed. In this way, the learning model, trained through unsupervised learning, can be used during inference. By performing inference with a wide-angle depth image containing defects as input, it is possible to output a wide-angle depth image with the defects corrected.

[0045] (Correction process) Next, the flow of the correction process in the distance correction unit 14 will be explained with reference to the flowchart in Figure 4.

[0046] In step S101, the stitching processing unit 13 concatenates two depth images to generate a wide-angle depth image.

[0047] In step S102, the distance correction unit 14 determines whether all D pixels included in the wide-angle depth image have been processed. Here, pixels included in the wide-angle depth image are referred to as D pixels.

[0048] If it is determined in step S102 that not all D pixels have been processed, the process proceeds to step S103. In step S103, the distance correction unit 14 obtains the depth value and pixel position (x, y) for the D pixels to be processed.

[0049] In step S104, the distance correction unit 14 determines whether the acquired depth value of the D pixel to be processed is a valid depth value.

[0050] In step S104, if it is determined that the depth value of the D pixel to be processed is not a valid depth value, the process proceeds to step S105. In step S105, the distance correction unit 14 obtains the pixel position (x, y) of the D pixel for which the depth value is invalid as the pixel correction position (x, y).

[0051] When the processing in step S105 is completed, the process returns to step S102. Also, if in step S104 it is determined that the depth value of the D pixel to be processed is a valid depth value, the processing in step S105 is skipped, and the process returns to step S102.

[0052] The above process is repeated, and if it is determined in step S102 that all D pixels have been processed, the process proceeds to step S106. That is, when all D pixels have been processed, the pixel positions (x, y) of D pixels in the wide-angle depth image whose depth values ​​are abnormal due to the overlapping fields of view of the two depth sensors are all identified as pixel correction positions (x, y).

[0053] In step S106, the distance correction unit 14 determines whether there are any depth values ​​in the wide-angle depth image that require correction.

[0054] In step S106, if it is determined that there are depth values ​​that require correction in the wide-angle depth image, the process proceeds to step S107. Here, the processes in steps S102 to S105 are repeated, and when the pixel correction position (x, y) of a D pixel with an abnormal depth value is identified, it is determined that there are depth values ​​that require correction.

[0055] In step S107, the distance correction unit 14 generates pixel correction position information based on the pixel correction position (x, y) of the D pixel whose depth value is abnormal. This pixel correction position information identifies the D pixel that requires depth value correction as a pixel that needs correction (a defective pixel) and includes information (coordinates (x, y)) for identifying that pixel's position.

[0056] In step S108, the inference unit 111 (Figure 3) of the distance correction unit 14 uses the learned model 121 to perform inference using the wide-angle depth image with defects in the depth value and the pixel correction position information as input, and generates a wide-angle depth image with the defects corrected. The learned model 121 is a trained model that was trained using a neural network during training with the wide-angle depth image with defects in the depth value and the defective pixel position information as input, and is capable of outputting a wide-angle depth image with the defects corrected.

[0057] Here, we have shown the case using learning model 121 (Figure 3), but you may also use other pre-trained models, such as a learning model that outputs a wide-angle depth image with defects corrected by performing inference on a wide-angle depth image with defects in the depth value as input.

[0058] When the processing in step S108 is completed, the series of processes ends. Also, if it is determined in step S106 that there are no depth values ​​that require correction in the wide-angle depth image, a wide-angle depth image without defects (a perfect wide-angle depth image) is generated and no correction is necessary, so the processes in steps S107 and S108 are skipped and the series of processes ends.

[0059] The above explains the correction process. In this correction process, the pixel positions (pixel correction positions) of D pixels with abnormal depth values ​​in the wide-angle depth image, which contains defects in depth values ​​due to the overlapping fields of view of the two depth sensors, are identified. Inference is then performed using the learning model 121 with the wide-angle depth image containing the defects and the pixel correction position information as input, thereby correcting the wide-angle depth image containing the defects. As a result, in the wide-angle depth image formed by concatenating the two depth images, the region containing defects corresponding to the overlapping fields of view of the two depth sensors is corrected.

[0060] (Correction of gaps) The above explanation showed a case where the wide-angle depth image, which includes the region corresponding to the overlapping area, is corrected when the fields of view of the two depth sensors overlap. However, when the fields of view of depth sensor 11-1 and depth sensor 11-2 do not overlap (their fields of view are far apart), the region corresponding to the gap between the two depth images obtained from each depth sensor may also be corrected.

[0061] Figure 5 shows the gap between the field of view of depth sensor 11-1 and the field of view of depth sensor 11-2. In Figure 5, the field of view FoV1 of depth sensor 11-1 is shown by a dashed line, the field of view FoV2 of depth sensor 11-2 is shown by a double dashed line, and the gap G between field of view FoV1 and field of view FoV2 is shown by a dot pattern.

[0062] In this case, if two depth images obtained from each depth sensor are combined to generate a wide-angle depth image, a wide-angle depth image containing defects in the depth values ​​will be generated due to the gap G in the field of view of the two depth sensors. Therefore, the technology according to this disclosure corrects such wide-angle depth images containing defects due to the gap in the field of view so that a wide-angle depth image without defects can be obtained. Here, when correcting the wide-angle depth image with defects, processing is performed on at least a portion of the wide-angle depth image using a trained model (learned model) that has been learned by machine learning.

[0063] The correction process here is the same as the correction process shown in Figure 4. That is, in the correction process shown in Figure 4, the pixel position (x, y) of a D pixel whose depth value is not normal due to the overlapping field of view of the two depth sensors was identified as the pixel correction position (x, y). Here, however, the pixel position (x, y) of a D pixel to which no depth value has been assigned is identified due to the gap in the field of view of the two depth sensors.

[0064] Then, by performing inference using a learned model with inputs from a wide-angle depth image containing a defect in the depth value and pixel correction position information, the defect in the wide-angle depth image is corrected. As a result, the region corresponding to the gap between the fields of view of the two depth sensors is corrected in the wide-angle depth image containing the defect. The learned model used here is a trained model that, through learning with inputs from a wide-angle depth image containing a defect in the depth value and pixel correction position information, outputs a wide-angle depth image with the defect in the depth value caused by the gap in the field of view corrected. For example, it is trained to infer the region corresponding to the gap from multiple depth images with different fields of view.

[0065] In this way, even when the fields of view of depth sensor 11-1 and depth sensor 11-2 do not overlap, a defect-free wide-angle depth image (a wide-angle depth image with interpolated connecting portions) can be generated. Furthermore, because the fields of view of the two depth sensors do not overlap, each depth sensor can avoid saturation beyond the range that the light-receiving element can detect. In addition, if there is an overlapping portion in the fields of view of the two depth sensors, the light source is set so as not to exceed the limit in the overlapping portion for eye-safe purposes, resulting in the areas other than the overlapping portion being dimly illuminated. However, if there is a gap between the fields of view of the two depth sensors, it is unnecessary to narrow such a light source, and the dimming of some areas can be avoided.

[0066] (Correction of overlapping areas of different fields of view) The above explanation described a case where the fields of view of two depth sensors overlap at the boundary. However, when two depth sensors with different fields of view, such as a wide-angle and a telephoto sensor, are arranged so that their fields of view overlap, the depth image containing defects due to the overlapping portion of the fields of view of those depth sensors may be corrected.

[0067] Figure 6 shows the overlapping area between the field of view of the wide-angle depth sensor 11-1 and the field of view of the telephoto depth sensor 11-2. In Figure 6, the field of view FoV1 of the wide-angle depth sensor 11-1 is shown by a dashed line, the field of view FoV2 of the telephoto depth sensor 11-2 is shown by a double-dotted line, and the overlapping area O of field of view FoV2 with field of view FoV1 is shown by a dot pattern.

[0068] In this case, if two depth images obtained from each depth sensor are combined to generate a new depth image, there is a possibility that a depth image containing defects in the depth value will be generated due to the overlapping portion O of the fields of view of the two depth sensors. In other words, in the overlapping portion O, the depth value may become abnormal due to double exposure by the light source of each depth sensor. Therefore, the technology according to this disclosure corrects such depth images containing defects due to the overlapping portion of the fields of view of the depth sensors so that a defect-free depth image can be obtained.

[0069] Here, when correcting depth images with defects in depth values, processing is performed using a pre-trained model (learned model) trained by machine learning on at least a portion of the depth image, as described above. In this case, a pre-trained model can be used that, through learning with depth images with defects in depth values ​​and pixel correction position information as input, outputs a depth image in which defects in depth values ​​caused by overlapping areas of the field of view have been corrected.

[0070] In this way, even when depth sensors 11-1 and 11-2 have different fields of view, such as wide-angle and telephoto, and are arranged so that their fields of view overlap, a depth image without defects can be generated. For example, when depth sensors corresponding to wide-angle and telephoto are combined, a correction process using a learning model is applied to a portion of the depth image obtained by combining the depth images from each depth sensor, thereby generating a depth image without defects and expanding the depth measurement range (the measurement range in the direction from near to far from the device).

[0071] (Correction of overlapping parts when there are multiple light sources) Even when a single depth sensor has multiple light sources and the illumination areas of the multiple light sources overlap, it is expected that the depth image may contain defects due to the overlapping illumination areas of the light sources.

[0072] Figure 7 shows the overlapping area of ​​light illuminating by light source LS1 and light source LS2 when two light sources, light source LS1 and light source LS2, are provided for one depth sensor 11. In Figure 7, the light L1 emitted from light source LS1 is shown by a dashed line, the light L2 emitted from light source LS2 is shown by a double dashed line, and the overlapping area O of the illumination areas of light sources LS1 and LS2 is shown by a dot pattern. The illumination area of ​​light from a light source is, so to speak, the area corresponding to the field of view of the light source, and in Figure 7, it can be said that the field of view of light source LS1 and the field of view of light source LS2 overlap.

[0073] In this case, when the depth sensor 11 generates a depth image, there is a possibility that a depth image containing defects in the depth value may be generated due to the overlapping portion O of the illumination areas of the two light sources. Therefore, the technology according to this disclosure corrects such depth images containing defects due to the overlapping portion of the illumination areas of the light sources so that a depth image without defects can be obtained. Here, when correcting a depth image with defects, processing using a trained model (learned model) learned by machine learning is performed on at least a part of the depth image in the same manner as described above.

[0074] To explain this in detail by applying it to the flowchart in Figure 4, the concatenation process in step S101 can be omitted, and the processing from step S102 onwards can be performed on a single depth image (wide-angle depth image) acquired by a single depth sensor 11. However, the learning model used in the inference process in step S108 can be a pre-trained model that has been trained to output a depth image corrected for defects in depth values ​​caused by the overlap of light source illumination areas, by taking a depth image with defects in depth values ​​and pixel correction position information as input.

[0075] <2. Variant>

[0076] (Other configuration examples) In the configuration shown in Figure 1, the distance correction unit 14 performs correction processing on the wide-angle depth image as combined distance data using the learned model 121. However, at least a portion of the processing, including the distance measurement calculation processing, stitching processing, and distance correction processing, can be performed using a trained model (learned model) learned by machine learning.

[0077] (A) Example of the first configuration Figure 8 shows an example configuration for applying correction processing to two depth images using a learned model.

[0078] In Figure 8, the distance measuring device 20 is equipped with distance correction units 21-1, 21-2, and 22, in place of the stitching processing unit 13 and distance correction unit 14, compared to the distance measuring device 10 in Figure 1. Note that in Figure 8, the same reference numerals are used for parts corresponding to those in Figure 1, and their explanations are omitted.

[0079] The distance correction unit 21-1 performs a correction process to correct the distance data supplied from the distance measurement calculation unit 12-1 based on the overlapping portion information, and supplies the resulting corrected distance data to the stitching processing unit 22. In other words, when the distance correction unit 21-1 corrects the depth image as distance data (corrects overlapping portions), it uses a learning model to perform inference using a depth image with defects in the depth value and pixel correction position information as input, and generates a depth image with defects corrected.

[0080] The distance correction unit 21-2 performs a correction process to correct the distance data supplied from the distance measurement calculation unit 12-2 based on the overlapping information, and supplies the resulting corrected distance data to the stitching processing unit 22. Similar to the distance correction unit 21-1, the distance correction unit 21-2 uses a learning model to generate a depth image with defects corrected.

[0081] The stitching processing unit 22 is supplied with corrected distance data from the distance correction unit 21-1 and corrected distance data from the distance correction unit 21-2. Based on internal and external calibration parameters, the stitching processing unit 22 performs a merging process to combine the corrected distance data and outputs the resulting merged distance data. In other words, the stitching processing unit 22 performs a process to concatenate the two corrected depth images supplied as corrected distance data, thereby generating a wide-angle depth image without defects.

[0082] Applying the correction processing performed by distance correction units 21-1 and 21-2 to the flowchart in Figure 4 described above, the process is as follows. That is, instead of performing the concatenation process in step S101, the processing from step S102 onward can be performed on each of the two depth images. However, the learning model used in the inference process in step S108 is a trained model that has been repeatedly trained using depth images with defects in depth values ​​and pixel correction position information as inputs during training, and can output depth images with defects in depth values ​​caused by overlapping parts and gaps in the field of view corrected. Then, the two corrected depth images are combined to generate a wide-angle depth image.

[0083] (B) Second example configuration Figure 9 shows the configuration when distance correction and stitching are performed together using a learned model.

[0084] In Figure 9, the distance measuring device 30 is equipped with a distance correction / stitching processing unit 31 instead of the stitching processing unit 13 and distance correction unit 14, compared to the distance measuring device 10 in Figure 1. Note that in Figure 9, the same reference numerals are used for parts corresponding to those in Figure 1, and their explanations are omitted.

[0085] The distance correction and stitching processing unit 31 is supplied with distance data from the distance measurement calculation unit 12-1 and distance data from the distance measurement calculation unit 12-2. Based on the overlap information and internal and external calibration parameters, the distance correction and stitching processing unit 31 generates a combined distance data from the two distance data sets with defects corrected, and outputs the resulting combined distance data.

[0086] In other words, the distance correction and stitching processing unit 31 generates a corrected wide-angle depth image as combined distance data by using a learned model to perform inference with two depth images as input. The learned model used in this inference process is, for example, a trained model that has been repeatedly trained using depth images with defects in depth values ​​as input during training, and can output a wide-angle depth image with defects in depth values ​​caused by overlapping or gapped areas of the field of view corrected.

[0087] (C) Third example of configuration Figure 10 shows a configuration in which distance measurement calculation, distance correction, and stitching processes are performed in an integrated manner using a learning model.

[0088] In Figure 10, the distance measuring device 40 is equipped with a distance calculation, distance correction, and stitching processing unit 41, instead of the distance calculation unit 12-1, distance calculation unit 12-2, stitching processing unit 13, and distance correction unit 14, compared to the distance measuring device 10 in Figure 1. In Figure 10, the same reference numerals are used for parts corresponding to those in Figure 1, and their explanations are omitted.

[0089] The distance measurement calculation, distance correction, and stitching processing unit 41 is supplied with RAW data from depth sensor 11-1 and RAW data from depth sensor 11-2. Based on the distance measurement calibration parameters, overlapping area information, and internal / external calibration parameters, the distance measurement calculation, distance correction, and stitching processing unit 41 processes the two RAW data to generate combined distance data with defects corrected, and outputs the resulting combined distance data.

[0090] In other words, the distance measurement calculation, distance correction, and stitching processing unit 41 generates a corrected wide-angle depth image as combined distance data by using a learned model to perform inference with two RAW data as inputs. The learned model used in this inference process is, for example, a trained model that has been repeatedly trained with RAW data containing defects as inputs during training, and can output a wide-angle depth image with defects in depth values ​​caused by overlapping or gapped areas of the field of view corrected.

[0091] (Further configuration examples) Figure 11 shows an example configuration for synchronizing the light source and the photodetector when the depth sensor includes a light source.

[0092] In Figure 11, the distance measuring device 50 has a newly added timing control unit 51 compared to the distance measuring device 10 in Figure 1. Furthermore, of the light source and light receiving element within the depth sensor 11-1, the light source is shown as light source 52-1. Similarly, light source 52-2 within the depth sensor 11-2 is shown. Note that in Figure 11, the same reference numerals are used for parts corresponding to those in Figure 1, and their explanations are omitted.

[0093] The timing control unit 51 generates control signals and supplies them to depth sensors 11-1 and 11-2, respectively. In depth sensor 11-1, the light source 52-1 that illuminates the object S with light and the light-receiving element that receives the light reflected from the object S operate synchronously according to the control signal from the timing control unit 51. In depth sensor 11-2, the light source 52-2 that illuminates the object S with light and the light-receiving element that receives the light reflected from the object S operate synchronously according to the control signal from the timing control unit 51.

[0094] (A technique to suppress degradation caused by overlapping fields of view) In the explanation above, image defects caused by overlapping fields of view of the depth sensor were corrected using a pre-trained model (learned model) trained through machine learning. However, by combining this with methods to suppress image degradation due to overlapping fields of view, as shown below, it is possible to generate images with fewer defects.

[0095] In other words, one method to suppress image degradation due to overlapping fields of view is to use different wavelengths for the light source and bandpass filter in depth sensor 11-1 and depth sensor 11-2. Additionally, using a dot pattern for the light source in depth sensor 11-1 and depth sensor 11-2 can reduce the probability of overlapping fields of view. Furthermore, there is a method that utilizes the dead time at the RAW frame capture level to perform alternating synchronized measurements (captures) with depth sensor 11-1 and depth sensor 11-2. By performing alternating synchronized measurements in this way, it is possible to avoid overlapping fields of view in time.

[0096] Furthermore, there is a method of performing measurements (images) by changing the modulation pattern and modulation frequency between the light source of depth sensor 11-1 and the light source of depth sensor 11-2. When performing measurements by changing the modulation pattern between light sources, the modulation patterns of the light sources are controlled to be synchronized and not overlap in order to avoid interference. When performing measurements by changing the modulation frequency between light sources, the measurement (image) patterns, such as dual frequency, are controlled so that they do not overlap in time. In addition, there is a method of performing measurements (images) by shifting the exposure timing between the light source and the photodetector in depth sensor 11-1 and depth sensor 11-2. By performing measurements by shifting the exposure timing in this way, it is possible to avoid overlapping fields of view in time.

[0097] Furthermore, the method for suppressing image degradation due to overlapping fields of view described here can be used independently, as well as in combination with correction using a learned model during inference. Even when the method for suppressing image degradation due to overlapping fields of view is used alone, it is possible to suppress image degradation due to overlapping fields of view and generate images with fewer defects.

[0098] As described above, the technology disclosed herein allows for the processing of at least a portion of the depth image acquired by depth sensor 11-1, the depth image acquired by depth sensor 11-2, and the image obtained from these depth images (for example, a wide-angle depth image obtained by concatenating two depth images) using a pre-trained model learned by machine learning, thereby correcting the pixels to be corrected (for example, defective pixels with defects in depth values) contained in any of the images. This makes it possible to correct the pixels to be corrected due to their respective fields of view, even when depth sensor 11-1 and depth sensor 11-2 are arranged in a predetermined relationship, such as when their fields of view overlap or when they do not overlap.

[0099] This process yields wide-angle depth images with defects corrected, which can be used in the following use cases, for example: Corrected wide-angle depth images can be used for progress management through building and terrain measurements at construction sites. Alternatively, corrected wide-angle depth images can be used as background images for computer graphics (CG) modeling environments in games and movies. Furthermore, corrected wide-angle depth images can be used for mobile objects such as automobiles and construction machinery to implement safety sensing functions for the surrounding environment and self-position estimation functions for autonomous driving.

[0100] Furthermore, the distance measuring device 10 described above can be considered as an information processing device having a processing unit (such as a stitching processing unit 13 and a distance correction unit 14) that processes depth images acquired by multiple sensors (such as depth sensor 11-1 and depth sensor 11-2). Distance measuring devices 20 to 50 can also be considered as information processing devices in the same way. In addition, although the distance measuring device 10 is shown with two depth sensors, depth sensor 11-1 and depth sensor 11-2, the technology of this disclosure can be applied similarly to cases where three or more depth sensors are provided. Also, although Figure 7 shows a case where two light sources, light source LS1 and light source LS2, are provided for one depth sensor 11, the technology of this disclosure can be applied similarly to cases where three or more light sources are provided.

[0101] (Example of extension) Figure 12 shows an example of a system configuration that includes an AI processing device.

[0102] Electronic device 20001 is a mobile terminal such as a smartphone, tablet, or mobile phone. Electronic device 20001 corresponds, for example, to the distance measuring device 10 (information processing device) in Figure 1 and has an optical sensor 20011 that corresponds to depth sensors 11-1 and 11-2 (Figure 1). The optical sensor is a sensor (image sensor) that converts light into electrical signals. Electronic device 20001 can connect to a base station 20020 installed at a predetermined location via wireless communication corresponding to a predetermined communication method, and then connect to a network 20040 such as the Internet via the core network 20030.

[0103] An edge server 20002 is installed closer to mobile terminals, such as between base station 20020 and core network 20030, to realize Mobile Edge Computing (MEC). A cloud server 20003 is connected to network 20040. Edge servers 20002 and cloud server 20003 can perform various processes according to their purpose. Note that edge server 20002 may also be installed within core network 20030.

[0104] AI processing is performed by electronic device 20001, edge server 20002, cloud server 20003, or optical sensor 20011. AI processing involves processing the technology related to this disclosure using AI such as machine learning. AI processing includes learning processing and inference processing. Learning processing is the process of generating a learning model. Learning processing also includes retraining processing, which will be described later. Inference processing is the process of performing inference using the learning model.

[0105] In electronic devices 20001, edge servers 20002, cloud servers 20003, or optical sensors 20011, AI processing is realized by a processor such as a CPU (Central Processing Unit) executing a program, or by using dedicated hardware such as a processor specialized for a specific application. For example, a GPU (Graphics Processing Unit) can be used as a processor specialized for a specific application.

[0106] Figure 13 shows an example configuration of the electronic device 20001. The electronic device 20001 includes a CPU 20101 that controls the operation of each part and performs various processing, a GPU 20102 specialized for image processing and parallel processing, main memory 20103 such as DRAM (Dynamic Random Access Memory), and auxiliary memory 20104 such as flash memory.

[0107] Auxiliary memory 20104 stores data such as programs and various parameters for AI processing. CPU 20101 loads the programs and parameters stored in auxiliary memory 20104 into main memory 20103 and executes the programs. Alternatively, CPU 20101 and GPU 20102 load the programs and parameters stored in auxiliary memory 20104 into main memory 20103 and execute the programs. This allows GPU 20102 to be used as GPGPU (General-Purpose computing on Graphics Processing Units).

[0108] Note that CPU20101 and GPU20102 may be configured as a SoC (System on a Chip). If CPU20101 is executing a program for AI processing, GPU20102 may not be required.

[0109] The electronic device 20001 also includes an optical sensor 20011 to which the technology relating to this disclosure is applied, an operating unit 20105 such as a physical button or touch panel, a sensor 20106 including at least one sensor, a display 20107 for displaying information such as images and text, a speaker 20108 for outputting sound, a communication I / F 20109 such as a communication module compatible with a predetermined communication method, and a bus 20110 for connecting them.

[0110] Sensor 20106 has at least one of various sensors, such as an optical sensor (image sensor), a sound sensor (microphone), a vibration sensor, an acceleration sensor, an angular velocity sensor, a pressure sensor, an odor sensor, and a biosensor. In AI processing, data acquired from at least one of the sensors of sensor 20106 can be used together with data acquired from optical sensor 20011 (image data). It is also possible that optical sensor 20011 corresponds to depth sensor 11-1 (Figure 1), and sensor 20106 corresponds to depth sensor 11-2 (Figure 1).

[0111] Furthermore, data acquired from two or more optical sensors using sensor fusion technology, or data obtained by integrating and processing them, may be used in AI processing. The two or more optical sensors may be a combination of optical sensor 20011 and the optical sensors within sensor 20106, or optical sensor 20011 may contain multiple optical sensors. For example, optical sensors may include RGB visible light sensors, distance measuring sensors such as ToF (Time of Flight), polarization sensors, event-based sensors, sensors that acquire IR images, and sensors capable of acquiring multiple wavelengths.

[0112] In electronic device 20001, AI processing can be performed using processors such as CPU 20101 and GPU 20102. When the processor of electronic device 20001 performs inference processing, it can start processing immediately after acquiring image data with the optical sensor 20011, enabling high-speed processing. Therefore, in electronic device 20001, when inference processing is used in applications that require information transmission with short latency, users can operate the device without experiencing any delay. Furthermore, when the processor of electronic device 20001 performs AI processing, it does not require the use of communication lines or server-specific computer equipment compared to using a server such as a cloud server 20003, thus enabling processing at a low cost.

[0113] Figure 14 shows an example configuration of edge server 20002. Edge server 20002 has a CPU 20201 that controls the operation of each part and performs various processing, and a GPU 20202 specialized for image processing and parallel processing. Edge server 20002 also has main memory 20203 such as DRAM, auxiliary memory 20204 such as HDD (Hard Disk Drive) or SSD (Solid State Drive), and communication I / F 20205 such as NIC (Network Interface Card), all of which are connected to bus 20206.

[0114] Auxiliary memory 20204 stores data such as programs and various parameters for AI processing. CPU20201 loads the programs and parameters stored in auxiliary memory 20204 into main memory 20203 and executes the programs. Alternatively, CPU20201 and GPU20202 can use GPU20202 as a GPGPU by loading the programs and parameters stored in auxiliary memory 20204 into main memory 20203 and executing the programs. Note that if CPU20201 is executing the AI ​​processing program, GPU20202 does not need to be provided.

[0115] In edge server 20002, AI processing can be performed using processors such as CPU 20201 and GPU 20202. When edge server 20002's processor performs AI processing, it can achieve lower processing latency because edge server 20002 is located closer to electronic device 20001 compared to cloud server 20003. Furthermore, edge server 20002 has higher processing capabilities, such as calculation speed, compared to electronic device 20001 and optical sensor 20011, allowing for a more versatile configuration. Therefore, when edge server 20002's processor performs AI processing, it can perform AI processing as long as data can be received, regardless of differences in the specifications and performance of electronic device 20001 and optical sensor 20011. Performing AI processing on edge server 20002 can reduce the processing load on electronic device 20001 and optical sensor 20011.

[0116] The configuration of Cloud Server 20003 is the same as that of Edge Server 20002, so the explanation will be omitted.

[0117] Cloud Server 20003 can perform AI processing using processors such as CPU 20201 and GPU 20202. Because Cloud Server 20003 has higher processing capabilities, including faster calculation speeds, compared to Electronic Device 20001 and Optical Sensor 20011, it can be configured more flexibly. Therefore, when Cloud Server 20003's processor performs AI processing, it can do so regardless of the specifications or performance differences of Electronic Device 20001 or Optical Sensor 20011. Furthermore, if it is difficult for the processors of Electronic Device 20001 or Optical Sensor 20011 to perform high-load AI processing, Cloud Server 20003's processor can handle that processing, and the results can be fed back to the processors of Electronic Device 20001 or Optical Sensor 20011.

[0118] Figure 15 shows an example configuration of the optical sensor 20011. The optical sensor 20011 can be configured as a single-chip semiconductor device having a stacked structure in which multiple substrates are stacked. The optical sensor 20011 is constructed by stacking two substrates, substrate 20301 and substrate 20302. The configuration of the optical sensor 20011 is not limited to a stacked structure; for example, the substrate including the imaging unit may also include a processor that performs AI processing, such as a CPU or a DSP (Digital Signal Processor).

[0119] The upper substrate 20301 is equipped with an imaging unit 20321, which consists of multiple pixels arranged in a two-dimensional configuration. The lower substrate 20302 is equipped with an imaging processing unit 20322, which performs image acquisition processing on the imaging unit 20321, an output interface 20323, which outputs the captured image and signal processing results to the outside, and an imaging control unit 20324, which controls image acquisition on the imaging unit 20321. The imaging unit 20321, imaging processing unit 20322, output interface 20323, and imaging control unit 20324 constitute the imaging block 20311.

[0120] The lower substrate 20302 is equipped with a CPU 20331 for controlling each part and performing various processes, a DSP 20332 for signal processing using captured images and external information, a memory 20333 such as SRAM (Static Random Access Memory) and DRAM (Dynamic Random Access Memory), and a communication interface 20334 for exchanging necessary information with the outside. The signal processing block 20312 is formed by the CPU 20331, DSP 20332, memory 20333, and communication interface 20334. AI processing can be performed by at least one of the processors, the CPU 20331 and the DSP 20332.

[0121] In this way, the signal processing block 20312 for AI processing can be mounted on the lower substrate 20302 in a stacked structure in which multiple substrates are stacked. As a result, image data acquired by the imaging block 20311 mounted on the upper substrate 20301 is processed by the signal processing block 20312 for AI processing mounted on the lower substrate 20302, so that a series of processes can be performed within a single semiconductor device chip.

[0122] In the optical sensor 20011, AI processing can be performed by a processor such as the CPU 20331. When the processor of the optical sensor 20011 performs AI processing such as inference processing, the entire process is carried out within a single semiconductor device chip, so information does not leak outside the sensor, thus enhancing information confidentiality. Furthermore, since there is no need to transmit data such as image data to other devices, the processor of the optical sensor 20011 can perform AI processing such as inference processing using image data at high speed. For example, when inference processing is used in applications that require real time, sufficient real time can be ensured. Here, ensuring real time means that information can be transmitted with a short delay time. In addition, when the processor of the optical sensor 20011 performs AI processing, various metadata can be passed by the processor of the electronic device 20001, thereby reducing processing and lowering power consumption.

[0123] Figure 16 shows an example configuration of the processing unit 20401. The processors of the electronic device 20001, edge server 20002, cloud server 20003, or optical sensor 20011 function as the processing unit 20401 by executing various processes according to the program. Multiple processors from the same or different devices may also function as the processing unit 20401.

[0124] The processing unit 20401 includes an AI processing unit 20411. The AI ​​processing unit 20411 performs AI processing. The AI ​​processing unit 20411 includes a learning unit 20421 and an inference unit 20422.

[0125] The learning unit 20421 performs a learning process to generate a learning model. In the learning process, a trained learning model is generated that has undergone machine learning to correct the pixels to be corrected contained in the image data. The learning unit 20421 may also perform a retraining process to update the generated learning model. In the following explanation, the generation and updating of the learning model will be described separately, but since updating the learning model can also be said to be generating a learning model, the meaning of updating the learning model will be included in the generation of the learning model.

[0126] Furthermore, the generated learning model is recorded in a storage medium such as the main memory or auxiliary memory of the electronic device 20001, edge server 20002, cloud server 20003, or optical sensor 20011, making it newly available for use in the inference processing performed by the inference unit 20422. This makes it possible to generate the electronic device 20001, edge server 20002, cloud server 20003, or optical sensor 20011, which perform inference processing based on the learning model. Moreover, the generated learning model may be recorded in a storage medium or electronic device independent of the electronic device 20001, edge server 20002, cloud server 20003, or optical sensor 20011, and provided for use in other devices. Note that the generation of these electronic devices 20001, edge server 20002, cloud server 20003, or optical sensor 20011 includes not only recording a new learning model in their storage medium during manufacturing, but also updating an already recorded generated learning model.

[0127] The inference unit 20422 performs inference processing using a learned model. In the inference processing, the learned model is used to identify pixels to be corrected in the image data and to correct the identified pixels to be corrected. Pixels to be corrected are pixels that meet predetermined conditions among multiple pixels in the image data.

[0128] Machine learning techniques can include neural networks and deep learning. A neural network is a model that mimics the neural circuits of the human brain and consists of three layers: an input layer, an intermediate layer (hidden layer), and an output layer. Deep learning is a model that uses a multi-layered neural network and repeatedly performs characteristic learning in each layer, enabling it to learn complex patterns hidden within large amounts of data.

[0129] For machine learning problem formulation, supervised learning can be used. For example, supervised learning learns features based on given labeled training data. This makes it possible to derive labels for unknown data. The training data can be image data actually acquired by a light sensor, aggregated and managed acquired image data, or datasets generated by a simulator.

[0130] In addition to supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning may also be used. Unsupervised learning involves analyzing a large amount of unlabeled training data to extract features, and then performing clustering or other operations based on the extracted features. This makes it possible to analyze trends and make predictions based on a vast amount of unknown data. Semi-supervised learning is a combination of supervised and unsupervised learning, where features are learned using supervised learning, and then a large amount of training data is provided using unsupervised learning, allowing for iterative learning while automatically calculating features. Reinforcement learning deals with the problem of an agent observing its current state in a given environment and deciding what action to take.

[0131] In this way, the processors of the electronic device 20001, edge server 20002, cloud server 20003, or optical sensor 20011 function as the AI ​​processing unit 20411, so that AI processing is performed in one or more of these devices.

[0132] The AI ​​processing unit 20411 only needs to have at least one of the learning unit 20421 and the inference unit 20422. In other words, the processor of each device may perform both learning and inference processing, or it may perform only one of the learning and inference processing. For example, if the processor of the electronic device 20001 performs both inference and learning processing, it will have a learning unit 20421 and an inference unit 20422, but if it only performs inference processing, it will only need to have an inference unit 20422.

[0133] The processor in each device may perform all processing related to the learning or inference process, or it may perform some processing on its own device's processor and then perform the remaining processing on the processor of another device. Furthermore, each device may have a common processor for performing each function of AI processing, such as learning and inference, or it may have separate processors for each function.

[0134] Furthermore, AI processing may be performed by devices other than those described above. For example, AI processing can be performed by other electronic devices that can be connected to electronic device 20001 via wireless communication or the like. Specifically, if electronic device 20001 is a smartphone, other electronic devices that perform AI processing may include other smartphones, tablet devices, mobile phones, PCs (personal computers), game consoles, television receivers, wearable devices, digital still cameras, digital video cameras, and the like.

[0135] Furthermore, AI processing such as inference processing can be applied to configurations using sensors mounted on mobile devices such as automobiles, or sensors used in telemedicine equipment, but these environments require low latency. In such environments, latency can be reduced by performing AI processing on the processor of the local device (for example, electronic equipment 20001 used as in-vehicle equipment or medical equipment) rather than performing AI processing on the processor of the cloud server 20003 via the network 20040. Moreover, even in cases where there is no environment to connect to the network 20040 such as the internet, or where high-speed connections are not possible, AI processing can be performed in a more appropriate environment by performing AI processing on the processor of the local device, such as electronic equipment 20001 or optical sensor 20011.

[0136] The above configuration is merely an example, and other configurations may be adopted. For example, the electronic device 20001 is not limited to mobile terminals such as smartphones, but may also be electronic devices such as PCs, game consoles, television receivers, wearable devices, digital still cameras, digital video cameras, in-vehicle equipment, or medical equipment. Furthermore, the electronic device 20001 may be connected to the network 20040 by wireless or wired communication compatible with a predetermined communication method such as wireless LAN (Local Area Network) or wired LAN. AI processing is not limited to processors such as CPUs and GPUs of each device, but may also utilize quantum computers or neuromorphic computers.

[0137] Incidentally, data such as learning models, image data, and corrected data can be used not only within a single device, but also exchanged between multiple devices and used within those devices. Figure 17 shows the flow of data between multiple devices.

[0138] Electronic devices 20001-1 to 20001-N (where N is an integer greater than or equal to 1) are, for example, owned by each user and can connect to a network 20040 such as the Internet via a base station (not shown), etc. During manufacturing, a learning device 20501 is connected to electronic device 20001-1, and the learning model provided by the learning device 20501 can be recorded in auxiliary memory 20104. The learning device 20501 generates a learning model using a dataset generated by the simulator 20502 as learning data and provides it to electronic device 20001-1. Note that the learning data is not limited to the dataset provided by the simulator 20502, but may also be image data actually acquired by an optical sensor or acquired image data that is aggregated and managed.

[0139] Although not shown in the diagram, the learning model can be recorded at the time of manufacture for electronic devices 20001-2 through 20001-N, just like for electronic device 20001-1. Hereafter, unless it is necessary to distinguish between electronic devices 20001-1 through 20001-N, they will be referred to as electronic device 20001.

[0140] Network 20040 is connected to the electronic device 20001, as well as the learning model generation server 20503, the learning model provision server 20504, the data provision server 20505, and the application server 20506, allowing them to exchange data with each other. Each server can be configured as a cloud server.

[0141] The learning model generation server 20503 has a configuration similar to the cloud server 20003 and can perform learning processing using a processor such as a CPU. The learning model generation server 20503 generates a learning model using the learning data. In the illustrated configuration, the case in which the electronic device 20001 records the learning model during manufacturing is illustrated, but the learning model may also be provided by the learning model generation server 20503. The learning model generation server 20503 transmits the generated learning model to the electronic device 20001 via the network 20040. The electronic device 20001 receives the learning model transmitted from the learning model generation server 20503 and records it in the auxiliary memory 20104. This generates the electronic device 20001 equipped with that learning model.

[0142] In other words, if the electronic device 20001 does not have a learning model recorded at the time of manufacture, a new learning model is recorded from the learning model generation server 20503, thereby generating an electronic device 20001 with a newly recorded learning model. Furthermore, if the electronic device 20001 already has a learning model recorded at the time of manufacture, the recorded learning model is updated with the learning model from the learning model generation server 20503, thereby generating an electronic device 20001 with the updated learning model. The electronic device 20001 can then perform inference processing using the appropriately updated learning model.

[0143] The learning model is not limited to being provided directly from the learning model generation server 20503 to the electronic device 20001; it may also be provided via the network 20040 by a learning model provision server 20504 that aggregates and manages various learning models. The learning model provision server 20504 may provide the learning model to other devices, not just the electronic device 20001, thereby creating other devices equipped with that learning model. Furthermore, the learning model may be provided by recording it on a removable memory card such as flash memory. The electronic device 20001 can read and record the learning model from the memory card inserted in the slot. This allows the electronic device 20001 to acquire the learning model even when used in harsh environments, when it does not have communication capabilities, or when it has communication capabilities but the amount of information that can be transmitted is small.

[0144] Electronic device 20001 can provide data such as image data, corrected data, and metadata to other devices via network 20040. For example, electronic device 20001 can send data such as image data and corrected data to the learning model generation server 20503 via network 20040. This allows the learning model generation server 20503 to use the image data and corrected data collected from one or more electronic devices 20001 as training data to generate a learning model. By using more training data, the accuracy of the learning process can be improved.

[0145] Image data and corrected data, etc., are not limited to being provided directly from the electronic device 20001 to the learning model generation server 20503; they may also be provided by the data provision server 20505, which aggregates and manages various types of data. The data provision server 20505 may collect data from devices other than the electronic device 20001, and may provide data to devices other than the learning model generation server 20503.

[0146] The learning model generation server 20503 may update an already generated learning model by performing a retraining process, adding data such as image data and corrected data provided by the electronic device 20001 or the data provision server 20505 to the training data. The updated learning model can be provided to the electronic device 20001. When performing training or retraining processing in the learning model generation server 20503, processing can be performed regardless of differences in the specifications or performance of the electronic device 20001.

[0147] Furthermore, if a user performs a modification operation on corrected data or metadata in the electronic device 20001 (for example, if the user enters correct information), feedback data related to that modification process may be used in the retraining process. For example, by sending feedback data from the electronic device 20001 to the learning model generation server 20503, the learning model generation server 20503 can perform a retraining process using the feedback data from the electronic device 20001 and update the learning model. In addition, when a user performs a modification operation in the electronic device 20001, an application provided by the application server 20506 may be used.

[0148] The retraining process may be performed by electronic device 20001. When electronic device 20001 updates the learning model by performing a retraining process using image data and feedback data, the learning model can be improved within the device. This generates electronic device 20001 equipped with the updated learning model. Furthermore, electronic device 20001 may send the updated learning model obtained from the retraining process to the learning model provision server 20504 so that it can be provided to other electronic devices 20001. This allows the updated learning model to be shared among multiple electronic devices 20001.

[0149] Alternatively, electronic device 20001 may send the difference information of the retrained learning model (difference information between the learning model before the update and the learning model after the update) as update information to the learning model generation server 20503. The learning model generation server 20503 can generate an improved learning model based on the update information from electronic device 20001 and provide it to other electronic devices 20001. By exchanging such difference information, privacy can be protected and communication costs can be reduced compared to exchanging all information. In addition, similar to electronic device 20001, the optical sensor 20011 mounted on electronic device 20001 may also perform the retraining process.

[0150] The application server 20506 is a server capable of providing various applications via the network 20040. The applications provide predetermined functions using data such as learning models, corrected data, and metadata. The electronic device 20001 can achieve predetermined functions by executing applications downloaded from the application server 20506 via the network 20040. Alternatively, the application server 20506 can also achieve predetermined functions by acquiring data from the electronic device 20001 via, for example, an API (Application Programming Interface) and executing applications on the application server 20506.

[0151] Thus, in a system including devices to which this technology is applied, data such as learning models, image data, and corrected data can be exchanged and distributed between each device, making it possible to provide various services using this data. For example, it is possible to provide a service that provides learning models via the learning model provision server 20504, and a service that provides data such as image data and corrected data via the data provision server 20505. It is also possible to provide a service that provides applications via the application server 20506.

[0152] Alternatively, the learning model provided by the learning model provision server 20504 may be provided with corrected data obtained as output by inputting image data acquired from the optical sensor 20011 of the electronic device 20001. Furthermore, a device such as an electronic device implementing the learning model provided by the learning model provision server 20504 may be generated and provided. In addition, data such as the learning model, corrected data, and metadata may be recorded on a readable storage medium, thereby generating and providing a storage medium on which such data is recorded, or a device such as an electronic device equipped with such a storage medium. The storage medium may be a non-volatile memory such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, or a volatile memory such as SRAM or DRAM.

[0153] The embodiments described herein are not limited to those described above, and various modifications are possible without departing from the spirit of this disclosure. Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0154] Furthermore, this disclosure can take the following form.

[0155] (1) The process involves processing at least a portion of a first image showing an object acquired by a first sensor with depth information, a second image showing an object acquired by a second sensor with depth information, and a third image obtained from the first and second images using a trained model trained by machine learning to correct the pixels to be corrected in any of the images. Equipped with a processing unit Information processing device. (2) The first sensor and the second sensor are arranged so that their fields of view overlap. The aforementioned processing unit, When generating the third image by concatenating the first image and the second image, the position corresponding to the overlapping portion of the field of view in the third image is identified as the pixel correction position. The depth information of the pixel correction position is corrected using the trained model. The information processing device described in (1) above. (3) The trained model is a model that, through training with the defective third image and the pixel correction position as input, outputs the third image in which the depth information defects due to the overlapping portion of the field of view have been corrected. The processing unit uses the trained model to perform inference using the third image, which has a defect in depth information due to the overlapping portion of the field of view, and the pixel correction position as inputs, thereby generating the third image with the defect corrected. The information processing device described in (2) above. (4) The first sensor and the second sensor are arranged so that their fields of view overlap. The aforementioned processing unit, When generating the third image by concatenating the first image and the second image, the position corresponding to the overlapping portion of the field of view in the first image and the second image is identified as the pixel correction position. The depth information of the pixel correction position is corrected using the trained model. The information processing device described in (1) above. (5) The trained model is a model that, through training with the defective first or second image and the pixel correction position as input, outputs the first or second image in which the depth information defects due to the overlapping portion of the field of view have been corrected. The aforementioned processing unit, Using the trained model, inference is performed with the first or second image having a defect in depth information due to the overlapping portion of the field of view and the pixel correction position as input to generate the first or second image with the defect corrected. The first image and the second image are concatenated to generate the third image. The information processing device described in (4) above. (6) The first sensor and the second sensor are arranged so that their fields of view do not overlap. The aforementioned processing unit, When generating the third image by concatenating the first image and the second image, the position corresponding to the gap in the field of view in the third image is identified as the pixel correction position. The depth information of the pixel correction position is corrected using the trained model. The information processing device described in (1) above. (7) The trained model is a model that, through training with the defective third image and the pixel correction position as input, outputs the third image in which the depth information defects caused by the gaps in the field of view have been corrected. The processing unit uses the trained model to perform inference using the third image, which has a defect in depth information due to the gap in the field of view, and the pixel correction position as inputs, thereby generating the third image with the defect corrected. The information processing device described in (6) above. (8) The first sensor and the second sensor are sensors having different fields of view, and are arranged such that their fields of view overlap. The aforementioned processing unit, When generating the third image by concatenating the first image and the second image, the position corresponding to the overlapping portion of the field of view in the third image is identified as the pixel correction position. The depth information of the pixel correction position is corrected using the trained model. The information processing device described in (1) above. (9) The trained model is a model that, through training with the defective third image and the pixel correction position as input, outputs the third image in which the depth information defects due to the overlapping portion of the field of view have been corrected. The processing unit uses the trained model to perform inference using the third image, which has a defect in depth information due to the overlapping portion of the field of view, and the pixel correction position as inputs, thereby generating the third image with the defect corrected. The information processing device described in (8) above. (10) The first sensor has a wide-angle field of view, The second sensor has a field of view that corresponds to telephoto The information processing apparatus described in (8) or (9) above. (11) Information processing device, The process involves processing at least a portion of a first image showing an object acquired by a first sensor with depth information, a second image showing an object acquired by a second sensor with depth information, and a third image obtained from the first and second images using a trained model trained by machine learning to correct the pixels to be corrected in any of the images. Information processing methods. (12) Computers, The process involves processing at least a portion of a first image showing an object acquired by a first sensor with depth information, a second image showing an object acquired by a second sensor with depth information, and a third image obtained from the first and second images using a trained model trained by machine learning to correct the pixels to be corrected in any of the images. Equipped with a processing unit A program that makes an information processing device function. (13) The system includes a processing unit that performs processing on at least a portion of an image showing an object with depth information acquired by a sensor having a first light source and a second light source, using a trained model learned by machine learning. The first light source and the second light source are arranged such that their light irradiation areas overlap. The aforementioned processing unit, The position corresponding to the overlapping portion of the irradiation area in the aforementioned image is identified as the pixel correction position. The depth information of the pixel correction position is corrected using the trained model. Information processing device. (14) The aforementioned trained model is a model that, through training with the defective image and the pixel correction position as input, outputs an image in which the depth information defects caused by the overlapping portion of the illumination area have been corrected. The processing unit uses the trained model to perform inference using the image with depth information defects due to overlapping irradiation areas and the pixel correction positions as inputs, thereby generating the image with defects corrected. The information processing device described in (13) above. (15) Information processing device, At least a portion of the image showing the depth information of an object acquired by a sensor having a first light source and a second light source arranged so that the light illumination areas overlap is processed using a trained model trained by machine learning. The position corresponding to the overlapping portion of the irradiation area in the aforementioned image is identified as the pixel correction position. The depth information of the pixel correction position is corrected using the trained model. Information processing methods. (16) Computers, The system includes a processing unit that performs processing on at least a portion of an image showing an object as depth information obtained by a sensor having a first light source and a second light source arranged so that the light illumination areas overlap, using a trained model learned by machine learning. The aforementioned processing unit, The position corresponding to the overlapping portion of the irradiation area in the aforementioned image is identified as the pixel correction position. The depth information of the pixel correction position is corrected using the trained model. A program that makes an information processing device function. [Explanation of Symbols]

[0156] 10,20,30,40,50 Distance measuring device, 11,11-1,11-2 Depth sensor, 12-1,12-2 Distance calculation unit, 13 Stitching processing unit, 14 Distance correction unit, 20 Learning device, 21-1,21-2 Distance correction unit, 22 Stitching processing unit, 31 Distance correction / stitching processing unit, 41 Distance calculation / distance correction / stitching processing unit, 51 Timing control unit, 52-1,52-2 Light source, 111 Inference unit, 121 Learning model, 200 Learning device, 221 Learning model

Claims

1. The process involves processing at least a portion of a first image showing an object acquired by a first sensor with depth information, a second image showing an object acquired by a second sensor with depth information, and a third image obtained from the first and second images using a trained model trained by machine learning, thereby correcting the pixels to be corrected in any of the images. Equipped with a processing unit Information processing device.

2. The first sensor and the second sensor are arranged so that their fields of view overlap. The aforementioned processing unit, When generating the third image by concatenating the first image and the second image, the position corresponding to the overlapping portion of the field of view in the third image is identified as the pixel correction position. The depth information of the pixel correction position is corrected using the trained model. The information processing apparatus according to claim 1.

3. The trained model is a model that, through training with the defective third image and the pixel correction position as input, outputs the third image in which the depth information defects due to the overlapping portion of the field of view have been corrected. The processing unit uses the trained model to perform inference using the third image, which has a defect in depth information due to the overlapping portion of the field of view, and the pixel correction position as inputs, thereby generating the third image with the defect corrected. The information processing apparatus according to claim 2.

4. The first sensor and the second sensor are arranged so that their fields of view overlap. The aforementioned processing unit, When generating the third image by concatenating the first image and the second image, the position corresponding to the overlapping portion of the field of view in the first image and the second image is identified as the pixel correction position. The depth information of the pixel correction position is corrected using the trained model. The information processing apparatus according to claim 1.

5. The trained model is a model that, through training with the defective first or second image and the pixel correction position as input, outputs the first or second image in which the depth information defects due to the overlapping portion of the field of view have been corrected. The aforementioned processing unit, Using the trained model, inference is performed with the first or second image having a defect in depth information due to the overlapping portion of the field of view and the pixel correction position as input to generate the first or second image with the defect corrected. The first image and the second image are concatenated to generate the third image. The information processing apparatus according to claim 4.

6. The first sensor and the second sensor are arranged so that their fields of view do not overlap. The aforementioned processing unit, When generating the third image by concatenating the first image and the second image, the position corresponding to the gap in the field of view in the third image is identified as the pixel correction position. The depth information of the pixel correction position is corrected using the trained model. The information processing apparatus according to claim 1.

7. The trained model is a model that, through training with the defective third image and the pixel correction position as input, outputs the third image in which the depth information defects caused by gaps in the field of view have been corrected. The processing unit uses the trained model to perform inference using the third image, which has a defect in depth information due to the gap in the field of view, and the pixel correction position as inputs, thereby generating the third image with the defect corrected. The information processing apparatus according to claim 6.

8. The first sensor and the second sensor are sensors having different fields of view, and are arranged so that their fields of view overlap. The aforementioned processing unit, When generating the third image by concatenating the first image and the second image, the position corresponding to the overlapping portion of the field of view in the third image is identified as the pixel correction position. The depth information of the pixel correction position is corrected using the trained model. The information processing apparatus according to claim 1.

9. The trained model is a model that, through training with the defective third image and the pixel correction position as input, outputs the third image in which the depth information defects due to the overlapping portion of the field of view have been corrected. The processing unit uses the trained model to perform inference using the third image, which has a defect in depth information due to the overlapping portion of the field of view, and the pixel correction position as inputs, thereby generating the third image with the defect corrected. The information processing apparatus according to claim 8.

10. The first sensor has a wide-angle field of view, The second sensor has a field of view that corresponds to telephoto The information processing apparatus according to claim 8.

11. Information processing device, The process involves processing at least a portion of a first image showing an object acquired by a first sensor with depth information, a second image showing an object acquired by a second sensor with depth information, and a third image obtained from the first and second images using a trained model trained by machine learning, thereby correcting the pixels to be corrected in any of the images. Information processing methods.

12. Computers, The process involves processing at least a portion of a first image showing an object acquired by a first sensor with depth information, a second image showing an object acquired by a second sensor with depth information, and a third image obtained from the first and second images using a trained model trained by machine learning, thereby correcting the pixels to be corrected in any of the images. Equipped with a processing unit A program that makes an information processing device function.

13. The system includes a processing unit that performs processing on at least a portion of an image showing an object with depth information acquired by a sensor having a first light source and a second light source, using a trained model learned by machine learning. The first light source and the second light source are arranged such that their light irradiation areas overlap. The aforementioned processing unit, The position corresponding to the overlapping portion of the irradiation area in the aforementioned image is identified as the pixel correction position. The depth information of the pixel correction position is corrected using the trained model. Information processing device.

14. The aforementioned trained model is a model that, through training with the defective image and the pixel correction position as input, outputs the image in which the depth information defects caused by the overlapping portion of the illumination area have been corrected. The processing unit uses the trained model to perform inference using the image with depth information defects due to overlapping irradiation areas and the pixel correction positions as inputs, thereby generating the image with defects corrected. The information processing apparatus according to claim 13.

15. Information processing device, At least a portion of the image showing the depth information of an object acquired by a sensor having a first light source and a second light source arranged so that the light illumination areas overlap is processed using a trained model learned by machine learning. The position corresponding to the overlapping portion of the irradiation area in the aforementioned image is identified as the pixel correction position. The depth information of the pixel correction position is corrected using the trained model. Information processing methods.

16. Computers, The system includes a processing unit that performs processing on at least a portion of an image showing an object as depth information obtained by a sensor having a first light source and a second light source arranged so that the light illumination areas overlap, using a trained model learned by machine learning. The aforementioned processing unit, The position corresponding to the overlapping portion of the irradiation area in the aforementioned image is identified as the pixel correction position. The depth information of the pixel correction position is corrected using the trained model. A program that makes an information processing device function.

Images