Learning system and learning method

Abstract

Provided are a learning system and a learning method capable of improving accuracy of a parallax correct answer value used for learning and improving learning accuracy of a parallax calculation model. The present invention provides a learning system including: an image acquisition unit (server-side transmission / reception unit 501) that acquires images captured by a plurality of cameras; a learning policy determination unit 504 that determines an execution order of learning for a parallax calculation model with respect to a plurality of common imaging regions; and a parallax calculation model learning unit 505 that inputs, to the parallax calculation model, images that are acquired from the plurality of cameras and that have a common imaging region to be learned in accordance with the determined execution order, and learns the parallax calculation model from the difference between the obtained parallax and correct answer value data, wherein the correct answer value data include parallax information of a common imaging region allocated to an execution order before the common imaging region to be learned.

Description

Learning system and learning method

[0001] The present invention relates to a learning system and a learning method for learning a parallax inference model that infers parallax in a common imaging area of ​​a plurality of imaging devices in an image processing device that measures the distance to a detection target in the common imaging area.

[0002] In recent years, multi-camera systems, in which multiple imaging devices (cameras) are installed on a moving object, have been developed for autonomous driving and driver assistance technologies. In multi-camera systems, cameras that share a common field of view can perform stereo distance measurement, enabling accurate estimation of the distance to an object.

[0003] In stereo distance measurement, the parallax, which is the difference in the position at which the same object is captured by two cameras (right and left cameras) that share a common field of view, is calculated. Because the parallax differs depending on the depth, the distance to the detected object can be calculated from the calculated parallax.

[0004] Conventionally, disparity has been calculated by calculating the similarity of each local region between images captured by left and right cameras (left and right images) using rule-based processing. In recent years, with the advancement of deep learning, typified by convolutional neural networks (CNN), learning-based methods have been developed. In learning-based methods, a disparity calculation model (also called a disparity inference model) is trained using multiple captured images and distance data that serves as a ground truth (see, for example, Patent Document 1).

[0005] Japanese Patent Application Laid-Open No. 2020-80047

[0006] As described in Patent Document 1, "True depth can be obtained using a 3D sensor such as LiDAR (Laser Imaging Detection and Ranging)," conventional learning-based methods use a distance measurement sensor such as LiDAR to create correct distance data. However, the point density of LiDAR is generally low, and learning accuracy decreases in areas with low point density.

[0007] One way to compensate for the point density of LiDAR is to use additional disparity information created by methods other than LiDAR. Examples of additional disparity information include disparity calculated by a rule-based method or disparity calculated from the results of feature point matching. Disparity calculated in another common imaging area (also called overlapping imaging area) (rule-based disparity calculation, feature point matching, disparity calculation model, etc.) can also be used.

[0008] However, in multi-camera systems, there are camera pairs with long baselines, different angles of view, and non-parallel cameras. In such camera pairs, the shapes of the left and right images change significantly, making it impossible to calculate additional disparity information correctly. If the disparity calculation model is trained using incorrect additional disparity information, the performance of the disparity calculation model may deteriorate.

[0009] The present invention has been made in consideration of the above circumstances, and aims to provide a learning system and a learning method that can increase the accuracy of the correct disparity values ​​used for learning and improve the learning accuracy of the disparity calculation model.

[0010] In order to solve the above problem, the learning system of the present invention is a learning system that includes an on-board information processing device mounted on a vehicle equipped with a plurality of cameras having a common imaging area whose fields of view overlap at least partially, and a server device that communicates with the on-board information processing device, and is equipped with an image acquisition unit that acquires images captured by the plurality of cameras, a learning policy determination unit that determines the execution order of learning of a disparity calculation model for the plurality of common imaging areas, and a disparity calculation model learning unit that inputs images acquired from the plurality of cameras having the common imaging area to be learned into the disparity calculation model according to the determined execution order, and learns the disparity calculation model from the difference between the obtained disparity and correct value data, and the correct value data includes disparity information of a common imaging area that is assigned to an execution order earlier than the common imaging area to be learned.

[0011] In addition, the learning method of the present invention is a learning method executed by a computer, and includes the steps of acquiring images captured by a plurality of cameras having a common imaging area in which at least a portion of the field of view overlaps, determining an execution order for learning of a disparity inference model that infers disparity using the images for each of the plurality of common imaging areas, and inputting images acquired from the plurality of cameras having the common imaging area to be learned into the disparity inference model according to the determined execution order, and learning the disparity inference model from the difference between the obtained disparity and correct value data, wherein the correct value data includes disparity information of a common imaging area assigned in an execution order earlier than the common imaging area to be learned.

[0012] According to the present invention, by ordering and training a disparity calculation model for a plurality of common imaging areas, in other words, by determining the training order of the disparity calculation model, the accuracy of the correct disparity value used for training can be increased, and the training accuracy of the disparity calculation model can be improved.

[0013] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments.

[0014] FIG. 1 is a block diagram showing an example of the configuration of an image processing device according to a first embodiment of the present invention. FIG. 2 is a hardware configuration diagram of the image processing device according to the first embodiment of the present invention. FIG. 3 is a diagram used to explain a multi-camera system, where (a) is a side view of an example of camera and LiDAR installation, (b) is an overhead view of a first common imaging area, (c) is an overhead view of a second common imaging area, and (d) is an overhead view of a third common imaging area. FIG. 4 is a diagram used to explain learning of a parallax calculation model. FIG. 5 is a diagram used to explain additional parallax information, where (a) is an explanatory diagram of additional parallax information obtained using parallax information (such as rule-based parallax) acquired without using a ranging sensor, (b) is an explanatory diagram of additional parallax information obtained by feature point matching, and (c) is an explanatory diagram of additional parallax information obtained by using parallax calculated in another common imaging area. FIG. 6 is a diagram used to explain the accuracy of additional parallax information, where (a) is an explanatory diagram of the accuracy of additional parallax information in the second common imaging area and (b) is an explanatory diagram of the accuracy of additional parallax information in the third common imaging area. FIG. 7 is a diagram used to explain shape changes when the same object is imaged with different cameras. 10 is a processing flow of a learning policy determination unit 200. FIG. 11 is a block diagram showing an example of the configuration of a learning system according to a second embodiment of the present invention.

[0015] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In each drawing, parts having the same function or configuration are designated by the same reference numerals, and repeated description may be omitted.

[0016] 1 shows a configuration diagram of an image processing device 1 according to a first embodiment of the present invention. The image processing device 1 of this embodiment includes a data storage unit 100, a learning policy determination unit 200, and a disparity calculation model learning unit 300. The image processing device 1 is configured by a computer capable of acquiring data from a vehicle equipped with a multi-camera system.

[0017] FIG. 2 illustrates a hardware configuration of an image processing apparatus 1 according to a first embodiment. The image processing apparatus 1 includes a processor 91, a storage device 92, an input device 93, an output device 94, and a communication interface (communication IF) 95. The processor 91, the storage device 92, the input device 93, the output device 94, and the communication interface (communication IF) 95 are connected via a bus 96. The processor 91 controls the image processing apparatus 1. The storage device 92 serves as a work area for the processor 91. The storage device 92 is a volatile or non-volatile recording medium for storing various programs and data, such as a read-only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), or a flash memory. The input device 93 inputs data. Examples of the input device 93 include a keyboard, a mouse, a touch panel, a numeric keypad, and a scanner. The output device 94 outputs data. Examples of the output device 94 include a display and a printer. The communication IF 95 connects to a network and transmits and receives data. The image processing device 1 can receive data acquired by a vehicle equipped with a multi-camera system by communicating with the vehicle via the communication IF 95. Note that the means by which the image processing device 1 acquires data from the vehicle is not limited to this method, and any method can be used.

[0018] A multi-camera system will be described using FIGS. 3( a) to 3(d). FIG. 3(a) shows an example of camera installation in a multi-camera system. In FIG. 3(a), a first camera 11 is mounted below the side mirror of a vehicle 10, and a second camera 12 and a third camera 13 are mounted on the front and rear of the roof of the vehicle 10. Each camera is mounted on the vehicle 10 so as to have an area that shares a field of view (hereinafter, a common imaging area). The common imaging area of ​​the first camera 11 and the second camera 12 is a first common imaging area 21 (FIG. 3(b)), the common imaging area of ​​the first camera 11 and the third camera 13 is a second common imaging area 22 (FIG. 3(c)), and the common imaging area of ​​the second camera 12 and the third camera 13 is a third common imaging area 23 (FIG. 3(d)). In each common imaging area, parallax can be estimated by stereoscopic vision, allowing the distance to the imaged object to be calculated. As described above, this embodiment is implemented in a multi-camera system in which a plurality of cameras are installed on a vehicle so as to have a plurality of common imaging areas (overlapping imaging areas) in which at least a portion of the fields of view overlap.

[0019] The camera arrangement of the multi-camera system is not limited to those shown in Figures 3(a) to 3(d), and any combination can be selected as long as the cameras are arranged to have a common imaging area. For example, a front camera with an imaging range in front of the vehicle may be installed, and parallax may be calculated using a camera pair that shares a field of view, or a rear camera with an imaging range behind the vehicle may be added. Cameras with different specifications, such as focal length, image size, and lens model, may be selected for each camera.

[0020] Furthermore, correct values ​​of distance data are required when training a disparity calculation model for each common imaging area. Distance data is acquired using a distance measurement sensor such as a LiDAR, so a distance measurement sensor is also installed on the vehicle 10. As shown in FIG. 3( a), a LiDAR 14 is often installed as a distance measurement sensor in the center of the roof of the vehicle 10, but any position can be used as long as it can measure the distances to the first to third common imaging areas. Multiple LiDARs may also be installed to cover the blind spots of each LiDAR.

[0021] The data storage unit 100 (FIG. 1) stores data acquired by a vehicle equipped with the aforementioned multi-camera system. The data storage unit 100 stores images captured by each camera and sensor data from the ranging sensors, along with the sensor ID and a timestamp. If the vehicle's behavior can be acquired from an IMU (Inertial Measurement Unit) or a vehicle CAN, the data storage unit 100 may also store the sensor data and the vehicle behavior. The data storage unit 100 also stores calibration data for each sensor (such as internal and external parameters for each camera and the installation position of the ranging sensor). The calibration data may be values ​​stored at the time the sensor was installed, or values ​​corrected during driving to account for aging and temperature changes.

[0022] The learning policy determination unit 200 ( FIG. 1 ) determines the order in which to learn the parallax calculation models based on the type and arrangement of each sensor. In the example of FIGS. 3A to 3D , when there are three types of parallax calculation models, namely, a first parallax calculation model that infers the parallax of the first common imaging area 21, a second parallax calculation model that infers the parallax of the second common imaging area 22, and a third parallax calculation model that infers the parallax of the third common imaging area 23, the learning policy determination unit 200 determines the order in which to learn the parallax calculation models.

[0023] The training of the disparity calculation model will be described with reference to FIG. 4. In training the disparity calculation model, images captured by two cameras and the correct values ​​of the disparity images are used. The captured images (right and left images) are input to the disparity calculation model, and an inference result for the disparity image is output. The correct values ​​of the prepared disparity images are compared with the inference result, and the parameters of the disparity calculation model are updated so that an inference result closer to the correct value is output (the difference between the correct value of the disparity image and the inference result becomes smaller). The parameters of the disparity calculation model may be updated so that the inference result and the correct value are similar, or the parameters may be updated so that similar feature vectors are output for the same pixel in the left and right images calculated from the correct values. Alternatively, disparity calculation models with different numbers of parameters may be prepared, and a disparity calculation model with a smaller number of parameters may be trained so that it is similar to the final output or the output of the intermediate layer of a disparity calculation model with a larger number of parameters.

[0024] The correct values ​​for disparity images use distance information measured by a distance measurement sensor such as LiDAR. The correct values ​​for disparity images can be created by converting the LiDAR distance into disparity and plotting it on image coordinates. When creating correct values ​​from LiDAR distance measurements, it is difficult to plot correct disparity values ​​for all pixels. This is because the number of LiDAR measurement points is fewer than the camera resolution. As a result, the number of correct values ​​created by LiDAR varies depending on the region of the image. In regions with a small number of correct values, the learning of the disparity calculation model does not progress, which may result in a decrease in accuracy.

[0025] One way to improve the learning accuracy in areas with low LiDAR point density is to add disparity information obtained by methods other than LiDAR to the correct value (hereinafter referred to as additional disparity information) to support the learning of the disparity calculation model.

[0026] The additional parallax information will be described with reference to FIGS.

[0027] FIG. 5A shows disparity obtained without using a distance measurement sensor, which corresponds to a rule-based disparity calculation method. Conventional disparity calculation methods, such as block matching, can calculate disparity from only the left and right images without requiring a correct answer value for learning. Therefore, a disparity calculation model can be trained using the disparity calculated by the rule-based method as the correct answer value. Alternatively, the inference result of a disparity calculation model trained in advance using a different data set may be used as the correct answer value.

[0028] In Figure 5(b), additional disparity information is created using a feature point matching technique. Feature point matching is an algorithm that searches for identical feature points (corners or pixels with large changes in brightness) between two images. Feature point matching searches for identical pixels, and disparity can be calculated from the difference in their coordinates. A representative feature point matching technique is SIFT (Scale Invariant Feature Transform). In recent years, methods that can match feature points at high density using deep learning have also been developed, so these may also be used.

[0029] In FIG. 5(c), additional parallax information is created using the parallax calculated in another common imaging region. Because the first common imaging region 21 shown in FIG. 3(a) and the third common imaging region 23 shown in FIG. 3(c) overlap, the first parallax calculation model for the first common imaging region 21 can be trained using the parallax calculated in the third common imaging region 23. While FIG. 5(c) illustrates an example in which information from the third common imaging region 23 is used to train the first common imaging region 21, similar processing is possible for other overlapping common imaging regions. From the third common imaging region 23, in addition to the additional parallax information calculated using the method in FIGS. 5(a) and 5(b) (hereinafter referred to as the third additional parallax information), the inference results of the third parallax calculation model trained using LiDAR and the third additional parallax information can be obtained. Recent disparity calculation models using deep learning can estimate high-density parallax across the entire image, enabling high-density parallax inference results to be obtained for the third common imaging region 23. The lack of points in the LiDAR can be compensated for by using this information (third additional parallax information, inference result of the third parallax calculation model) as additional parallax information (hereinafter, first additional parallax information) for the first common imaging region 21. Furthermore, in the first common imaging region 21, the additional parallax information calculated by the method of FIGS. 5A and 5B may also be included as the first additional parallax information.

[0030] In order to improve the accuracy of the parallax calculation model using the additional parallax information, it is necessary to calculate the additional parallax information with high accuracy. The accuracy of the additional parallax information will be described with reference to Figures 6(a) and 6(b). Figure 6(a) shows an example in the second common imaging area 22. The first camera 11 and the third camera 13 constituting the second common imaging area 22 have a long base line length (the distance between the cameras), so that the same object is captured with a significantly different shape.

[0031] Using Figure 7, we will explain the shape change when the same object is imaged with different cameras. For an object on the road surface, in a stereo camera with camera height H and baseline length (distance between cameras) B, the point pr(x,y,1) on the right image corresponding to point pl(x,y,1) on the left image is calculated using equation (1). In this case, the degree of deformation increases according to the value of B / H, so the shape change is more severe for a camera pair with a longer baseline length. [Equation 1]

[0032] Furthermore, since the first camera 11 and the third camera 13 are installed at different camera heights, the surfaces that are imaged differ even for the same object. The first camera 11 is installed at a low position, so it can mainly image the side of the parallel running vehicle 30. On the other hand, the third camera 13 is installed on the roof of the vehicle itself, so it captures images looking down on the parallel running vehicle 30. Therefore, both the side and top of the parallel running vehicle 30 are imaged.

[0033] 6(b), the second camera 12 and the third camera 13 constituting the third common imaging area 23 have short baseline lengths and are installed at the same height, so that the parallel traveling vehicle 30 can be imaged with almost the same shape. In addition, there are few occlusion areas, so the same surface of the object can be observed in both images.

[0034] In stereo distance measurement, disparity is estimated by searching for identical pixels between two images. The methods shown in FIGS. 5( a) and 5(b) often search for identical pixels based on similarities in local regions of the images. However, if the two images are captured differently (shape changes, image capture surface, etc.), the identical pixels cannot be identified and disparity cannot be calculated correctly. When there are significant shape changes and many occlusion areas, as in FIG. 6(a), the accuracy of the additional disparity information decreases, which may worsen the training accuracy of the disparity calculation model. On the other hand, in FIG. 6(b), there are few shape changes and few occlusion areas, and the additional disparity information can be calculated accurately. Therefore, adding it to training can improve the training accuracy of the disparity calculation model.

[0035] 6A and 6B , by using the disparity information of the common imaging region for which additional parallax information can be calculated with high accuracy for training a different common imaging region, that is, by using the disparity information of the common imaging region for which additional parallax information can be calculated with high accuracy for training a parallax calculation model of a common imaging region assigned later in the execution order, it is possible to improve the training accuracy of the other parallax calculation models. In the example of Figures 6A and 6B , training is started from the third common imaging region 23 in Figure 6B, and the parallax information calculated at that time is used as the additional parallax information for the second common imaging region 22 in Figure 6A, it is possible to improve the training accuracy of the second parallax calculation model for the second common imaging region 22.

[0036] The learning policy determination unit 200 (FIG. 1) includes a sensor placement priority determination unit 201 , a sensor type priority determination unit 202 , and a common imaging area order determination unit 203 .

[0037] The sensor placement priority determination unit 201 estimates the difficulty of parallax calculation (inference) for each of the first common imaging area 21, the second common imaging area 22, and the third common imaging area 23 from the positional relationship of each sensor, and determines the learning priorities of the first parallax calculation model, the second parallax calculation model, and the third parallax calculation model. As described in FIG. 5C , when the common imaging areas overlap, the parallax calculated in another common imaging area can be used for learning the parallax calculation model. Therefore, by starting learning of the parallax calculation model from a common imaging area with high accuracy of additional parallax information and using the additional parallax information created at that time for learning another parallax calculation model, the parallax calculation model can be learned with high accuracy.

[0038] As explained in Figure 7, the change in shape between two images becomes larger as the distance between the cameras increases. Therefore, by starting learning preferentially from camera pairs with a short distance between the cameras, which are considered to have a lower difficulty level for disparity calculation, the learning accuracy of subsequent disparity calculation models can be improved.

[0039] In addition, cameras located closer to the LiDAR that provides the correct value have less occlusion and are able to learn the correct distance, so the learning priority for the common imaging area of ​​camera pairs that are close to the LiDAR is set higher.

[0040] The sensor type priority determination unit 202 estimates the difficulty of parallax calculation (inference) from the sensor type (differences in specifications such as focal length, image size, lens model, etc.) (characteristics) and determines the learning priority. Since sensors of the same type have higher accuracy of additional parallax information, a higher learning priority is set.

[0041] The common imaging area order determination unit 203 determines the order in which the learning of the parallax calculation model is to be executed, based on the priorities output by the sensor placement priority determination unit 201 and the sensor type priority determination unit 202 .

[0042] 8 shows the processing flow of the learning policy determination unit 200. The learning policy determination unit 200 acquires images captured by each camera and sensor data from the distance measurement sensor, which are stored in the data storage unit 100.

[0043] First, in step S101, a priority is calculated for all common imaging areas based on the difficulty of calculating parallax (sensor placement priority determination unit 201). The difficulty is an index indicating how accurately parallax can be calculated in the common imaging area, and uses the magnitude of the shape change between two images caused by the distance between the cameras, as described in FIG. 7. In FIG. 7, the degree of deformation of an object on the road surface is calculated, but the degree of deformation relative to a plane other than the road surface can also be calculated. Therefore, when it is desired to accurately calculate the distance to a vehicle traveling parallel to the road, the degree of shape change relative to a plane perpendicular to the road surface may be used as the difficulty index, or any other index may be used.

[0044] In step S102, the sensor placement priority determination unit 201 calculates a priority based on the positional relationship between the LiDAR and the camera (hereinafter referred to as the reference camera) that calculates the disparity. The LiDAR generates a correct value based on the acquired point cloud data. If the LiDAR and the reference camera are located far apart, the correct value may include a distance different from the object captured by the reference camera due to differences in the occlusion areas of the two sensors. For example, if a camera captures a pedestrian, but a LiDAR installed in a different location can see the road surface behind the pedestrian, the disparity of the background road surface may be mistakenly plotted in the pixel where the pedestrian is captured. On the other hand, if the LiDAR and the reference camera are installed close to each other, the difference in the observable areas between the two sensors is small, allowing the disparity to be plotted correctly. Considering the above, the closer the distance between the LiDAR and the reference camera, the more accurately the disparity calculation model can be trained, and therefore the higher the priority is set.

[0045] In step S102, the priority may be calculated based on the number of LiDAR points included in the common imaging area. If the positions, orientations, and angles of view of the camera and LiDAR are known, the number of LiDAR points that will appear within the imaging range can be predicted. Since the greater the number of LiDAR points, the higher the learning accuracy of the disparity calculation model. Therefore, the greater the number of LiDAR points in each common imaging area, the higher the priority may be set.

[0046] In step S103, a priority is calculated based on the sensor type (sensor type priority determination unit 202). The cameras that make up the multi-camera system may have different focal lengths, angles of view, image sizes, lens models, and so on. When preprocessing required for parallax calculation, such as distortion correction and rectification, is performed on cameras of different types, the spatial resolution differs even in the same area, making it difficult to calculate parallax. Therefore, the more similar the type of camera, the lower the difficulty of parallax calculation, and a higher priority is set.

[0047] In step S104, the priorities calculated in steps S101, S102, and S103 are integrated (common imaging area order determination unit 203). An average of the priorities may be used, or weighted priorities may be added together.

[0048] In step S105, the order is determined so that learning proceeds in order from the common imaging area with the highest priority calculated in step S104 (common imaging area order determination unit 203). In this way, the disparity information of the common imaging area for which additional disparity information can be calculated with high accuracy is used for learning of the subsequent common imaging area, thereby improving the learning accuracy of the subsequent disparity calculation model.

[0049] FIG. 8 shows an example in which the priority is calculated in steps S101, S102, and S103, but at least one of steps S101, S102, and S103 may be used, or the priority may be determined based on another index.

[0050] The disparity calculation model learning unit 300 learns a disparity calculation model based on the sensor data stored in the data storage unit 100 and the learning order output by the learning policy determination unit 200. The disparity calculation model learning unit 300 includes a disparity correct value creation unit 301, a learning and evaluation unit 302, and a model storage unit 303.

[0051] The disparity correct value creation unit 301 creates disparity images to be given as correct values ​​to the disparity calculation model in accordance with the learning order received from the learning policy determination unit 200. First, a disparity image is created based on the LiDAR data stored in the data storage unit 100. At this time, if the data storage unit 100 includes the vehicle behavior acquired from an IMU (Inertial Measurement Unit) or a vehicle CAN, a higher density of correct values ​​can be created by overlaying the LiDAR data on the vehicle behavior in chronological order.

[0052] Next, additional parallax information is added to the created parallax image. If there is a common imaging area (hereinafter referred to as the fifth common imaging area) that precedes the common imaging area (hereinafter referred to as the fourth common imaging area) for which the correct value is created, and if the fourth common imaging area and the fifth common imaging area have an overlapping area, the parallax calculated in the fifth common imaging area is added to the correct value of the fourth common imaging area. In other words, the correct value of the fourth common imaging area includes the parallax calculated in the fifth common imaging area. If there are multiple fifth common imaging areas, the parallaxes of all the common imaging areas may be added to the correct value, or only the parallax that is earlier in the order (higher accuracy) may be added to the correct value.

[0053] Furthermore, in the fourth common imaging region, the parallax (disparity obtained by rule-based model and feature point matching) is calculated using the method of FIGS. 5A and 5B and added to the correct answer value.

[0054] In the above example, the correct value is created by integrating the disparity of the LiDAR and the additional disparity information of the fourth common imaging region and the fifth common imaging region into the correct value. If multiple correct disparity values ​​are obtained for the same pixel, the average value of the multiple disparities is set as the correct value. Also, if LiDAR is obtained, the LiDAR value may be used as the correct value instead of the average value. Also, additional disparity information from an earlier order may be used as the correct value.

[0055] In addition, in a mass-produced vehicle that is not equipped with a LiDAR, only the additional parallax information of the fourth common imaging region and the fifth common imaging region may be used as the correct value.

[0056] The learning and evaluation unit 302 uses the correct values ​​created by the correct disparity value creation unit 301 to learn a disparity calculation model and evaluate its accuracy. The correct values ​​created by the correct disparity value creation unit 301 are divided into data sets for learning and evaluation, and the disparity calculation model is learned using the learning data set as shown in FIG. 4. Furthermore, the estimation accuracy of the learned disparity calculation model is evaluated using the evaluation data set. If there is no problem with the accuracy, it is output to the model storage unit 303.

[0057] The model storage unit 303 stores the disparity calculation model output by the learning and evaluation unit 302. The stored disparity calculation model is subjected to processes such as pruning and quantization to increase the processing speed in the actual device, and then incorporated into the actual device.

[0058] <Example 2> A learning system 2 according to Example 2 of the present invention will be described with reference to Fig. 9. Fig. 9 is a configuration diagram of the learning system 2 according to Example 2 of the present invention, which is configured when considering cooperation (communication) between edge devices and servers and clouds. Note that a description of configurations that are the same as or equivalent to those described in Example 1 will be omitted, and the following description will mainly focus on differences.

[0059] The edge device 400 is an edge device including a mobile object such as a vehicle or a mobile robot, and an infrastructure sensor such as a surveillance camera, and is installed so that the fields of view of the multiple cameras overlap. The edge device 400 includes a sensor unit 401, a camera change detection unit 402, a camera anomaly detection unit 403, a model verification unit 404, a model inference unit 405, and an edge-side transceiver unit 406. Hereinafter, the edge device 400 will be described as an in-vehicle information processing device mounted on a vehicle equipped with a multi-camera system, but is not limited to an in-vehicle information processing device.

[0060] The sensor unit 401 is composed of multiple cameras, which are installed to share a field of view. Images captured by the cameras are transmitted to the server 500 at regular intervals for use in re-learning the disparity calculation model. A distance measurement sensor such as a LiDAR may also be installed, and the distance measurement results are transmitted to the server 500 together with the camera images. If information that reveals the vehicle's driving status and driving environment can be obtained, it may also be transmitted to the server 500. This information may include location information from an IMU (Inertial Measurement Unit), a vehicle CAN, and a Global Navigation Satellite System (GNSS), the operating status of the wipers and headlights, the time, and illuminance.

[0061] The data to be transmitted may be limited to only data that meets certain conditions. For example, only data on roads that have not been used to train the disparity calculation model may be transmitted based on GNSS position information, or only scenes with few training images may be transmitted based on the time, weather, etc.

[0062] The camera change detection unit 402 detects that at least one of the cameras constituting the multi-camera system has been changed and requests the server 500 to retrain the disparity calculation model. Cameras installed in mass-produced vehicles may be replaced with new cameras due to aging, malfunctions, or other reasons. Due to the availability of parts and improvements in the performance of the camera itself, the specifications of the new camera, such as focal length, image size, and lens model, may differ from those of the previous camera. In this case, the images captured by the new camera differ from those captured by the previous camera. If the input to the disparity calculation model differs from that at the time of training, the accuracy of parallax estimation decreases, and therefore the disparity calculation model must be retrained. The camera change detection unit 402 stores the serial number of the camera at the time of production and outputs a retraining request to the server 500 when a camera with a different serial number is installed. Furthermore, if the camera specifications, such as focal length, image size, and lens model, are known, the server 500 may also output a retraining request when any of these specifications is changed. Furthermore, if the installation conditions such as the position and direction of the camera are known, a request for re-learning may be output to the server 500 at the timing when any of these changes is significant.

[0063] The camera anomaly detection unit 403 detects an abnormality in at least one of the cameras constituting the multi-camera system and outputs an abnormality flag to the server 500. In a multi-camera system, cameras may be installed outside the vehicle / outdoors, and lens dirt may change the color and contrast of the captured image. Images may also be captured incorrectly due to mud, dirt, raindrops, or other obstructions, or due to lens cracks caused by impact. When the image color and contrast change due to lens dirt, re-learning the parallax calculation model can improve the parallax accuracy. However, obstructions or lens cracks prevent stereo vision, so re-learning the parallax calculation model does not improve performance. The camera anomaly detection unit 403 outputs (notifies) the camera's abnormal state (change in sensor state) to the server 500, which determines the timing for re-learning the parallax calculation model. If the type of abnormal state (such as dirt, adhesions, or cracks) can also be estimated, the type of abnormality is also transmitted to the server 500.

[0064] In addition, the camera abnormality detection unit 403 not only transmits the abnormal status of the camera to the server 500, but also displays the camera abnormality to the vehicle driver and on the infrastructure system management screen, etc., to alert the user to the camera abnormality and encourage them to take action to recover from the abnormality.

[0065] The model verification unit 404 verifies the operation of the re-trained model distributed from the server 500. By verifying in the background whether the re-trained model operates without problems in the real environment, it is possible to prevent system malfunctions caused by erroneous output from the disparity calculation model. Using an image of the real environment as input, it is verified in the background whether the output is the same as that of the model before re-training, or whether there is any abnormal behavior when the system is controlled using the output of the re-trained model.

[0066] The model inference unit 405 is a block that operates the disparity calculation model in a real environment, inputs camera images, outputs disparity, and operates the multi-camera system based on the output disparity. In addition to using a trained model at the time of commercialization, it is also possible to receive a model whose operation has been verified for a certain period of time by the model verification unit 404, and update the model for use.

[0067] The edge-side transmitting / receiving unit 406 communicates with a server-side transmitting / receiving unit 501 (described later) and transmits images captured by the sensor unit 401 to the server 500. The edge-side transmitting / receiving unit 406 also receives a parallax calculation model from the server 500.

[0068] The server 500 receives information from the edge device 400, re-learns the disparity calculation model, and distributes it to the edge device 400. The server 500 includes a server-side transmitting / receiving unit 501, a data storage unit 502, a learning timing determination unit 503, a learning policy determination unit 504, and a disparity calculation model learning unit 505.

[0069] The server-side transmitting / receiving unit 501 (hereinafter also referred to as an image acquisition unit) communicates with the edge-side transmitting / receiving unit 406 of the edge device 400, and receives (acquires) images captured by the sensor unit 401. In addition, the server-side transmitting / receiving unit 501 transmits a disparity calculation model to the edge device 400.

[0070] The data storage unit 502 stores data transmitted from the edge device 400 and a disparity calculation model to be distributed to the edge device 400 .

[0071] The learning timing determination unit 503 refers to the data stored in the data storage unit 502 and determines the timing to re-learn the disparity calculation model.

[0072] The disparity calculation model is re-learned when data that has not been used to learn the disparity calculation model operating in the model inference unit 405 is found from the data on the driving conditions and driving environment output from the sensor unit 401. Data that is not included in the learning image is identified from GNSS position information, the operating status of the wipers and headlights, illuminance, and weather forecast.

[0073] Furthermore, if the camera installed in the edge device 400 is changed or an abnormality occurs, the disparity calculation model is retrained. There are multiple types of abnormal camera conditions, and if the color or contrast of the captured image changes due to lens dirt or the like, the performance of the disparity calculation model can be maintained even in scenes with dirty lenses by retraining the disparity calculation model using that image. On the other hand, in situations where stereoscopic vision is difficult because the camera image cannot be captured normally due to adhesions such as mud or dirt, or cracks in the lens, the disparity calculation model is not retrained. Furthermore, if such an image is sent to the server 500, it is excluded from the disparity calculation model retraining targets.

[0074] The learning policy determination unit 504 and the disparity calculation model learning unit 505 have the same functions as the learning policy determination unit 200 and the disparity calculation model learning unit 300 described in the first embodiment.

[0075] The learning policy determination unit 504 may re-learn the disparity calculation model for all common imaging areas of the multi-camera system, or may re-learn only for common imaging areas that satisfy specific conditions, such as when a camera is updated (the sensor configuration has changed), when an abnormality has occurred (the sensor state has changed), or when a new driving scene has been acquired (additional data has been collected). When re-learning the disparity calculation model for all common imaging areas, the learning order is determined for each common imaging area described in Example 1, and learning of the disparity calculation model is advanced according to that order. On the other hand, when re-learning the disparity calculation model for some common imaging areas that satisfy the conditions, the learning order is determined for all common imaging areas that overlap with the common imaging area to be re-learned, and learning of the disparity calculation model is advanced according to that order.

[0076] In the disparity calculation model learning unit 505, since mass-produced vehicles are often not equipped with LiDAR for acquiring correct values, in such cases the disparity calculation model is re-learned using only the additional disparity information. The re-learned disparity calculation model is stored in the data storage unit 502. The re-learned disparity calculation model stored in the data storage unit 502 is delivered to the model verification unit 404 via the edge-side transceiver unit 406 of the edge device 400, and after its operation in the real environment has been verified for a certain period of time, it is output to the model inference unit 405 and used in the multi-camera system of the vehicle in which the edge device 400 is installed.

[0077] <Modifications> In the above-described embodiment, the camera configuration of the multi-camera system may be changed between collecting training data and operating the system. Using the camera configurations of FIGS. 3(a) to 3(d) as an example, the first camera 11 and the third camera 13 have a longer baseline, which reduces the accuracy of the additional parallax information. Therefore, a fourth camera (not shown) is additionally installed (added) between the first camera 11 and the third camera 13 only during data collection. After calculating additional parallax information between the first camera 11 and the fourth camera and between the third camera 13 and the fourth camera, all of the additional parallax information is used to train a parallax calculation model for the first camera 11 and the third camera 13. By installing the fourth camera in the middle, a camera pair with a short baseline is created, and the highly accurate parallax information calculated there can be used to improve the accuracy of the third parallax calculation model. The fourth camera is installed only during data collection and can be removed when the system is actually operating.

[0078] <Summary> As described above, the learning method according to this embodiment is a learning method executed by a computer, and includes the steps of acquiring images captured by a plurality of cameras (11, 12, 13) having (a plurality of) common imaging areas (21, 22, 23) whose fields of view at least partially overlap, determining an execution order for learning of a parallax inference model that infers parallax using the images for each of the plurality of common imaging areas (21, 22, 23) (learning policy determination unit 200; FIG. 8 ), and inputting images acquired from the plurality of cameras having a common imaging area to be learned into the parallax inference model according to the determined execution order, and learning the parallax inference model from the difference between the obtained parallax and ground truth data (parallax calculation model learning unit 300; FIG. 4 ), wherein the ground truth data includes parallax information of a common imaging area that is assigned to an execution order earlier than the common imaging area to be learned.

[0079] As a result, a parallax calculation model is learned in order for the plurality of common imaging regions (21, 22, 23).

[0080] The method also includes a step of estimating the difficulty of parallax inference for each of the multiple common imaging areas (21, 22, 23) based on at least one of the characteristics and arrangement of the multiple cameras (11, 12, 13) (sensor arrangement priority determination unit 201 and sensor type priority determination unit 202 of the learning policy determination unit 200; steps S101 and S103 in Figure 8), and determining the order of execution of learning of the parallax inference model for the multiple common imaging areas (21, 22, 23) based on the results of the estimation.

[0081] This allows the order of learning the disparity calculation models to be determined depending on the difficulty of disparity inference.

[0082] It also includes a step of generating the correct value data based on point cloud data acquired by a LiDAR installed in a vehicle equipped with the multiple cameras (11, 12, 13), and determining the order of execution of learning of the parallax inference model for multiple common imaging areas (21, 22, 23) based on the positional relationship between the multiple cameras and the LiDAR (sensor placement priority determination unit 201 of the learning policy determination unit 200; step S102 in Figure 8).

[0083] This determines the learning order of the disparity calculation model based on the positional relationship of the LiDAR.

[0084] The correct value data also includes at least one of the parallax values ​​calculated by a rule-based model or feature point matching based on the acquired image (FIG. 5).

[0085] This allows the correct disparity value to be created using a method different from LiDAR.

[0086] In addition, the order of execution of learning the parallax inference model is determined according to at least one of changes in sensor configuration, changes in sensor state, and the status of collection of additional data, and the parallax inference model is re-learned (Example 2).

[0087] This will enable model updates after commercialization.

[0088] In addition, based on the number (number of points) of LiDAR point cloud data contained in each of the multiple common imaging areas (21, 22, 23), the order in which the learning of the parallax inference model is executed for the multiple common imaging areas (21, 22, 23) is determined (step S102 in Figure 8).

[0089] This allows the learning order of the disparity calculation model to be determined, for example, based on the number (number of points) of LiDAR point cloud data that can be predicted from the positional relationship of the LiDAR.

[0090] Furthermore, the learning system 2 according to this embodiment is a learning system 2 including an in-vehicle information processing device (edge ​​device 400) mounted on a vehicle 10 equipped with a plurality of cameras (11, 12, 13) having common imaging areas (21, 22, 23) whose fields of view overlap at least partially, and a server device (server 500) that communicates with the in-vehicle information processing device (edge ​​device 400), and is equipped with an image acquisition unit (server-side transceiver unit 501) that acquires images captured by the plurality of cameras (11, 12, 13), a learning policy determination unit 504 that determines the execution order of learning of a disparity calculation model for the plurality of common imaging areas (21, 22, 23), and a disparity calculation model learning unit 505 that inputs images acquired from the plurality of cameras having the common imaging area to be learned into the disparity calculation model according to the determined execution order, and learns the disparity calculation model from the difference between the obtained disparity and correct value data, and the correct value data includes disparity information of the common imaging area that is assigned to an execution order earlier than the common imaging area to be learned.

[0091] As a result, a parallax calculation model is learned in order for the plurality of common imaging regions (21, 22, 23).

[0092] In addition, in the learning system 2 according to this embodiment, the learning policy determination unit 504 estimates the difficulty of inferring parallax for each of the multiple common imaging areas (21, 22, 23) based on at least one of the characteristics and arrangement of the multiple cameras (11, 12, 13), and determines the order in which to execute learning of the parallax calculation model for the multiple common imaging areas (21, 22, 23) based on the estimation result.

[0093] This allows the order of learning the disparity calculation models to be determined depending on the difficulty of inferring disparity.

[0094] In addition, the learning system 2 according to this embodiment further includes a disparity correct value creation unit 301 that generates correct value data based on point cloud data acquired by a LiDAR installed in the vehicle, and the learning policy determination unit 504 determines the execution order of learning the disparity calculation model for multiple common imaging areas (21, 22, 23) based on the positional relationship between the multiple cameras and the LiDAR.

[0095] This determines the learning order of the disparity calculation model based on the relative positions of the camera and LiDAR.

[0096] In addition, the learning system 2 according to this embodiment further includes a disparity correct value creation unit 301 that generates correct value data based on point cloud data acquired by a LiDAR arranged in a vehicle, and the learning policy determination unit 504 determines the execution order of learning the disparity calculation model for the multiple common imaging areas (21, 22, 23) based on the number (number of points) of LiDAR point cloud data included in each of the multiple common imaging areas (21, 22, 23).

[0097] This allows the learning order of the disparity calculation model to be determined, for example, based on the number (number of points) of LiDAR point cloud data that can be predicted from the positional relationship of the LiDAR.

[0098] Furthermore, in the learning system 2 according to this embodiment, the correct answer data includes at least one of the disparity calculated by the rule-based model or feature point matching (FIG. 5).

[0099] This allows the correct disparity value to be created using a method different from LiDAR.

[0100] Furthermore, in the learning system 2 according to this embodiment, when the in-vehicle information processing device (edge ​​device 400) detects that at least one of the multiple cameras has been changed, it requests the server device (server 500) to re-learn the disparity calculation model for the common imaging area that includes the imaging area of ​​the changed camera (camera change detection unit 402), and the server device (server 500) generates a re-learned model by re-learning the disparity calculation model and distributes the re-learned model to the in-vehicle information processing device (edge ​​device 400).

[0101] This allows the disparity calculation model to be re-learned when the camera is changed, making it possible to update the model after commercialization.

[0102] Furthermore, in the learning system 2 according to this embodiment, when the in-vehicle information processing device (edge ​​device 400) detects that an abnormality has occurred in at least one of the multiple cameras, the in-vehicle information processing device notifies the server device (server 500) of the abnormal state of the camera in which the abnormality has been detected (camera abnormality detection unit 403), and the server device (server 500) determines, based on the abnormal state, whether or not to re-learn the disparity calculation model for the common imaging area that includes the imaging area of ​​the camera in which the abnormality has been detected, re-learns the disparity calculation model for which it has been determined that re-learning should be performed, thereby generating a re-learned model, and distributes the re-learned model to the in-vehicle information processing device (edge ​​device 400).

[0103] This allows the disparity calculation model to be appropriately re-learned depending on the abnormality that occurs in the camera, making it possible to update the model after commercialization.

[0104] Furthermore, in the learning system 2 according to this embodiment, an in-vehicle information processing device (edge ​​device 400) transmits a plurality of images captured by a plurality of cameras to a server device (server 500), and the server device (server 500) determines, based on the plurality of images, whether or not the plurality of images contain data that has not been used in learning at least one of the plurality of disparity calculation models. If the plurality of disparity calculation models contain data that has not been used in learning at least one of the plurality of disparity calculation models, the server device generates a re-learned model by re-learning the disparity calculation model, and distributes the re-learned model to the in-vehicle information processing device (edge ​​device 400).

[0105] This allows the trained disparity calculation model to properly re-train scenes that have not yet been trained, making it possible to update the model after commercialization.

[0106] According to this embodiment, the disparity calculation model is trained in an ordered manner for a plurality of common imaging areas, in other words, by determining the training order of the disparity calculation model, the accuracy of the correct disparity value used for training can be increased, and the training accuracy of the disparity calculation model can be improved.

[0107] In other words, according to this embodiment, in multiple common imaging areas in a multi-camera system, learning is started from an area where the difficulty of disparity calculation is low, and the additional disparity information calculated during the learning process is used to learn subsequent common imaging areas, thereby improving the learning accuracy of the disparity calculation model.

[0108] Although the embodiments of the present invention have been described above, the above embodiments merely show some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above embodiments.

[0109] It should be noted that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations.

[0110] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a storage device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0111] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0112] 1 Image processing device (embodiment 1) 100 Data storage unit 200 Learning policy determination unit 201 Sensor placement priority determination unit 202 Sensor type priority determination unit 203 Common imaging area order determination unit 300 Parallax calculation model learning unit 301 Parallax correct value creation unit 302 Learning and evaluation unit 303 Model storage unit 2 Learning system (embodiment 2) 400 Edge device (on-vehicle information processing device) 401 Sensor unit 402 Camera change detection unit 403 Camera abnormality detection unit 404 Model verification unit 405 Model inference unit 406 Edge side transmission and reception unit 500 Server (server device) 501 Server side transmission and reception unit (image acquisition unit) 502 Data storage unit 503 Learning timing determination unit 504 Learning policy determination unit 505 Parallax calculation model learning unit

Claims

1. A learning system including an on-board information processing device mounted on a vehicle equipped with multiple cameras having a common imaging area with at least a portion of their fields of view overlapping, and a server device that communicates with the on-board information processing device, comprising: an image acquisition unit that acquires images captured by the multiple cameras; a learning policy determination unit that determines an execution order for learning a disparity calculation model for the multiple common imaging areas; and a disparity calculation model learning unit that inputs images acquired from the multiple cameras having the common imaging area to be learned into the disparity calculation model according to the determined execution order, and learns the disparity calculation model from the difference between the obtained disparity and ground truth value data, wherein the ground truth value data includes disparity information of a common imaging area that is assigned to an execution order earlier than the common imaging area to be learned.

2. A learning system as described in claim 1, wherein the learning policy determination unit estimates the difficulty of inferring disparity for each of a plurality of common imaging areas based on at least one of the characteristics and arrangement of the plurality of cameras, and determines the order in which to execute learning of the disparity calculation model for the plurality of common imaging areas based on the result of the estimation.

3. A learning system as described in claim 1, further comprising a disparity correct value creation unit that generates the correct value data based on point cloud data acquired by a LiDAR arranged on the vehicle, and the learning policy determination unit determines the execution order of learning of the disparity calculation model for multiple common imaging areas based on the positional relationship between the multiple cameras and the LiDAR.

4. A learning system as described in claim 1, further comprising a disparity correct value creation unit that generates the correct value data based on point cloud data acquired by a LiDAR arranged on the vehicle, and the learning policy determination unit determines the order in which learning of the disparity calculation model is performed for multiple common imaging areas based on the number of LiDAR point cloud data included in each of the multiple common imaging areas.

5. A learning system according to claim 1, wherein the correct answer data includes at least one of a rule-based model and a disparity calculated by feature point matching.

6. A learning system as described in claim 1, wherein, when the in-vehicle information processing device detects that at least one of the plurality of cameras has been changed, the in-vehicle information processing device requests the server device to re-learn a disparity calculation model for a common imaging area that includes the imaging area of ​​the changed camera, the server device generates a re-learned model by re-learning the disparity calculation model, and distributes the re-learned model to the in-vehicle information processing device.

7. A learning system as described in claim 1, wherein, when the in-vehicle information processing device detects that an abnormality has occurred in at least one of the multiple cameras, it notifies the server device of the abnormal state of the camera in which the abnormality has been detected, and the server device determines, based on the abnormal state, whether or not to re-train a disparity calculation model for a common imaging area that includes the imaging area of ​​the camera in which the abnormality has been detected, generates a re-trained model by re-training the disparity calculation model for which it has determined that re-training should be performed, and distributes the re-trained model to the in-vehicle information processing device.

8. A learning system as described in claim 1, wherein the in-vehicle information processing device transmits a plurality of images captured by the plurality of cameras to the server device, and the server device determines, based on the plurality of images, whether or not data that has not been used in training at least one of the plurality of disparity calculation models is included, and if data that has not been used in training at least one of the plurality of disparity calculation models is included, generates a re-trained model by re-training the disparity calculation model, and distributes the re-trained model to the in-vehicle information processing device.

9. A learning method executed by a computer, comprising: a step of acquiring images captured by a plurality of cameras having a common imaging area with at least a portion of their fields of view overlapping; a step of determining an execution order for learning of a disparity inference model that infers disparity using the images for each of the plurality of common imaging areas; and a step of inputting images acquired from the plurality of cameras having a common imaging area to be learned into the disparity inference model according to the determined execution order, and learning the disparity inference model from the difference between the obtained disparity and ground truth data, wherein the ground truth data includes disparity information of a common imaging area assigned in an execution order earlier than the common imaging area to be learned.

10. A learning method as claimed in claim 9, comprising a step of estimating the difficulty of inferring parallax for each of a plurality of common imaging areas based on at least one of the characteristics and arrangement of the plurality of cameras, and determining the order in which to execute learning of the parallax inference model for the plurality of common imaging areas based on the result of said estimation.

11. A learning method as described in claim 9, comprising a step of generating the correct value data based on point cloud data acquired by a LiDAR arranged on a vehicle equipped with the multiple cameras, and determining the order of execution of learning of a disparity inference model for multiple common imaging areas based on the positional relationship between the multiple cameras and the LiDAR.

12. A learning method according to claim 9, wherein the correct answer data includes at least one of a parallax calculated by a rule-based model or feature point matching based on the acquired image.

13. A learning method according to claim 9, comprising a step of generating the ground truth data based on point cloud data acquired by a LiDAR disposed on a vehicle equipped with the plurality of cameras, and determining the order of execution of learning of the disparity inference model for the plurality of common imaging areas based on the number of LiDAR point cloud data included in each of the plurality of common imaging areas.

14. A learning method according to claim 9, wherein the order of execution of learning of the disparity inference model is determined in accordance with at least one of a change in sensor configuration, a change in sensor state, and a status of collection of additional data, and the disparity inference model is re-trained.

Images