Learning system and learning method

Abstract

To provide a learning system and a learning method capable of enhancing accuracy of a parallax correct answer value used for learning and improving learning accuracy of a parallax calculation model.SOLUTION: A learning system includes: an image acquisition unit (server-side transmitting / receiving unit 501) which acquires images captured by a plurality of cameras; a learning policy determination unit 504 which determines an execution order of learning for a parallax calculation model with respect to a plurality of common imaging regions; and a parallax calculation model learning unit 505 which inputs, into the parallax calculation model, the images that are acquired from the plurality of cameras and that have a common imaging area to be learned according to the determined execution order, and subjects the parallax calculation model to learning according to the difference between the acquired parallax and correct value data. Therein the correct answer value data include parallax information of the common imaging region assigned to the execution order preceding the common imaging region to be learned.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a learning system and a learning method for learning a parallax inference model that infers parallax in a common imaging area of ​​a plurality of imaging devices in an image processing device that measures the distance to a detection target in the common imaging area. [Background technology]

[0002] In recent years, multi-camera systems in which multiple imaging devices (cameras) are installed on a moving object have been developed for autonomous driving and driver assistance technology. In multi-camera systems, cameras that share a common field of view can perform stereo distance measurement, enabling accurate estimation of the distance to an object.

[0003] In stereo distance measurement, the parallax, which is the difference in the position at which the same object is captured by two cameras (right and left cameras) that share a common field of view, is calculated. Because the parallax differs depending on the depth, the distance to the detected object can be calculated from the calculated parallax.

[0004] Conventionally, disparity has been calculated by calculating the similarity of each local region between images captured by left and right cameras (left and right images) using rule-based processing. In recent years, learning-based methods have been developed with the advancement of deep learning, typified by convolutional neural networks (CNNs). In learning-based methods, a disparity calculation model (also called a disparity inference model) is trained using multiple captured images and distance data that serves as the ground truth (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2020-80047 Summary of the Invention [Problem to be solved by the invention]

[0006] As described in Patent Document 1, "True depth can be obtained using a 3D sensor such as LiDAR (Laser Imaging Detection and Ranging)," conventional learning-based methods use a distance measurement sensor such as LiDAR to create correct distance data. However, the point density of LiDAR is generally low, and learning accuracy decreases in areas with low point density.

[0007] One way to compensate for the point density of LiDAR is to use additional disparity information created by methods other than LiDAR. Examples of additional disparity information include disparity calculated by a rule-based method or disparity calculated from the results of feature point matching, and other methods obtained by methods other than LiDAR. Also, disparity calculated in another common imaging area (also called overlapping imaging area) (rule-based disparity calculation, feature point matching, disparity calculation model, etc.) can be used.

[0008] However, in multi-camera systems, there are camera pairs with long baselines, different angles of view, and non-parallel cameras. In such camera pairs, the shapes of the left and right images change significantly, making it impossible to calculate additional disparity information correctly. If the disparity calculation model is trained using incorrect additional disparity information, the performance of the disparity calculation model may deteriorate.

[0009] The present invention has been made in consideration of the above circumstances, and aims to provide a learning system and a learning method that can increase the accuracy of the correct disparity values ​​used for learning and improve the learning accuracy of the disparity calculation model. [Means for solving the problem]

[0010] In order to solve the above problem, the learning system of the present invention is a learning system that includes an on-board information processing device mounted on a vehicle equipped with a plurality of cameras having a common imaging area whose fields of view overlap at least partially, and a server device that communicates with the on-board information processing device, and is equipped with an image acquisition unit that acquires images captured by the plurality of cameras, a learning policy determination unit that determines the execution order of learning of a disparity calculation model for the plurality of common imaging areas, and a disparity calculation model learning unit that inputs images acquired from the plurality of cameras having the common imaging area to be learned into the disparity calculation model according to the determined execution order, and learns the disparity calculation model from the difference between the obtained disparity and correct value data, and the correct value data includes disparity information of a common imaging area that is assigned to an execution order earlier than the common imaging area to be learned.

[0011] In addition, the learning method of the present invention is a learning method executed by a computer, and includes the steps of acquiring images captured by a plurality of cameras having a common imaging area in which at least a portion of the field of view overlaps, determining an execution order for learning of a disparity inference model that infers disparity using the images for each of the plurality of common imaging areas, and inputting images acquired from the plurality of cameras having the common imaging area to be learned into the disparity inference model according to the determined execution order, and learning the disparity inference model from the difference between the obtained disparity and correct value data, wherein the correct value data includes disparity information of a common imaging area assigned in an execution order earlier than the common imaging area to be learned. [Effects of the Invention]

[0012] According to the present invention, by ordering and training a disparity calculation model for a plurality of common imaging areas, in other words, by determining the training order of the disparity calculation model, the accuracy of the correct disparity value used for training can be increased, and the training accuracy of the disparity calculation model can be improved.

[0013] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a block diagram showing an example of the configuration of an image processing apparatus according to a first embodiment of the present invention. [Figure 2] FIG. 1 is a hardware configuration diagram of an image processing apparatus according to a first embodiment of the present invention. [Figure 3] This figure is used to explain the multi-camera system. (a) is a side view of an example of camera and LiDAR installation, (b) is an overhead view of the first common imaging area, (c) is an overhead view of the second common imaging area, and (d) is an overhead view of the third common imaging area. [Figure 4] 10A and 10B are diagrams illustrating learning of a disparity calculation model. [Figure 5] 10A and 10B are diagrams explaining additional parallax information, where (a) is additional parallax information obtained using parallax information (such as rule-based parallax) without using a ranging sensor, (b) is additional parallax information obtained using feature point matching, and (c) is an explanatory diagram of additional parallax information obtained using parallax calculated in another common imaging area. [Figure 6] 10A and 10B are diagrams illustrating the accuracy of additional parallax information in the second common imaging region and the third common imaging region, respectively. [Figure 7] 10A and 10B are diagrams illustrating changes in shape when the same object is imaged with different cameras. [Figure 8] 3 shows a processing flow of the learning policy determination unit 200. [Figure 9] FIG. 10 is a block diagram showing an example of the configuration of a learning system according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In each drawing, parts having the same function or configuration are designated by the same reference numerals, and repeated description may be omitted.

[0016] Example 1 1 shows a configuration diagram of an image processing device 1 according to a first embodiment of the present invention. The image processing device 1 of this embodiment includes a data storage unit 100, a learning policy determination unit 200, and a disparity calculation model learning unit 300. The image processing device 1 is configured by a computer capable of acquiring data from a vehicle equipped with a multi-camera system.

[0017] FIG. 2 shows a hardware configuration diagram of an image processing device 1 according to a first embodiment. The image processing device 1 includes a processor 91, a storage device 92, an input device 93, an output device 94, and a communication interface (communication IF) 95. The processor 91, the storage device 92, the input device 93, the output device 94, and the communication interface (communication IF) 95 are connected via a bus 96. The processor 91 controls the image processing device 1. The storage device 92 serves as a work area for the processor 91. The storage device 92 is a volatile or non-volatile recording medium that stores various programs and data, and may be, for example, a read-only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), or a flash memory. The input device 93 inputs data. Examples of the input device 93 include a keyboard, a mouse, a touch panel, a numeric keypad, and a scanner. The output device 94 outputs data. Examples of the output device 94 include a display and a printer. The communication IF 95 connects to a network and transmits and receives data. The image processing device 1 can receive data acquired by a vehicle equipped with a multi-camera system by communicating with the vehicle via the communication IF 95. Note that the means by which the image processing device 1 acquires data from the vehicle is not limited to this method, and any method can be used.

[0018] A multi-camera system will be described using FIGS. 3(a) to 3(d). FIG. 3(a) shows an example of camera installation in a multi-camera system. In FIG. 3(a), a first camera 11 is mounted below the side mirror of a vehicle 10, and a second camera 12 and a third camera 13 are mounted on the front and rear of the roof of the vehicle 10. Each camera is mounted on the vehicle 10 so as to have an area that shares a field of view (hereinafter referred to as a common imaging area). The common imaging area of ​​the first camera 11 and the second camera 12 is a first common imaging area 21 (FIG. 3(b)), the common imaging area of ​​the first camera 11 and the third camera 13 is a second common imaging area 22 (FIG. 3(c)), and the common imaging area of ​​the second camera 12 and the third camera 13 is a third common imaging area 23 (FIG. 3(d)). In each common imaging area, parallax can be estimated by stereoscopic vision, and the distance to the imaged object can be calculated. As described above, this embodiment is implemented in a multi-camera system in which a plurality of cameras are installed on a vehicle so as to have a plurality of common imaging areas (overlapping imaging areas) in which at least a portion of the fields of view overlap.

[0019] The camera arrangement of the multi-camera system is not limited to those shown in Figures 3(a) to 3(d), and any combination can be selected as long as they are arranged to have a common imaging area. For example, a front camera with an imaging range in front of the vehicle can be installed, and parallax can be calculated using a camera pair that shares a field of view, or a rear camera with an imaging range behind the vehicle can be added. Cameras with different specifications, such as focal length, image size, and lens model, can be selected for each camera.

[0020] Furthermore, correct values ​​of distance data are required when training a disparity calculation model for each common imaging area. Distance data is acquired by a distance measurement sensor such as a LiDAR, so a distance measurement sensor is also installed on the vehicle 10. As shown in FIG. 3(a), a LiDAR 14 is often installed as a distance measurement sensor in the center of the roof of the vehicle 10, but any position can be used as long as it can measure the distances to the first to third common imaging areas, and multiple LiDARs may be installed to cover the blind spots of each LiDAR.

[0021] The data storage unit 100 (FIG. 1) stores data acquired by a vehicle equipped with the aforementioned multi-camera system. The data storage unit 100 stores images captured by each camera and sensor data from the distance measurement sensors, along with the sensor ID and a timestamp. If the vehicle's behavior can be acquired from an IMU (Inertial Measurement Unit) or vehicle CAN, the data storage unit 100 may also store the sensor data and the vehicle behavior. The data storage unit 100 also stores calibration data for each sensor (internal / external parameters for each camera, installation position of the distance measurement sensor, etc.). The calibration data may be values ​​at the time the sensors are installed, or values ​​corrected during driving to account for deterioration over time and temperature changes.

[0022] The learning policy determination unit 200 (FIG. 1) determines the order in which to learn the parallax calculation models based on the type and arrangement of each sensor. In the example of FIGS. 3(a) to 3(d), when there are three types of parallax calculation models, namely, a first parallax calculation model that infers the parallax of the first common imaging area 21, a second parallax calculation model that infers the parallax of the second common imaging area 22, and a third parallax calculation model that infers the parallax of the third common imaging area 23, the learning policy determination unit 200 determines the order in which to learn the parallax calculation models.

[0023] The training of the disparity calculation model will be explained using Figure 4. In training the disparity calculation model, images captured by two cameras and the correct values ​​of the disparity images are used. The captured images (right and left images) are input to the disparity calculation model, and the inference result of the disparity image is output. The correct values ​​of the prepared disparity images are compared with the inference result, and the parameters of the disparity calculation model are updated to output an inference result closer to the correct value (reducing the difference between the correct value of the disparity image and the inference result). The parameters of the disparity calculation model may be updated so that the inference result and the correct value are similar, or the parameters may be updated so that similar feature vectors are output for the same pixel in the left and right images calculated from the correct values. Alternatively, disparity calculation models with different numbers of parameters may be prepared, and a disparity calculation model with a smaller number of parameters may be trained so that it is similar to the final output or the output of the intermediate layer of a disparity calculation model with a larger number of parameters.

[0024] The correct values ​​for disparity images use distance information measured by a distance measurement sensor such as LiDAR. The correct values ​​for disparity images can be created by converting the LiDAR distance into disparity and plotting it on image coordinates. When creating correct values ​​from LiDAR distance measurements, it is difficult to plot correct disparity values ​​for all pixels. This is because the number of LiDAR measurement points is fewer than the camera resolution. As a result, the number of correct values ​​created by LiDAR varies depending on the region of the image. In regions with a small number of correct values, the learning of the disparity calculation model does not progress, which may result in a decrease in accuracy.

[0025] One way to improve the learning accuracy in areas with low LiDAR point density is to add disparity information obtained by methods other than LiDAR to the correct value (hereinafter referred to as additional disparity information) to support the learning of the disparity calculation model.

[0026] The additional parallax information will be described with reference to FIGS. 5(a) to 5(c).

[0027] Figure 5(a) shows the disparity obtained without using a ranging sensor, which corresponds to a rule-based disparity calculation method. Conventional disparity calculation methods, such as block matching, can calculate disparity from only the left and right images without requiring a correct answer value for learning. Therefore, the disparity calculated by the rule-based method can be used as the correct answer value to train a disparity calculation model. Alternatively, the inference result of a disparity calculation model previously trained on a different dataset can be used as the correct answer value.

[0028] In Figure 5(b), additional disparity information is created using a feature point matching technique. Feature point matching is an algorithm that searches for identical feature points (corners or pixels with large changes in brightness) between two images. By searching for identical pixels using feature point matching, disparity can be calculated from the difference in their coordinates. A representative feature point matching technique is SIFT (Scale Invariant Feature Transform). In recent years, methods that can match feature points at high density using deep learning have also been developed, so these may also be used.

[0029] In Figure 5(c), additional disparity information is created using the disparity calculated for another common imaging region. Because the first common imaging region 21 shown in Figure 3(a) and the third common imaging region 23 shown in Figure 3(c) overlap, the first disparity calculation model for the first common imaging region 21 can be trained using the disparity calculated for the third common imaging region 23. Figure 5(c) illustrates an example in which information from the third common imaging region 23 is used to train the first common imaging region 21, but similar processing is possible for other overlapping common imaging regions. From the third common imaging region 23, in addition to the additional disparity information calculated using the method in Figures 5(a) and (b) (hereinafter referred to as the third additional disparity information), the inference results of the third disparity calculation model trained using LiDAR and the third additional disparity information can be obtained. Recent disparity calculation models using deep learning can estimate high-density disparity across the entire image, enabling high-density disparity inference results to be obtained for the third common imaging region 23. The lack of points in the LiDAR can be compensated for by using this information (third additional parallax information, inference result of the third parallax calculation model) as additional parallax information (hereinafter, first additional parallax information) for the first common imaging area 21. Furthermore, in the first common imaging area 21, the additional parallax information calculated by the method of FIGS. 5(a) and 5(b) may also be included as the first additional parallax information.

[0030] In order to improve the accuracy of the parallax calculation model using the additional parallax information, it is necessary to calculate the additional parallax information with high accuracy. The accuracy of the additional parallax information will be explained using Figures 6(a) and 6(b). Figure 6(a) shows an example in the second common imaging area 22. The first camera 11 and the third camera 13 constituting the second common imaging area 22 have a long base line length (the distance between the cameras), so the same object is captured with a significantly different shape.

[0031] Using Figure 7, we will explain the shape change when the same object is imaged with different cameras. For an object on the road surface, in a stereo camera with camera height H and baseline length (distance between cameras) B, the point pr(x,y,1) on the right image corresponding to point pl(x,y,1) on the left image is calculated using equation (1). In this case, the degree of deformation increases according to the value of B / H, so the shape change is more severe for a camera pair with a longer baseline length. [Number 1] TIFF2025157800000002.tif1889

[0032] Furthermore, since the first camera 11 and the third camera 13 are installed at different camera heights, the surfaces that are imaged differ even for the same object. The first camera 11 is installed at a low position, so it can mainly image the side of the parallel running vehicle 30. On the other hand, the third camera 13 is installed on the roof of the vehicle itself, so it captures an image looking down on the parallel running vehicle 30. Therefore, both the side and top of the parallel running vehicle 30 are imaged.

[0033] 6(b), the second camera 12 and the third camera 13 constituting the third common imaging area 23 have short base lengths and are installed at the same height, so that the parallel running vehicle 30 can be imaged with almost the same shape. In addition, there are few occlusion areas, so the same surface of the object can be observed in both images.

[0034] Stereo distance measurement estimates disparity by searching for identical pixels between two images. The methods shown in Figures 5(a) and (b) often search for identical pixels based on the similarity of local regions of the images, but if the two images are captured differently (shape changes, imaging surface, etc.), they cannot identify identical pixels and cannot accurately calculate disparity. When there are significant shape changes and many occlusion areas, as in Figure 6(a), the accuracy of the additional disparity information decreases, potentially worsening the training accuracy of the disparity calculation model. On the other hand, in Figure 6(b), there are few shape changes and few occlusion areas, so additional disparity information can be calculated accurately. Therefore, adding this information to the training can improve the training accuracy of the disparity calculation model.

[0035] 6(a) and 6(b), by starting learning from the third common imaging region 23 in FIG. 6(b) and using the disparity information calculated at that time as the additional disparity information for the second common imaging region 22 in FIG. 6(a), the learning accuracy of the second parallax calculation model for the second common imaging region 22 can be improved.

[0036] The learning policy determination unit 200 (FIG. 1) includes a sensor placement priority determination unit 201, a sensor type priority determination unit 202, and a common imaging area order determination unit 203.

[0037] The sensor placement priority determination unit 201 estimates the difficulty of parallax calculation (inference) for each of the first common imaging area 21, the second common imaging area 22, and the third common imaging area 23 from the positional relationship of each sensor, and determines the learning priorities of the first parallax calculation model, the second parallax calculation model, and the third parallax calculation model. As described in FIG. 5(c), when the common imaging areas overlap, the parallax calculated in another common imaging area can be used for learning the parallax calculation model. Therefore, by starting learning of the parallax calculation model from a common imaging area with high accuracy of additional parallax information and using the additional parallax information created at that time for learning another parallax calculation model, the parallax calculation model can be learned with high accuracy.

[0038] As explained in Figure 7, the change in shape between two images becomes larger as the distance between the cameras increases. Therefore, by starting learning preferentially from camera pairs with a short distance between the cameras, which are considered to be less difficult to calculate disparity, the learning accuracy of subsequent disparity calculation models can be improved.

[0039] In addition, cameras located closer to the LiDAR that provides the correct value have less occlusion and are able to learn the correct distance, so the learning priority for the common imaging area of ​​camera pairs that are close to the LiDAR is set higher.

[0040] The sensor type priority determination unit 202 estimates the difficulty of parallax calculation (inference) from the sensor type (differences in specifications such as focal length, image size, lens model, etc.) (characteristics) and determines the learning priority. Since sensors of the same type have higher accuracy of additional parallax information, a higher learning priority is set.

[0041] The common imaging area order determination unit 203 determines the order in which the learning of the parallax calculation model is to be executed, based on the priorities output by the sensor placement priority determination unit 201 and the sensor type priority determination unit 202.

[0042] 8 shows the processing flow of the learning policy determination unit 200. The learning policy determination unit 200 acquires images captured by each camera and sensor data from the distance measurement sensor, which are stored in the data storage unit 100.

[0043] First, in step S101, a priority is calculated for all common imaging areas based on the difficulty of calculating parallax (sensor placement priority determination unit 201). The difficulty is an index indicating how accurately parallax can be calculated in the common imaging area, and uses the magnitude of the shape change between two images caused by the distance between the cameras, as explained in FIG. 7. In FIG. 7, the degree of deformation of an object on the road surface is calculated, but the degree of deformation on a plane other than the road surface can also be calculated. Therefore, when it is desired to accurately calculate the distance to a vehicle traveling parallel to the road, the degree of shape change on a plane perpendicular to the road surface may be used as an index of difficulty, or any other index may be used.

[0044] In step S102, the sensor placement priority determination unit 201 calculates a priority based on the positional relationship between the LiDAR and the camera (hereinafter referred to as the reference camera) that calculates the disparity. The LiDAR generates a correct value based on the acquired point cloud data. If the LiDAR and the reference camera are located far apart, the correct value may include a distance different from the object captured by the reference camera due to differences in the occlusion areas of the two sensors. For example, if a camera captures an image of a pedestrian, but a LiDAR installed in a different location can see the road surface behind the pedestrian, the disparity of the background road surface may be mistakenly plotted in the pixel where the pedestrian is captured. On the other hand, if the LiDAR and the reference camera are installed close to each other, the difference in the observable areas between the two sensors is small, allowing the disparity to be plotted correctly. Considering the above, the closer the distance between the LiDAR and the reference camera is, the higher the priority is set because the more accurately the disparity calculation model can be learned.

[0045] In step S102, the priority may be calculated based on the number of LiDAR points included in the common imaging area. If the positions, orientations, and angles of view of the camera and LiDAR are known, the number of LiDAR points that will appear within the imaging range can be predicted. Since the more LiDAR points there are, the higher the learning accuracy of the disparity calculation model. Therefore, the more LiDAR points there are in each common imaging area, the higher the priority may be set.

[0046] In step S103, a priority is calculated based on the sensor type (sensor type priority determination unit 202). The cameras that make up the multi-camera system may have different focal lengths, angles of view, image sizes, lens models, etc. For cameras of different types, if preprocessing required for parallax calculation, such as distortion correction and rectification transformation, is performed, the spatial resolution will differ even in the same area, making it difficult to calculate parallax. For this reason, the more similar the type of camera, the lower the difficulty of parallax calculation, and a higher priority is set.

[0047] In step S104, the priorities calculated in steps S101, S102, and S103 are integrated (common imaging area order determination unit 203). An average of the priorities may be used, or weighted priorities may be added together.

[0048] In step S105, the order is determined so that learning proceeds in order from the common imaging area with the highest priority calculated in step S104 (common imaging area order determination unit 203). As a result, the disparity information of the common imaging area for which additional disparity information can be calculated with high accuracy is used for learning of the subsequent common imaging area, thereby improving the learning accuracy of the subsequent disparity calculation model.

[0049] FIG. 8 shows an example in which the priority is calculated in steps S101, S102, and S103, but at least one of steps S101, S102, and S103 may be used, or the priority may be determined based on another index.

[0050] The disparity calculation model learning unit 300 learns a disparity calculation model based on the sensor data stored in the data storage unit 100 and the learning order output by the learning policy determination unit 200. The disparity calculation model learning unit 300 includes a disparity correct value creation unit 301, a learning and evaluation unit 302, and a model storage unit 303.

[0051] The disparity correct value creation unit 301 creates disparity images to be given as correct values ​​to the disparity calculation model in accordance with the learning order received from the learning policy determination unit 200. First, a disparity image is created based on the LiDAR data stored in the data storage unit 100. At this time, if the data storage unit 100 contains the vehicle behavior acquired from an IMU (Inertial Measurement Unit) or a vehicle CAN, it is possible to create a higher density of correct values ​​by overlaying the LiDAR data on the vehicle behavior in chronological order.

[0052] Next, additional parallax information is added to the created parallax image. If there is a common imaging area (hereinafter referred to as the fifth common imaging area) that precedes the common imaging area (hereinafter referred to as the fourth common imaging area) for which the correct value is created, and if the fourth common imaging area and the fifth common imaging area have an overlapping area, the parallax calculated in the fifth common imaging area is added to the correct value of the fourth common imaging area. In other words, the correct value of the fourth common imaging area includes the parallax calculated in the fifth common imaging area. If there are multiple fifth common imaging areas, the parallaxes of all the common imaging areas may be added to the correct value, or only the parallax that is earlier in the order (higher accuracy) may be added to the correct value.

[0053] Furthermore, in the fourth common imaging region, the disparity (disparity obtained by the rule-based model and feature point matching) is calculated using the method of FIGS. 5(a) and 5(b) and added to the correct answer value.

[0054] In the above example, the correct value is created by integrating the disparity of the LiDAR and the additional disparity information of the fourth common imaging region and the fifth common imaging region into the correct value. If multiple correct disparity values ​​are obtained for the same pixel, the average value of the multiple disparities is set as the correct value. Also, if LiDAR is obtained, the LiDAR value may be used as the correct value instead of the average value. Also, additional disparity information from an earlier order may be used as the correct value.

[0055] In addition, in a mass-produced vehicle in which a LiDAR is not installed, only the additional parallax information of the fourth common imaging region and the fifth common imaging region may be used as the correct value.

[0056] The learning and evaluation unit 302 uses the correct values ​​created by the correct disparity value creation unit 301 to learn a disparity calculation model and evaluate its accuracy. The correct values ​​created by the correct disparity value creation unit 301 are divided into data sets for learning and evaluation, and the disparity calculation model is learned using the learning data set as shown in FIG. 4. In addition, the estimation accuracy of the learned disparity calculation model is evaluated using the evaluation data set. If there is no problem with the accuracy, it is output to the model storage unit 303.

[0057] The model storage unit 303 stores the disparity calculation model output by the learning and evaluation unit 302. The stored disparity calculation model is subjected to processes such as pruning and quantization to increase the processing speed in the actual device, and then incorporated into the actual device.

[0058] <Example 2> A learning system 2 according to a second embodiment of the present invention will be described with reference to Fig. 9. Fig. 9 is a configuration diagram of the learning system 2 according to the second embodiment of the present invention, which is configured when cooperation (communication) between edge devices and servers and clouds is taken into consideration. Note that a description of configurations that are the same as or equivalent to those described in the first embodiment will be omitted, and the following description will mainly focus on differences.

[0059] The edge device 400 is an edge device including a mobile object such as a vehicle or a mobile robot, and an infrastructure sensor such as a surveillance camera, and is installed so that the fields of view of the multiple cameras overlap. The edge device 400 includes a sensor unit 401, a camera change detection unit 402, a camera anomaly detection unit 403, a model verification unit 404, a model inference unit 405, and an edge-side transmission / reception unit 406. Hereinafter, the edge device 400 will be described as an in-vehicle information processing device mounted on a vehicle equipped with a multi-camera system, but is not limited to an in-vehicle information processing device.

[0060] The sensor unit 401 is composed of multiple cameras, which are installed so as to share a field of view. Images captured by the cameras are transmitted to the server 500 at regular intervals for use in re-learning the disparity calculation model. A distance measurement sensor such as LiDAR may also be installed, and the distance measurement results are transmitted to the server 500 together with the camera images. If information that reveals the vehicle's driving status and driving environment can be obtained, it may also be transmitted to the server 500, and includes position information from an IMU (Inertial Measurement Unit), a vehicle CAN, and a GNSS (Global Navigation Satellite System), the operating status of the wipers and headlights, the time, illuminance, etc.

[0061] The data to be transmitted may be limited to only data that meets certain conditions. For example, only data on roads that have not been used to train the disparity calculation model may be transmitted based on GNSS position information, or only scenes with few training images may be transmitted based on the time, weather, etc.

[0062] The camera change detection unit 402 detects that at least one of the cameras constituting the multi-camera system has been changed and requests the server 500 to retrain the disparity calculation model. Cameras installed in mass-produced vehicles may be replaced with new cameras due to aging, malfunctions, or other reasons. Due to the availability of parts and improvements in the camera's performance, the specifications of the new camera, such as focal length, image size, and lens model, may differ from those of the previous camera. In this case, the images captured by the new camera differ from those captured by the previous camera. If the input to the disparity calculation model differs from that at the time of training, the accuracy of disparity estimation decreases, and therefore the disparity calculation model must be retrained. The camera change detection unit 402 stores the serial number of the camera at the time of production and outputs a retraining request to the server 500 when a camera with a different serial number is installed. Furthermore, if the camera specifications, such as focal length, image size, and lens model, are known, the server 500 may also output a retraining request when any of these specifications is changed. Furthermore, if the installation conditions such as the position and direction of the camera are known, a request for re-learning may be output to the server 500 at the timing when any of these changes is significant.

[0063] The camera anomaly detection unit 403 detects an abnormality in at least one of the cameras constituting the multi-camera system and outputs an abnormality flag to the server 500. In a multi-camera system, cameras may be installed outside the vehicle / outdoors, and lens dirt may change the color and contrast of the captured image. Images may also be captured incorrectly due to mud, dirt, raindrops, or other obstructions, or due to lens cracks caused by impact. When the color and contrast of the image change due to lens dirt, the parallax accuracy can be improved by re-learning the parallax calculation model. However, because stereo vision is disabled due to obstructions or lens cracks, re-learning the parallax calculation model does not improve performance. The camera anomaly detection unit 403 outputs (notifies) the camera's abnormal state (change in sensor state) to the server 500, which determines the timing for re-learning the parallax calculation model. If the type of abnormal state (such as dirt, adhesions, or cracks) can also be estimated, the type of abnormality is also transmitted to the server 500.

[0064] In addition, the camera abnormality detection unit 403 not only transmits the abnormal status of the camera to the server 500, but also displays the camera abnormality to the vehicle driver and on the infrastructure system management screen, etc., to alert the user to the camera abnormality and encourage them to take action to recover from the abnormality.

[0065] The model verification unit 404 verifies the operation of the re-trained model distributed from the server 500. By verifying in the background whether the re-trained model operates without problems in the real environment, it is possible to prevent system malfunctions caused by erroneous output from the disparity calculation model. Using video of the real environment as input, it is verified in the background whether the output is the same as that of the model before re-training, or whether there is any abnormal behavior when the system is controlled using the output of the re-trained model.

[0066] The model inference unit 405 is a block that operates the disparity calculation model in a real environment, inputs camera images and outputs disparity, and the multi-camera system operates based on the output disparity. In addition to using a trained model at the time of productization, it is also possible to receive a model whose operation has been verified for a certain period of time by the model verification unit 404, and update the model for use.

[0067] The edge-side transmitting and receiving unit 406 communicates with a server-side transmitting and receiving unit 501 (described later) and transmits images captured by the sensor unit 401 to the server 500. The edge-side transmitting and receiving unit 406 also receives a disparity calculation model from the server 500.

[0068] The server 500 receives information from the edge device 400, re-learns the disparity calculation model, and distributes it to the edge device 400. The server 500 includes a server-side transmitting / receiving unit 501, a data storage unit 502, a learning timing determination unit 503, a learning policy determination unit 504, and a disparity calculation model learning unit 505.

[0069] The server-side transmitting / receiving unit 501 (hereinafter also referred to as an image acquisition unit) communicates with the edge-side transmitting / receiving unit 406 of the edge device 400, and receives (acquires) images captured by the sensor unit 401. In addition, the server-side transmitting / receiving unit 501 transmits a disparity calculation model to the edge device 400.

[0070] The data storage unit 502 stores data transmitted from the edge device 400 and a disparity calculation model to be distributed to the edge device 400 .

[0071] The learning timing determination unit 503 refers to the data stored in the data storage unit 502 and determines the timing to re-learn the disparity calculation model.

[0072] The disparity calculation model is re-learned when data that has not been used to learn the disparity calculation model operating in the model inference unit 405 is found from the data on the driving conditions and driving environment output from the sensor unit 401. Data that is not included in the learning image is identified from GNSS position information, the operating status of the wipers and headlights, illuminance, and weather forecast.

[0073] Furthermore, if the camera installed in the edge device 400 is changed or an abnormality occurs, the disparity calculation model is retrained. There are multiple types of abnormal camera conditions, and if the color or contrast of the captured image changes due to lens dirt or the like, the performance of the disparity calculation model can be maintained even in scenes with dirty lenses by retraining the disparity calculation model using that image. On the other hand, in situations where stereo vision is difficult because the camera image cannot be captured normally due to dirt, mud, or other attachments, or a crack in the lens, the disparity calculation model is not retrained. Furthermore, if such an image is sent to the server 500, it is excluded from the disparity calculation model retraining targets.

[0074] The learning policy determination unit 504 and the disparity calculation model learning unit 505 have the same functions as the learning policy determination unit 200 and the disparity calculation model learning unit 300 described in the first embodiment.

[0075] The learning policy determination unit 504 may re-learn the disparity calculation model for all common imaging areas of the multi-camera system, or may re-learn only for common imaging areas that satisfy specific conditions, such as when a camera is updated (the sensor configuration has changed), an abnormality has occurred (the sensor state has changed), or a new driving scene has been acquired (additional data has been collected). When re-learning the disparity calculation model for all common imaging areas, the learning order is determined for each common imaging area described in the first embodiment, and learning of the disparity calculation model proceeds according to that order. On the other hand, when re-learning the disparity calculation model for some common imaging areas that satisfy the conditions, the learning order is determined for all common imaging areas that overlap with the common imaging area to be re-learned, and learning of the disparity calculation model proceeds according to that order.

[0076] In the disparity calculation model learning unit 505, since mass-produced vehicles are often not equipped with LiDAR for acquiring correct values, in such cases the disparity calculation model is re-learned using only the additional disparity information. The re-learned disparity calculation model is stored in the data storage unit 502. The re-learned disparity calculation model stored in the data storage unit 502 is delivered to the model verification unit 404 via the edge-side transceiver unit 406 of the edge device 400, and after its operation in the actual environment has been verified for a certain period of time, it is output to the model inference unit 405 and used in the multi-camera system of the vehicle in which the edge device 400 is installed.

[0077] <Modification> In the above-described embodiment, the camera configuration of the multi-camera system may be changed between collecting training data and operating the system. Using the camera configurations of FIGS. 3(a) to 3(d) as an example, the first camera 11 and the third camera 13 have a longer baseline, which reduces the accuracy of the additional parallax information. Therefore, a fourth camera (not shown) is additionally installed (added) between the first camera 11 and the third camera 13 only during data collection. After calculating additional parallax information between the first camera 11 and the fourth camera and between the third camera 13 and the fourth camera, all of the additional parallax information is used to train a parallax calculation model for the first camera 11 and the third camera 13. By installing the fourth camera in the middle, a camera pair with a short baseline is created, and the highly accurate parallax information calculated there can be used to improve the accuracy of the third parallax calculation model. The fourth camera is installed only during data collection and can be removed when the system is actually operating.

[0078] <Summary> As described above, the learning method of this embodiment is a learning method executed by a computer, and includes the steps of acquiring images captured by a plurality of cameras (11, 12, 13) having (a plurality of) common imaging areas (21, 22, 23) whose fields of view overlap at least partially, determining an execution order for learning of a disparity inference model that infers disparity using the images for each of the plurality of common imaging areas (21, 22, 23) (learning policy determination unit 200; Figure 8), and inputting images acquired from the plurality of cameras having the common imaging area to be learned into the disparity inference model according to the determined execution order, and learning the disparity inference model from the difference between the obtained disparity and ground truth value data (disparity calculation model learning unit 300; Figure 4), where the ground truth value data includes disparity information of a common imaging area assigned in an execution order earlier than the common imaging area to be learned.

[0079] As a result, a disparity calculation model is learned in order for a plurality of common imaging regions (21, 22, 23).

[0080] The method also includes a step of estimating the difficulty of parallax inference for each of a plurality of common imaging areas (21, 22, 23) based on at least one of the characteristics and arrangement of the plurality of cameras (11, 12, 13) (sensor arrangement priority determination unit 201 and sensor type priority determination unit 202 of the learning policy determination unit 200; steps S101 and S103 in Figure 8), and determining the order of execution of learning of the parallax inference model for the plurality of common imaging areas (21, 22, 23) based on the result of the estimation.

[0081] This allows the order of learning the disparity calculation models to be determined depending on the difficulty of disparity inference.

[0082] It also includes a step of generating the correct value data based on point cloud data acquired by a LiDAR placed on a vehicle equipped with the multiple cameras (11, 12, 13), and determining the order of execution of learning of the parallax inference model for multiple common imaging areas (21, 22, 23) based on the positional relationship between the multiple cameras and the LiDAR (sensor placement priority determination unit 201 of the learning policy determination unit 200; step S102 in Figure 8).

[0083] This determines the learning order of the disparity calculation model based on the positional relationship of the LiDAR.

[0084] The correct value data includes at least one of a rule-based model and a parallax calculated by feature point matching based on the acquired image (FIG. 5).

[0085] This allows the correct disparity value to be created using a method different from LiDAR.

[0086] In addition, the execution order of learning the parallax inference model is determined according to at least one of a change in the sensor configuration, a change in the sensor state, and the status of collection of additional data, and the parallax inference model is re-learned (Example 2).

[0087] This will enable model updates after commercialization.

[0088] In addition, based on the number (number of points) of LiDAR point cloud data included in each of the multiple common imaging areas (21, 22, 23), the order in which learning of the parallax inference model is executed for the multiple common imaging areas (21, 22, 23) is determined (step S102 in Figure 8).

[0089] This allows the learning order of the disparity calculation model to be determined, for example, from the number (number of points) of LiDAR point cloud data that can be predicted from the positional relationship of the LiDAR.

[0090] Furthermore, the learning system 2 according to this embodiment includes an in-vehicle information processing device (edge ​​device 400) mounted on a vehicle 10 equipped with multiple cameras (11, 12, 13) having common imaging areas (21, 22, 23) whose fields of view overlap at least partially, and a server device (server 500) that communicates with the in-vehicle information processing device (edge ​​device 400), and includes an image acquisition unit (server-side transceiver unit 501) that acquires images captured by the multiple cameras (11, 12, 13), a learning policy determination unit 504 that determines the execution order of learning the disparity calculation model for the multiple common imaging areas (21, 22, 23), and a disparity calculation model learning unit 505 that inputs images acquired from the multiple cameras having the common imaging area to be learned into the disparity calculation model according to the determined execution order, and learns the disparity calculation model from the difference between the obtained disparity and the correct value data, and the correct value data includes disparity information of the common imaging area that is assigned to an execution order earlier than the common imaging area to be learned.

[0091] As a result, a disparity calculation model is learned in order for a plurality of common imaging regions (21, 22, 23).

[0092] In addition, in the learning system 2 according to this embodiment, the learning policy determination unit 504 estimates the difficulty of inferring disparity for each of the multiple common imaging areas (21, 22, 23) based on at least one of the characteristics and arrangement of the multiple cameras (11, 12, 13), and determines the order of execution of learning of the disparity calculation model for the multiple common imaging areas (21, 22, 23) based on the estimation result.

[0093] This allows the order of learning the disparity calculation models to be determined depending on the difficulty of disparity inference.

[0094] In addition, the learning system 2 according to this embodiment further includes a disparity correct value creation unit 301 that generates correct value data based on point cloud data acquired by a LiDAR installed in the vehicle, and a learning policy determination unit 504 determines the execution order of learning the disparity calculation model for multiple common imaging areas (21, 22, 23) based on the positional relationship between the multiple cameras and the LiDAR.

[0095] This determines the learning order of the disparity calculation model based on the relative positions of the camera and LiDAR.

[0096] In addition, the learning system 2 according to this embodiment further includes a disparity correct value creation unit 301 that generates correct value data based on point cloud data acquired by a LiDAR arranged in a vehicle, and a learning policy determination unit 504 determines the execution order of learning the disparity calculation model for the multiple common imaging areas (21, 22, 23) based on the number (number of points) of LiDAR point cloud data included in each of the multiple common imaging areas (21, 22, 23).

[0097] This allows the learning order of the disparity calculation model to be determined, for example, from the number (number of points) of LiDAR point cloud data that can be predicted from the positional relationship of the LiDAR.

[0098] Furthermore, in the learning system 2 according to this embodiment, the correct answer data includes at least one of the disparity calculated by the rule-based model or feature point matching (FIG. 5).

[0099] This allows the correct disparity value to be created using a method different from LiDAR.

[0100] Furthermore, in the learning system 2 according to this embodiment, when the in-vehicle information processing device (edge ​​device 400) detects that at least one of the multiple cameras has been changed, it requests the server device (server 500) to re-learn the disparity calculation model for the common imaging area that includes the imaging area of ​​the changed camera (camera change detection unit 402), and the server device (server 500) generates a re-learned model by re-learning the disparity calculation model, and distributes the re-learned model to the in-vehicle information processing device (edge ​​device 400).

[0101] This allows the disparity calculation model to be re-learned when the camera is changed, making it possible to update the model after commercialization.

[0102] Furthermore, in the learning system 2 according to this embodiment, when the in-vehicle information processing device (edge ​​device 400) detects that an abnormality has occurred in at least one of the multiple cameras, it notifies the server device (server 500) of the abnormal state of the camera in which the abnormality has been detected (camera abnormality detection unit 403), and the server device (server 500) determines, based on the abnormal state, whether or not to re-learn the disparity calculation model for the common imaging area that includes the imaging area of ​​the camera in which the abnormality has been detected, re-learns the disparity calculation model for which it has determined that re-learning should be performed, thereby generating a re-learned model, and delivers the re-learned model to the in-vehicle information processing device (edge ​​device 400).

[0103] This allows the disparity calculation model to be appropriately re-learned depending on the abnormality that occurs in the camera, making it possible to update the model after commercialization.

[0104] Furthermore, in the learning system 2 according to this embodiment, an in-vehicle information processing device (edge ​​device 400) transmits a plurality of images captured by a plurality of cameras to a server device (server 500), and the server device (server 500) determines, based on the plurality of images, whether or not the plurality of images contain data that has not been used in learning at least one of the plurality of disparity calculation models. If the plurality of disparity calculation models contain data that has not been used in learning at least one of the plurality of disparity calculation models, the server device generates a re-learned model by re-learning the disparity calculation model, and distributes the re-learned model to the in-vehicle information processing device (edge ​​device 400).

[0105] This allows the trained disparity calculation model to properly re-train scenes that have not yet been trained, making it possible to update the model after commercialization.

[0106] According to this embodiment, the disparity calculation model is trained in an ordered manner for a plurality of common imaging areas, in other words, by determining the training order of the disparity calculation model, the accuracy of the correct disparity value used for training can be increased, and the training accuracy of the disparity calculation model can be improved.

[0107] In other words, according to this embodiment, in multiple common imaging areas in a multi-camera system, learning is started from an area where the difficulty of disparity calculation is low, and the additional disparity information calculated during the learning process is used to learn subsequent common imaging areas, thereby improving the learning accuracy of the disparity calculation model.

[0108] Although the embodiments of the present invention have been described above, the above embodiments merely show some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above embodiments.

[0109] It should be noted that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations.

[0110] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a storage device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0111] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0112] 1 Image processing device (Example 1) 100 Data storage unit 200 Learning Policy Decision Department 201 Sensor placement priority determination unit 202 Sensor type priority determination unit 203 Common imaging area order determination unit 300 Parallax calculation model learning unit 301 Parallax Correct Value Creation Unit 302 Learning and Assessment Department 303 Model Storage Unit 2 Learning System (Example 2) 400 Edge devices (in-vehicle information processing devices) 401 Sensor unit 402 Camera change detection unit 403 Camera abnormality detection unit 404 Model Validation Department 405 Model Inference Unit 406 Edge side transceiver 500 Servers (Server Equipment) 501 Server side transmission / reception unit (image acquisition unit) 502 Data Storage Unit 503 Learning Timing Decision Unit 504 Learning Policy Decision Department 505 Parallax calculation model learning unit

Claims

1. A learning system including an in-vehicle information processing device mounted on a vehicle equipped with a plurality of cameras having a common imaging area in which at least a portion of the field of view overlaps, and a server device that communicates with the in-vehicle information processing device, an image acquisition unit that acquires images captured by the plurality of cameras; a learning policy determination unit that determines an execution order of learning of a disparity calculation model for a plurality of common imaging regions; a parallax calculation model learning unit that inputs images acquired from a plurality of cameras having a common imaging area to be learned into a parallax calculation model in accordance with the determined execution order, and learns the parallax calculation model from a difference between the obtained parallax and ground truth value data; A learning system, wherein the correct answer data includes disparity information of a common imaging area assigned in an execution order earlier than a common imaging area to be learned.

2. The learning system according to claim 1, The learning policy determination unit estimating a degree of difficulty of inferring parallax for each of a plurality of common imaging regions based on at least one of characteristics and arrangements of the plurality of cameras; A learning system that determines an execution order of learning of a disparity calculation model for a plurality of common imaging regions based on a result of the estimation.

3. The learning system according to claim 1, a disparity correct value generation unit that generates the correct value data based on point cloud data acquired by a LiDAR arranged on the vehicle; The learning system, wherein the learning policy determination unit determines an execution order of learning of a disparity calculation model for a plurality of common imaging areas based on a positional relationship between the plurality of cameras and the LiDAR.

4. The learning system according to claim 1, a disparity correct value generation unit that generates the correct value data based on point cloud data acquired by a LiDAR arranged on the vehicle; The learning system, wherein the learning policy determination unit determines an execution order of learning of a disparity calculation model for a plurality of common imaging areas based on the number of LiDAR point cloud data included in each of the plurality of common imaging areas.

5. The learning system according to claim 1, A learning system, wherein the correct answer data includes at least one of a rule-based model and a disparity calculated by feature point matching.

6. The learning system according to claim 1, When the in-vehicle information processing device detects that at least one of the plurality of cameras has been changed, the in-vehicle information processing device requests the server device to re-learn a disparity calculation model for a common imaging area that includes an imaging area of ​​the changed camera; The server device: generating a re-trained model by re-training the disparity calculation model; A learning system that distributes the re-learned model to the in-vehicle information processing device.

7. The learning system according to claim 1, When the in-vehicle information processing device detects that an abnormality has occurred in at least one of the plurality of cameras, the in-vehicle information processing device notifies the server device of the abnormal state of the camera in which the abnormality has been detected; The server device: determining whether to re-learn a disparity calculation model for a common imaging area that includes an imaging area of ​​the camera in which the abnormality was detected, based on the abnormal state; generating a re-learned model by re-learning the disparity calculation model for which re-learning is determined to be performed; A learning system that distributes the re-learned model to the in-vehicle information processing device.

8. The learning system according to claim 1, the in-vehicle information processing device transmits a plurality of images captured by the plurality of cameras to the server device; The server device: determining whether or not data that has not been used in training at least one of the disparity calculation models is included based on the plurality of images; generating a re-trained model by re-training the disparity calculation model when data that has not been used in training at least one of the plurality of disparity calculation models is included; A learning system that distributes the re-learned model to the in-vehicle information processing device.

9. 1. A computer-implemented method of learning, comprising: acquiring images captured by a plurality of cameras having a common imaging area with at least a partial overlap of their fields of view; determining an execution order of learning a disparity inference model that infers disparity using the images for each of a plurality of common imaging regions; and inputting images acquired from a plurality of cameras having a common imaging area to be learned into a disparity inference model in accordance with the determined execution order, and learning the disparity inference model from the difference between the obtained disparity and the ground truth data; A learning method, wherein the correct answer data includes disparity information of a common imaging area that is assigned in an execution order earlier than a common imaging area to be learned.

10. 10. The learning method according to claim 9, estimating a degree of difficulty of inferring parallax for each of a plurality of common imaging regions based on at least one of characteristics and arrangements of the plurality of cameras; A learning method that determines an execution order of learning of a disparity inference model for a plurality of common imaging regions based on the result of the estimation.

11. 10. The learning method according to claim 9, generating the ground truth data based on point cloud data acquired by a LiDAR disposed on the vehicle having the plurality of cameras; A learning method that determines the order in which learning of a disparity inference model is executed for multiple common imaging areas based on the positional relationship between the multiple cameras and the LiDAR.

12. 10. The learning method according to claim 9, A learning method, wherein the correct value data includes at least one of a rule-based model and a disparity calculated by feature point matching based on the acquired image.

13. 10. The learning method according to claim 9, generating the ground truth data based on point cloud data acquired by a LiDAR disposed on the vehicle having the plurality of cameras; A learning method that determines the order in which training of a disparity inference model is performed for multiple common imaging areas based on the number of LiDAR point cloud data included in each of the multiple common imaging areas.

14. 10. The learning method according to claim 9, A learning method that determines an execution order of learning of a disparity inference model according to at least one of a change in sensor configuration, a change in sensor state, and a collection status of additional data, and re-learns the disparity inference model.

Images