Learning system and learning method

Abstract

Provided is a learning system and learning method capable of improving the accuracy of parallax correct values used for learning and improving the learning accuracy of a parallax calculation model. The system includes: an image acquisition unit (server-side transceiver 501) that acquires images captured by a plurality of cameras; a learning policy determination unit (504) that determines the learning execution order of a parallax calculation model for a plurality of common photographing areas; and a parallax calculation model learning unit (505) that inputs images acquired by the plurality of cameras for the common photographing areas having learning targets to the parallax calculation model in the determined execution order, and learns the parallax calculation model based on the difference between the obtained parallax and correct value data, wherein the correct value data contains parallax information for common photographing areas having execution orders before the common photographing areas of the learning targets.

Description

Technical Field

[0001] This invention relates to a learning system and a learning method, which are used in an image processing device that detects the distance to a target object in a common shooting area of ​​multiple shooting devices, and learns a disparity estimation model to estimate the disparity of the common shooting area. Background Technology

[0002] In recent years, multi-camera systems, which involve mounting multiple cameras on a moving object, have been developed in fields such as autonomous driving and driver assistance technologies. In multi-camera systems, distance measurement based on stereo vision can be achieved between cameras sharing a field of view, thereby enabling the estimation of accurate distances to objects.

[0003] In stereo vision-based ranging, the disparity is calculated as the difference in the positions of the same object captured by two cameras (right and left cameras) sharing a field of view. Since the disparity varies with depth, the distance to the detected object can be calculated from the obtained disparity.

[0004] Traditionally, disparity is calculated by rule-based processing, which computes similarity in local regions between images captured by left and right cameras (left and right images). In recent years, with the development of deep learning, represented by CNNs (Convolutional Neural Networks), learning-based methods are being developed. In these learning methods, multiple captured images and distance data as correct values ​​are used to learn a disparity calculation model (also called a disparity inference model) (see, for example, Patent Document 1).

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2020-80047 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] As described in Patent Document 1, "the true depth can be obtained, for example, by a 3D sensor such as LiDAR (Laser Imaging Detection and Ranging)." In traditional learning methods, distance data as the correct value is generated using ranging sensors such as LiDAR. However, in general, LiDAR has a low point density, and the learning accuracy decreases in areas with low point density.

[0010] As a method to compensate for LiDAR point density, additional disparity information derived from methods other than LiDAR can be considered. Examples of additional disparity information include disparity calculated using rule-based methods, disparity calculated from feature point matching results, and other disparities obtained through methods other than LiDAR. Furthermore, disparity calculated in other shared shooting areas (also known as overlapping shooting areas) can also be used (rule-based disparity calculation, feature point matching, disparity calculation models, etc.).

[0011] However, in multi-camera systems, there are camera pairs with long baselines, different viewing angles, and non-parallel configurations. In such camera pairs, the shapes of the left and right images change drastically, making it impossible to accurately calculate the additional disparity information. Using incorrect additional disparity information to learn the disparity calculation model may degrade its performance.

[0012] The present invention was made in view of the above circumstances, and its purpose is to provide a learning system and learning method that can improve the accuracy of the disparity correct values ​​used in learning and improve the learning accuracy of the disparity calculation model.

[0013] Methods for solving problems

[0014] To address the aforementioned issues, the present invention relates to a learning system comprising an in-vehicle information processing device and a server device. The in-vehicle information processing device is mounted on a vehicle having multiple cameras with shared shooting areas having at least partially overlapping fields of view. The server device communicates with the in-vehicle information processing device. The learning system includes: an image acquisition unit that acquires images captured by the multiple cameras; a learning policy determination unit that determines the learning execution order of a disparity calculation model for the multiple shared shooting areas; and a disparity calculation model learning unit that, according to the determined execution order, inputs images acquired by the multiple cameras having shared shooting areas of the learning object into the disparity calculation model, and learns the disparity calculation model based on the difference between the obtained disparity and correct value data, wherein the correct value data includes disparity information of shared shooting areas assigned to the shared shooting areas of the learning object in the execution order prior to the shared shooting areas of the learning object.

[0015] Furthermore, the learning method involved in this invention is a computer-executed learning method, comprising: acquiring images captured by multiple cameras having at least partially overlapping fields of view in a common shooting area; determining, for each of the multiple common shooting areas, a disparity inference model learning execution order for using the images to infer disparity; and inputting, according to the determined execution order, the images acquired by the multiple cameras having the common shooting area of ​​the learning object into the disparity inference model, and learning the disparity inference model based on the difference between the obtained disparity and correct value data, wherein the correct value data includes disparity information of the common shooting areas assigned to the common shooting area of ​​the learning object in the execution order prior to the common shooting area of ​​the learning object.

[0016] The effects of the invention

[0017] According to the present invention, by learning the disparity calculation model sequentially from multiple common shooting areas, in other words, by determining the learning order of the disparity calculation model, the accuracy of the correct disparity values ​​used for learning can be improved, and the learning accuracy of the disparity calculation model can be enhanced.

[0018] Other than those described above, the issues, structures, and effects will become clear through the following description of the implementation methods. Attached Figure Description

[0019] [ Figure 1 This diagram shows a block diagram illustrating an example configuration of an image processing apparatus according to Embodiment 1 of the present invention.

[0020] [ Figure 2 [Hardware configuration diagram of the image processing device involved in Embodiment 1 of the present invention.]

[0021] [ Figure 3 The diagrams provided illustrate a multi-camera system. (a) is a side view of an example of a camera and LiDAR setup. (b) is a top view of the first common shooting area. (c) is a top view of the second common shooting area. (d) is a top view of the third common shooting area.

[0022] [ Figure 4 [A diagram illustrating the learning of the disparity calculation model.]

[0023] [ Figure 5 The diagrams provided illustrate additional parallax information: (a) additional parallax information obtained without using a ranging sensor (such as rule-based parallax); (b) additional parallax information obtained through feature point matching; and (c) an illustration of additional parallax information obtained using parallax calculated from other common shooting areas.

[0024] [ Figure 6 The diagrams provided illustrate the accuracy of the additional parallax information, with (a) representing the second common shooting area and (b) representing the accuracy of the additional parallax information in the third common shooting area.

[0025] [ Figure 7 This is a diagram illustrating how the shape of the same object changes when photographed by different cameras.

[0026] [ Figure 8 The learning policy decision-making department's processing flow 200.

[0027] [ Figure 9 This diagram illustrates a block diagram of a learning system configuration example according to Embodiment 2 of the present invention. Detailed Implementation

[0028] Hereinafter, embodiments of the present invention will be described using the accompanying drawings. Furthermore, in the drawings, parts having the same function or structure are labeled with the same symbols, and sometimes repeated descriptions are omitted.

[0029] <Example 1>

[0030] Figure 1 This diagram illustrates the configuration of an image processing apparatus 1 according to Embodiment 1 of the present invention. The image processing apparatus 1 of this embodiment 1 includes a data storage unit 100, a learning policy determination unit 200, and a disparity calculation model learning unit 300. The image processing apparatus 1 is configured as a computer capable of acquiring data from a vehicle equipped with a multi-camera system.

[0031] Figure 2 This diagram illustrates the hardware configuration of the image processing apparatus 1 according to Embodiment 1. The image processing apparatus 1 includes a processor 91, a storage device 92, an input device 93, an output device 94, and a communication interface (communication IF) 95. The processor 91, storage device 92, input device 93, output device 94, and communication interface (communication IF) 95 are connected via a bus 96. The processor 91 controls the image processing apparatus 1. The storage device 92 serves as the working area of ​​the processor 91. Furthermore, the storage device 92 is a volatile or non-volatile recording medium for storing various programs and data, such as ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), or flash memory. The input device 93 inputs data. Examples of input devices 93 include a keyboard, mouse, touchscreen, numeric keypad, and scanner. The output device 94 outputs data. Examples of output devices 94 include a display and a printer. The communication IF 95 connects to a network and transmits and receives data. The image processing device 1 communicates with a vehicle equipped with a multi-camera system via a communication IF95, thereby enabling it to receive data acquired by the vehicle. Furthermore, the method by which the image processing device 1 acquires data from the vehicle is not limited to this method; any method can be used.

[0032] use Figure 3 Sections (a) through (d) describe the multi-camera system. Figure 3 (a) shows an example of camera setup in a multi-camera system. Figure 3 In (a), the first camera 11 is positioned below the rearview mirror of the vehicle 10, and the second camera 12 and the third camera 13 are mounted on the front and rear of the roof of the vehicle 10. Each camera is positioned to share a common field of view area on the vehicle 10 (hereinafter referred to as the common shooting area), and the common shooting area of ​​the first camera 11 and the second camera 12 is the first common shooting area 21. Figure 3(b) The common shooting area of ​​the first camera 11 and the third camera 13 is the second common shooting area 22. Figure 3 (c) The common shooting area of ​​the second camera 12 and the third camera 13 is the third common shooting area 23. Figure 3 (d)). In each common shooting area, parallax can be estimated by stereo vision, and the distance to the subject can be calculated. This embodiment is implemented in a multi-camera system in which multiple cameras are installed on a vehicle in such a manner as described above, having multiple common shooting areas (overlapping shooting areas) with at least partial overlap in field of view.

[0033] The camera configuration of a multi-camera system is not limited to Figure 3 For (a) to (d), any combination can be chosen as long as they are configured in a way that shares a common shooting area. For example, a front-facing camera with the front of the vehicle as its shooting range can be set up, and the parallax can be calculated by a pair of cameras sharing the field of view. A rear-facing camera with the rear of the vehicle as its shooting range can also be added. Each camera can also be selected with different specifications such as focal distance, image size, and lens model.

[0034] Furthermore, accurate distance data is required when learning the parallax calculation model in each shared shooting area. Since the distance data is acquired by ranging sensors such as LiDAR, these sensors are also installed on vehicle 10. Figure 3 As shown in (a), the LiDAR14, which serves as a ranging sensor, is mostly located in the center of the roof of the vehicle 10. However, as long as it can measure the distance of the first to third common shooting areas, it can be located at any position. Multiple LiDARs can also be set up to cover the blind spots of each LiDAR.

[0035] Data storage unit 100 ( Figure 1 The data storage unit 100 stores the data acquired by the vehicle equipped with the multi-camera system. In the data storage unit 100, the images captured by each camera and the sensor data from the ranging sensor are stored together with the sensor ID and timestamp. Furthermore, if the vehicle's own behavior can be acquired from the IMU (Inertial Measurement Unit) and the vehicle's CAN bus, the data storage unit 100 can also store the sensor data and this vehicle behavior together. In addition, calibration data for each sensor (intrinsic / extrinsic parameters of each camera, installation position of the ranging sensor, etc.) is also stored. The calibration data can store the values ​​at the time of sensor installation, or, to cope with years of degradation and temperature changes, the values ​​calibrated during driving.

[0036] Learning Policy Decision-Making Department 200 ( Figure 1 The order in which the parallax calculation model is learned depends on the type and configuration of each sensor. Figure 3In examples (a) to (d), when there are three parallax calculation models—the first parallax calculation model for calculating the parallax of the first common shooting area 21, the second parallax calculation model for calculating the parallax of the second common shooting area 22, and the third parallax calculation model for calculating the parallax of the third common shooting area 23—the policy is to determine the order in which the parallax calculation models are learned.

[0037] use Figure 4 The learning process for the disparity calculation model is explained. In the learning of the disparity calculation model, images captured by two cameras and the correct values ​​of the disparity image are used. The captured images (right and left images) are input into the disparity calculation model, and the output is the inferred result of the disparity image. The prepared correct values ​​of the disparity image are compared with the inferred result, and the parameters of the disparity calculation model are updated by outputting an inferred result that is closer to the correct value (the difference between the correct value and the inferred result of the disparity image becomes smaller). Alternatively, the parameters of the disparity calculation model can be updated by making the inferred result similar to the correct value, or by outputting similar feature vectors in the same pixel of the left and right images calculated from the correct value. Furthermore, disparity calculation models with different numbers of parameters can be prepared, and the model with fewer parameters can be learned in a way that is similar to the final output or intermediate layer output of the disparity calculation model with more parameters.

[0038] The correct values ​​for a disparity image are derived using distance information measured by ranging sensors such as LiDAR. These correct values ​​can be created by converting the distance from the LiDAR to disparity and plotting it onto image coordinates. However, when creating correct values ​​from LiDAR ranging results, it is difficult to plot the correct disparity values ​​across all pixels. This is because the number of ranging points for LiDAR is less than the resolution of a camera. Therefore, the number of correct values ​​generated by LiDAR will be biased across different regions of the image. In areas with a low number of correct values, the disparity calculation model struggles to learn, and accuracy may decrease.

[0039] As a method to improve the learning accuracy in areas with low LiDAR point density, it is possible to add disparity information obtained through methods other than LiDAR (hereinafter referred to as supplementary disparity information) to the correct values ​​to assist the learning of the disparity calculation model.

[0040] use Figure 5 Sections (a) to (c) describe the additional parallax information.

[0041] Figure 5(a) shows disparity acquired without using a ranging sensor, corresponding to a rule-based disparity calculation method. In disparity calculation methods, such as traditional block matching, disparity can be calculated from the left and right images without needing to learn correct values. Therefore, the disparity calculated by the rule-based method can be used as the correct value to learn the disparity calculation model. Alternatively, the inference results of the disparity calculation model pre-learned on different datasets can also be used as the correct value.

[0042] Figure 5 (b) uses feature point matching to create additional disparity information. Feature point matching is an algorithm that searches for the same feature point (corner point or pixel with drastic brightness changes) between two images. By searching for the same pixel through feature point matching, disparity can be calculated from its coordinate difference. A representative method of feature point matching is SIFT (Scale Invariant Feature Transform). In addition, in recent years, methods for high-density feature point matching through deep learning have also been developed, and these methods can also be used.

[0043] Figure 5 In (c), additional parallax information is created by using the parallax calculated in other common shooting areas. Figure 3 The first common shooting area 21 shown in (a) and Figure 3 The third common shooting region 23 shown in (c) overlaps with the first common shooting region 21, so the disparity calculated in the third common shooting region 23 can be used to learn the first disparity calculation model for the first common shooting region 21. Figure 5 Example (c) describes using information from the third common shooting region 23 for learning the first common shooting region 21; however, otherwise, the same processing can be performed whenever the common shooting regions overlap. From the third common shooting region 23, except... Figure 5 In addition to the additional disparity information calculated by methods (a) and (b) (hereinafter referred to as the third additional disparity information), the calculation results of the third disparity calculation model learned from LiDAR and the third additional disparity information can also be obtained. In recent years, disparity calculation models using deep learning have been able to calculate high-density disparity across the entire image, thus obtaining high-density disparity calculation results in the third common imaging region 23. By using this information (the third additional disparity information and the calculation results of the third disparity calculation model) as additional disparity information (hereinafter referred to as the first additional disparity information) in the first common imaging region 21, the point deficiency of LiDAR can be compensated. Furthermore, in the first common imaging region 21, the following can also be used: Figure 5 The additional disparity information calculated by methods (a) and (b) is also included in the first additional disparity information.

[0044] To improve the accuracy of the disparity calculation model using additional disparity information, the additional disparity information needs to be calculated with high precision. Figure 6 (a) and (b) explain the accuracy of the additional parallax information. Figure 6 Example 22 is shown in (a). Due to the long baseline length (distance between cameras), the first camera 11 and the third camera 13 constituting the second common shooting area 22 can also capture the same object with a large change in shape.

[0045] use Figure 7 The shape changes of the same object when photographed by different cameras are explained. For an object on the road surface, in a stereo camera with camera height H and baseline length (distance between cameras) B, the point pr(x,y,1) on the right image corresponding to the point pl(x,y,1) in the left image is calculated by equation (1). At this time, the degree of deformation increases according to the value of B / H, so the camera with a longer baseline length shows more drastic shape changes.

[0046] [Number 1]

[0047]

[0048] Furthermore, because the first camera 11 and the third camera 13 are positioned at different heights, the same object is captured from different angles. The first camera 11, being positioned lower, primarily captures the side view of the parallel vehicle 30. On the other hand, the third camera 13, positioned on the roof of the vehicle, captures an image of the parallel vehicle 30 from above. Therefore, both the side and top views of the parallel vehicle 30 are captured.

[0049] On the other hand, Figure 6 In the third common shooting area 23 shown in (b), the baseline lengths of the second camera 12 and the third camera 13 constituting the third common shooting area 23 are also short, and the cameras are also set at the same height, so they can capture the parallel vehicles 30 in almost the same shape. In addition, there is less occlusion, and the same side of the object can be observed in the two images.

[0050] Stereo vision-based ranging calculates disparity by searching for the same pixel between two images. Figure 5 Methods (a) and (b) primarily rely on the similarity of local image regions to search for the same pixel. However, when two images are captured differently (shape changes, shooting plane, etc.), they cannot identify the same pixel and cannot correctly calculate the parallax. For example... Figure 6 As in (a), when there are drastic shape changes and many occluded areas, the accuracy of the added disparity information decreases, which may lead to a deterioration in the learning accuracy of the disparity calculation model. On the other hand, in Figure 6In (b), there are fewer shape changes and occlusion areas, which can calculate the additional disparity information with high accuracy. Therefore, by adding it to the learning process, the learning accuracy of the disparity calculation model can be improved.

[0051] By using the disparity information of a shared shooting region capable of calculating additional disparity information with high accuracy for the learning of different shared shooting regions—that is, by using the disparity information of a shared shooting region capable of calculating additional disparity information with high accuracy for the learning of the disparity calculation model of a shared shooting region assigned to a subsequent execution order—the learning accuracy of other disparity calculation models can be improved. Figure 6 In examples (a) and (b), from Figure 6 Learning begins in the third common shooting area 23 of (b), and the parallax information calculated at this time is used as... Figure 6 Additional disparity information of the second common shooting region 22 in (a) can improve the learning accuracy of the second disparity calculation model for the second common shooting region 22.

[0052] Learning Policy Decision-Making Department 200 ( Figure 1 It includes a sensor configuration priority determination unit 201, a sensor type priority determination unit 202, and a common shooting area order determination unit 203.

[0053] The sensor configuration priority determination unit 201 estimates the ease of disparity calculation (estimation) for the first common imaging area 21, the second common imaging area 22, and the third common imaging area 23 based on the positional relationship of each sensor, and determines the learning priority of the first disparity calculation model, the second disparity calculation model, and the third disparity calculation model. For example, using... Figure 5 As explained in (c), when co-shooting areas overlap, the disparity calculated in other co-shooting areas can be used for learning the disparity calculation model. Therefore, by starting the learning of the disparity calculation model from co-shooting areas with high accuracy in adding disparity information, and using the additional disparity information created at this time for learning other disparity calculation models, the disparity calculation model can be learned with high accuracy.

[0054] If used Figure 7 As explained, the shape change between two images increases with the distance between the cameras. Therefore, we should start learning from the camera pair with the shorter distance between them, as the disparity calculation is easier and this can improve the learning accuracy of the subsequent disparity calculation model.

[0055] Furthermore, LiDAR cameras that are close to the LiDAR provide the correct values ​​have less obstruction and are able to learn the correct distances. Therefore, the learning priority of the common shooting area of ​​camera pairs that are close to the LiDAR is set high.

[0056] The sensor type priority determination unit 202 estimates the ease of parallax calculation (estimation) based on the sensor type (differences in specifications such as focal distance, image size, and lens model) and determines the learning priority. Additional parallax information from the same type of sensor has high accuracy, therefore the learning priority is set high.

[0057] The joint shooting area order determination unit 203 determines the order in which the parallax calculation model learning is performed based on the priority output by the sensor configuration priority determination unit 201 and the sensor type priority determination unit 202.

[0058] use Figure 8 The processing flow of the learning policy decision unit 200 is shown. The learning policy decision unit 200 acquires images captured by each camera and sensor data from the ranging sensor stored in the data storage unit 100.

[0059] First, in step S101, a priority based on the ease of parallax calculation is calculated for all shared shooting areas (sensor configuration priority determination unit 201). Ease of calculation is an index representing how accurately parallax can be calculated within a shared shooting area. Figure 7 The text describes the magnitude of shape changes between two images caused by the distance between cameras. Figure 7 The algorithm can calculate the degree of deformation of objects on the road surface, but it can also calculate the degree of deformation on planes outside the road surface. Therefore, when a high-precision distance to parallel vehicles is desired, the degree of shape change relative to the vertical plane of the road surface can be used as a difficulty index, or any index can be used.

[0060] In step S102, the priority of the positional relationship between the LiDAR and the camera (hereinafter referred to as the reference camera) from which the disparity is to be calculated is calculated (sensor configuration priority determination unit 201). The LiDAR generates correct values ​​based on the acquired point group data. When the LiDAR and the reference camera are separated, due to the difference in the occlusion areas of the two sensors, the correct values ​​may include distances different from those of the objects captured by the reference camera. For example, the camera may capture a pedestrian, but the LiDAR, which is positioned differently, may see a certain road surface behind the pedestrian, potentially incorrectly plotting the disparity of the background road surface onto the pixels where the pedestrian is captured. On the other hand, when the LiDAR and the reference camera are close together, the observable area has a smaller difference between the two sensors, thus allowing for correct disparity plotting. Considering the above, the closer the LiDAR and the reference camera are, the more accurately the disparity calculation model can be learned, therefore the priority is set high.

[0061] Furthermore, in step S102, priority can also be calculated based on the number of LiDAR points contained in the shared shooting area. If the position, orientation, and angle of view of the camera and LiDAR are known, the number of LiDAR points appearing within the shooting range can be predicted. The more LiDAR points there are, the higher the learning accuracy of the parallax calculation model; therefore, the higher the priority can be set for each shared shooting area with more LiDAR points.

[0062] In step S103, a priority is calculated based on the sensor type (sensor type priority determination unit 202). Cameras constituting a multi-camera system may have different focal distances, angles of view, image sizes, lens models, etc. After preprocessing such as distortion correction and parallelization conversion for parallax calculation, the spatial resolution of the same area differs among different types of cameras, making parallax calculation difficult. Therefore, the difficulty of parallax calculation for the same type of camera is set to low, and a high priority is assigned.

[0063] In step S104, the priorities calculated in steps S101, S102, and S103 are combined (common shooting area order determination unit 203). The average of each priority can be used, or the priorities can be weighted and then added together.

[0064] In step S105, the learning order is determined so that learning proceeds sequentially, starting with the co-shooting regions with high priority calculated in step S104 (co-shooting region order determination unit 203). Therefore, by using the disparity information of co-shooting regions from which additional disparity information can be calculated with high accuracy for the learning of subsequent co-shooting regions, the learning accuracy of the subsequent disparity calculation model can be improved.

[0065] exist Figure 8 The example shown is an example of calculating the priority in steps S101, S102, and S103, but at least one of steps S101, S102, and S103 can also be used, or the priority can be determined based on other indicators.

[0066] The disparity calculation model learning unit 300 learns the disparity calculation model based on the sensor data stored in the data storage unit 100 and the learning sequence output by the learning policy decision unit 200. The disparity calculation model learning unit 300 includes a disparity accuracy value generation unit 301, a learning and evaluation unit 302, and a model storage unit 303.

[0067] The disparity accuracy value generation unit 301 generates a disparity image as the accuracy value assigned to the disparity calculation model according to the learning sequence received from the learning policy determination unit 200. First, the disparity image is generated based on the LiDAR data stored in the data storage unit 100. At this time, if the data storage unit 100 contains the vehicle's behavior acquired from the IMU (Inertial Measurement Unit) and the vehicle's CAN bus, a higher density accuracy value can be generated by overlaying the vehicle's behavior with the LiDAR data in a time series.

[0068] Next, additional parallax information is added to the generated parallax image. If there exists a common shooting region (hereinafter referred to as the 5th common shooting region) that precedes the correct value (hereinafter referred to as the 4th common shooting region), and the 4th and 5th common shooting regions overlap, then the parallax calculated by the 5th common shooting region is added to the correct value of the 4th common shooting region. In other words, the correct value of the 4th common shooting region includes the parallax calculated by the 5th common shooting region. If there are multiple 5th common shooting regions, the parallax of all common shooting regions can be added to the correct value, or only the parallax of the earlier (higher accuracy) regions can be added to the correct value.

[0069] In addition, in the fourth joint shooting area, through Figure 5 The disparity (obtained by rule-based model and feature point matching) is calculated using methods (a) and (b) and then appended to the correct value.

[0070] In the example above, the parallax of LiDAR and the additional parallax information from the 4th and 5th shared imaging areas is combined to create the correct value. When multiple correct parallax values ​​are obtained for the same pixel, the average of these multiple parallax values ​​is set as the correct value. Alternatively, if LiDAR is available, the LiDAR value can be used as the correct value instead of the average. Furthermore, additional parallax information from earlier sequences can also be used as the correct value.

[0071] In addition, in mass-produced vehicles without LiDAR, the additional parallax information from the 4th and 5th common imaging areas can be used as the correct value.

[0072] The learning and evaluation unit 302 uses the correct values ​​produced by the disparity accuracy production unit 301 to learn the disparity calculation model and evaluate its accuracy. The correct values ​​produced by the disparity accuracy production unit 301 are divided into learning and evaluation datasets. The learning dataset is then used according to... Figure 4 A disparity calculation model is learned. Furthermore, the estimation accuracy is evaluated using an evaluation dataset for the learned disparity calculation model. If the accuracy is satisfactory, the model is output to the model storage unit 303.

[0073] The model storage unit 303 stores the disparity calculation model output by the learning and evaluation unit 302. After the stored disparity calculation model undergoes pruning, quantization, and other processes to improve the processing speed of the actual machine, it is embedded into the actual machine.

[0074] <Example 2>

[0075] use Figure 9 The learning system 2 involved in Embodiment 2 of the present invention will be described. Figure 9 This is a configuration diagram of the learning system 2 according to Embodiment 2 of the present invention, which takes into account the configuration when the edge device collaborates (communicates) with the server and the cloud. In addition, the description of the configuration that is the same as or equivalent to the configuration described in Embodiment 1 is omitted, and the differences are mainly described below.

[0076] Edge device 400 is an edge device mounted on a mobile body such as a vehicle or mobile robot, and includes infrastructure sensors such as surveillance cameras, arranged in a manner where the fields of view of multiple cameras overlap. Edge device 400 includes a sensor unit 401, a camera change detection unit 402, a camera anomaly detection unit 403, a model verification unit 404, a model calculation unit 405, and an edge transceiver unit 406. Hereinafter, edge device 400 will be described as an in-vehicle information processing device mounted on a vehicle equipped with a multi-camera system, but is not limited to an in-vehicle information processing device.

[0077] The sensor unit 401 consists of multiple cameras arranged in a shared field of view. Images captured by the cameras are used for relearning the parallax calculation model and sent to the server 500 at regular intervals. Additionally, ranging sensors such as LiDAR can also be installed, and the ranging results are sent to the server 500 along with the camera images. Furthermore, if information indicating the vehicle's driving status or driving environment can be obtained, it can also be sent to the server 500, including IMU (Inertial Measurement Unit), vehicle CAN, GNSS (Global Navigation Satellite System) location information, and the operating status, time, and illuminance of the windshield wipers and headlights.

[0078] The data transmitted can also be limited to data that meets specific conditions. For example, based on GNSS location information, only unused driving road data learned by the parallax calculation model can be transmitted, or only scenes with few learning images can be transmitted based on time, weather, or other limitations.

[0079] The camera change detection unit 402 detects that at least one camera in a multi-camera system has been changed and requests the server 500 to relearn the disparity calculation model. Cameras installed on mass-produced vehicles may be replaced with new ones due to deterioration or malfunction over time. The specifications of the new camera may differ from the original camera due to component procurement conditions or the camera's own performance improvements, such as focal distance, image size, and lens model. In this case, the image captured by the new camera will differ from the image captured by the original camera. If the input to the disparity calculation model differs from the input during learning, the accuracy of the disparity estimation will decrease, thus requiring relearning of the disparity calculation model. The camera change detection unit 402 stores the serial number of the camera at the time of productization and sends a relearning request to the server 500 when a camera with a different serial number is installed. Furthermore, if the camera specifications such as focal distance, image size, and lens model are known, a relearning request can also be sent to the server 500 at any time of specification change. Additionally, if the camera's position, orientation, and other setting conditions are known, a relearning request can also be sent to the server 500 at any time of significant change.

[0080] The camera anomaly detection unit 403 detects an anomaly in at least one camera in a multi-camera system and outputs an anomaly flag to the server 500. In a multi-camera system, cameras are sometimes located outside vehicles / buildings, and lens dirt may cause changes in the color or contrast of the captured image. Furthermore, images may not be acquired properly due to obstructions such as mud or dirt, raindrops, or lens cracks caused by impacts. While relearning the disparity calculation model can improve disparity accuracy in cases where lens dirt causes color or contrast changes, performance cannot be expected to improve in cases where stereo vision is impossible due to obstructions or lens cracks. The camera anomaly detection unit 403 outputs (notifies) the camera's anomaly status (change in sensor status) to the server 500, which then determines when to relearn the disparity calculation model. If the type of anomaly (dirt, attachments, cracks, etc.) can also be estimated, the anomaly type is also sent to the server 500.

[0081] In addition, the camera anomaly detection unit 403 not only sends the abnormal status of the camera to the server 500, but also displays the camera anomaly to the vehicle driver, the management screen of the infrastructure system, etc., prompting the user to pay attention to the camera anomaly or take abnormal recovery actions.

[0082] The model validation unit 404 verifies the operation of the relearned model distributed from server 500. By verifying in the background whether the relearned model operates without problems in the real environment, malfunctions caused by incorrect output from the disparity calculation model can be suppressed. Using real-world imagery as input, the unit verifies in the background whether the output is the same as the model before relearning, or whether there is any abnormal behavior when using the output of the relearned model to control the system.

[0083] The model calculation unit 405 is a module that enables the parallax calculation model to operate in a real-world environment. It takes camera images as input and outputs parallax, and enables the multi-camera system to operate based on the output parallax. In addition to using the learned model from the product stage, it can also receive a model whose operation has been verified for a certain period of time in the model verification unit 404, and use it after updating the model.

[0084] The edge transceiver unit 406 communicates with the server transceiver unit 501 (described later) and sends the images captured by the sensor unit 401 to the server 500. Furthermore, the edge transceiver unit 406 receives a parallax calculation model from the server 500.

[0085] Server 500 receives information from edge device 400, learns the disparity calculation model, and distributes it to edge device 400. Server 500 includes server-side transceiver unit 501, data storage unit 502, learning timing determination unit 503, learning policy determination unit 504, and disparity calculation model learning unit 505.

[0086] The server-side transceiver unit 501 (hereinafter also referred to as the image acquisition unit) communicates with the edge-side transceiver unit 406 of the edge device 400 to receive (acquire) images captured by the sensor unit 401. In addition, the server-side transceiver unit 501 sends a parallax calculation model to the edge device 400.

[0087] The data storage unit 502 stores the data sent from the edge device 400 and the parallax calculation model distributed to the edge device 400.

[0088] The learning timing determination unit 503 determines the timing for the parallax calculation model to relearn based on the data stored in the data storage unit 502.

[0089] From the driving condition or driving environment data output by the sensor unit 401, when it is found that the disparity calculation model in the model estimation unit 405 is learning unused data, the disparity calculation model is relearned. Data not included in the learned image is determined from GNSS location information, the working status of windshield wipers and headlights, illuminance, and weather forecast.

[0090] Furthermore, when the camera installed on the edge device 400 changes or when an anomaly occurs, the disparity calculation model is relearned. There are various camera anomaly states. If the color or contrast of the captured image changes due to lens dirt, relearning the disparity calculation model using that image can maintain its performance even in scenes with dirty lenses. On the other hand, if stereoscopic vision is difficult due to dirt, debris, or lens cracks preventing the camera from capturing images properly, the disparity calculation model is not relearned. Moreover, if such an image is sent to the server 500, it is excluded from the disparity calculation model's relearning process.

[0091] The learning policy decision unit 504 and the disparity calculation model learning unit 505 have the same functions as the learning policy decision unit 200 and the disparity calculation model learning unit 300 described in Embodiment 1.

[0092] In the learning policy decision unit 504, the disparity calculation model can be relearned in all common shooting areas of the multi-camera system, or it can be relearned only in common shooting areas that meet specific conditions, such as when the camera is updated (sensor configuration changes), an anomaly occurs (sensor state changes), or a new driving scene is acquired (additional data is collected). When relearning the disparity calculation model in all common shooting areas, the learning order is determined according to each common shooting area described in Example 1, and the disparity calculation model is learned in that order. On the other hand, when relearning the disparity calculation model in a subset of common shooting areas that meet the conditions, the learning order is determined for the common shooting area to be relearned and all common shooting areas that overlap with it, and the disparity calculation model is learned in that order.

[0093] In the disparity calculation model learning unit 505, since most mass-produced vehicles do not have LiDAR for acquiring correct values, the disparity calculation model is relearned using only additional disparity information. The relearned disparity calculation model is stored in the data storage unit 502. The relearned disparity calculation model stored in the data storage unit 502 is distributed to the model verification unit 404 via the edge transceiver unit 406 of the edge device 400. After verifying the actions in a real environment for a certain period of time, it is output to the model estimation unit 405 for use by the multi-camera system of the vehicle equipped with the edge device 400.

[0094] <Variation Example>

[0095] In the above embodiments, the camera configuration of the multi-camera system can be changed during learning data collection and system operation. Figure 3Taking the camera configurations (a) to (d) as an example, the accuracy of the added disparity information decreases due to the increased baseline length of the first camera 11 and the third camera 13. Therefore, a fourth camera (not shown) is added only during data collection, positioned between the first camera 11 and the third camera 13. After calculating the added disparity information for the first camera 11 and the fourth camera, and for the third camera 13 and the fourth camera, all the added disparity information is used to learn the disparity calculation model for the first camera 11 and the third camera 13. By setting the fourth camera in the middle to create a camera pair with a short baseline length, and using the highly accurate disparity information calculated there, the accuracy of the third disparity calculation model can be improved. The fourth camera can be set only during data collection and can be removed during actual system operation.

[0096] <Summary>

[0097] As described above, the learning method involved in this embodiment is a computer-executed learning method, including: the step of acquiring images captured by multiple cameras (11, 12, 13) having at least partially overlapping fields of view in common shooting areas (21, 22, 23); and the step of determining the execution order of learning a disparity estimation model for each of the multiple common shooting areas (21, 22, 23) to estimate disparity using the images (learning policy determination unit 200). Figure 8 ); and the steps of inputting images acquired by multiple cameras with common shooting areas of the learning object into the disparity inference model according to the determined execution order, and learning the disparity inference model based on the difference between the obtained disparity and the correct value data (disparity calculation model learning unit 300). Figure 4 The correct value data includes parallax information of the common shooting regions that were assigned to the common shooting regions of the learning object in the order of execution.

[0098] Therefore, the parallax calculation model is learned sequentially for multiple shared shooting areas (21, 22, 23).

[0099] In addition, the step of estimating the difficulty of parallax calculation for multiple common shooting areas (21, 22, 23) based on at least one of the characteristics and configuration of the multiple cameras (11, 12, 13) is (the sensor configuration priority determination unit 201 and sensor type priority determination unit 202 of the learning policy determination unit 200). Figure 8 In steps S101 and S103, based on the estimated results, the order of learning execution of the disparity estimation model is determined for multiple common shooting areas (21, 22, 23).

[0100] Therefore, the learning order of the disparity calculation model is determined based on the ease or difficulty of disparity estimation.

[0101] Furthermore, the step of generating the correct value data is included based on point cluster data acquired by LiDAR configured on a vehicle equipped with the plurality of cameras (11, 12, 13), and the sensor configuration priority determination unit 201 of the learning policy determination unit 200 is based on the positional relationship between the plurality of cameras and the LiDAR. Figure 8 In step S102, the order of parallax estimation model learning execution is determined for multiple common shooting areas (21, 22, 23).

[0102] Therefore, the learning order of the disparity calculation model is determined based on the positional relationship of LiDAR.

[0103] Furthermore, the correct value data includes at least one of the disparities obtained from the acquired image through rule-based models or feature point matching. Figure 5 ).

[0104] Therefore, the correct value of parallax can be produced using methods other than LiDAR.

[0105] Furthermore, the disparity inference model learning execution order is determined based on at least one of the changes in sensor configuration, changes in sensor status, and additional data collection conditions, and the disparity inference model is then learned again (Example 2).

[0106] This enables model updates after productization.

[0107] Furthermore, based on the number of LiDAR point cluster data contained in each of the multiple shared shooting areas (21, 22, 23), the learning execution order of the disparity inference model is determined for the multiple shared shooting areas (21, 22, 23). Figure 8 Step S102).

[0108] Therefore, the learning order of the disparity calculation model can be determined, for example, based on the number of LiDAR point cluster data that can be predicted according to the LiDAR positional relationships.

[0109] Furthermore, the learning system 2 involved in this embodiment is a learning system 2, comprising: an in-vehicle information processing device (edge ​​device 400) mounted on a vehicle 10 having multiple cameras (11, 12, 13) having at least partially overlapping fields of view in common shooting areas (21, 22, 23); and a server device (server 500) communicating with the in-vehicle information processing device (edge ​​device 400). The learning system includes: an image acquisition unit (server-side transceiver unit 501) for acquiring images captured by the multiple cameras (11, 12, 13); a learning policy determination unit 504 for determining the learning execution order of the disparity calculation model for the multiple common shooting areas (21, 22, 23); and a disparity calculation model learning unit 505 for inputting images acquired by the multiple cameras having common shooting areas of the learning object into the disparity calculation model according to the determined execution order, and learning the disparity calculation model based on the difference between the obtained disparity and the correct value data, wherein the correct value data includes disparity information of the common shooting areas assigned to the common shooting areas of the learning object in the execution order.

[0110] Therefore, the parallax calculation model is learned sequentially for multiple shared shooting areas (21, 22, 23).

[0111] Furthermore, in the learning system 2 involved in this embodiment, the learning policy decision unit 504 estimates the disparity calculation difficulty for multiple common shooting areas (21, 22, 23) based on at least one of the characteristics and configuration of multiple cameras (11, 12, 13), and determines the learning execution order of the disparity calculation model for multiple common shooting areas (21, 22, 23) based on the estimation results.

[0112] Therefore, the learning order of the disparity calculation model is determined based on the ease or difficulty of disparity estimation.

[0113] Furthermore, the learning system 2 involved in this embodiment also includes a disparity correct value generation unit 301 that generates correct value data based on point cluster data acquired by a LiDAR configured on the vehicle, and a learning policy determination unit 504 that determines the learning execution order of the disparity calculation model for multiple common shooting areas (21, 22, 23) based on the positional relationship between multiple cameras and the LiDAR.

[0114] Therefore, the learning order of the parallax calculation model is determined based on the positional relationship between the camera and the LiDAR.

[0115] Furthermore, the learning system 2 involved in this embodiment also includes a disparity correct value generation unit 301 that generates correct value data based on point cluster data acquired by LiDAR configured on the vehicle, and a learning policy determination unit 504 that determines the learning execution order of the disparity calculation model for the multiple common shooting areas (21, 22, 23) based on the number of LiDAR point cluster data contained in each of the multiple common shooting areas (21, 22, 23).

[0116] Therefore, the learning order of the disparity calculation model can be determined, for example, based on the number of LiDAR point cluster data that can be predicted according to the LiDAR positional relationships.

[0117] Furthermore, in the learning system 2 involved in this embodiment, the correct value data includes at least one of the disparities obtained through rule-based models or feature point matching. Figure 5 ).

[0118] Therefore, the correct value of parallax can be produced using methods other than LiDAR.

[0119] Furthermore, in the learning system 2 involved in this embodiment, when the vehicle information processing device (edge ​​device 400) detects that at least one of the multiple cameras has been changed, it requests the server device (server 500) to relearn the disparity calculation model of the common shooting area including the shooting area of ​​the changed camera (camera change detection unit 402). The server device (server 500) generates a relearning model by relearning the disparity calculation model and distributes the relearning model to the vehicle information processing device (edge ​​device 400).

[0120] Therefore, the parallax calculation model can be relearned when the camera is changed, enabling model updates after productization.

[0121] Furthermore, in the learning system 2 involved in this embodiment, when the vehicle information processing device (edge ​​device 400) detects that at least one of the multiple cameras is abnormal, it notifies the server device (server 500) (camera abnormality detection unit 403) of the abnormal state of the detected camera. Based on the abnormal state, the server device (server 500) determines whether to perform relearning on the disparity calculation model of the common shooting area including the shooting area of ​​the camera that detected the abnormality. It performs relearning on the disparity calculation model that is determined to be relearned to generate a relearning model and distributes the relearning model to the vehicle information processing device (edge ​​device 400).

[0122] Therefore, the parallax calculation model can be relearned appropriately based on the abnormal state of the camera, so as to realize the model update after productization.

[0123] Furthermore, in the learning system 2 involved in this embodiment, the vehicle information processing device (edge ​​device 400) sends multiple images captured by multiple cameras to the server device (server 500). Based on the multiple images, the server device (server 500) determines whether it contains data that has not been used in the learning of at least one of the multiple disparity calculation models. If it contains data that has not been used in the learning of at least one of the multiple disparity calculation models, a relearning model is generated by relearning the disparity calculation model, and the relearning model is distributed to the vehicle information processing device (edge ​​device 400).

[0124] Therefore, it is possible to appropriately perform relearning of scenarios that the learned disparity calculation model has not yet learned, thereby enabling model updates after productization.

[0125] According to this embodiment, by learning the disparity calculation model sequentially from multiple common shooting areas, in other words, by determining the learning order of the disparity calculation model, the accuracy of the correct disparity values ​​used for learning can be improved, and the learning accuracy of the disparity calculation model can be enhanced.

[0126] In other words, according to this embodiment, in multiple common shooting areas of a multi-camera system, learning begins with the area where parallax calculation is easier, and the additional parallax information calculated during the learning process is used for learning subsequent common shooting areas, thereby improving the learning accuracy of the parallax calculation model.

[0127] The embodiments of the present invention have been described above, but the above embodiments are only a part of the application examples of the present invention and are not intended to limit the technical scope of the present invention to the specific configuration of the above embodiments.

[0128] Furthermore, the present invention is not limited to the above embodiments and includes various modifications. For example, the above embodiments are described in detail to facilitate understanding of the present invention and are not necessarily limited to having all the described configurations.

[0129] Furthermore, the aforementioned components, functions, processing units, and processing methods can be partially or entirely implemented in hardware, such as through integrated circuit design. Alternatively, the aforementioned components and functions can be implemented in software by a processor interpreting and executing programs that implement each function. The programs, tables, files, and other information implementing each function can be stored in storage devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs.

[0130] Furthermore, the control lines and information lines shown are those deemed necessary for the instructions; not all control lines and information lines may be shown on the product itself. In reality, it can be assumed that almost all components are interconnected.

[0131] Symbol Explanation

[0132] 1. Image processing apparatus (Example 1)

[0133] 100 Data Storage Department

[0134] 200 Learning Policy Decision-Making Department

[0135] 201 Sensor Configuration Priority Determination Department

[0136] 202 Sensor Type Priority Determination Department

[0137] 203 Joint Shooting Area Order Determination Department

[0138] 300 Parallax Calculation Model Learning Department

[0139] 301 Parallax Correct Value Production Department

[0140] 302 Learning and Assessment Department

[0141] 303 Model Storage Department

[0142] 2. Learning System (Example 2)

[0143] 400 Edge devices (vehicle information processing devices)

[0144] 401 Sensor Department

[0145] 402 Camera Alteration Detection Department

[0146] 403 Camera Anomaly Detection Department

[0147] 404 Model Validation Department

[0148] 405 Model Estimation Department

[0149] 406 Edge-side transceiver unit

[0150] 500 Server (Server Device)

[0151] 501 Server-side transceiver unit (image acquisition unit)

[0152] 502 Data Storage Department

[0153] 503 Learning Timing Decision Department

[0154] 504 Learning Policy Decision-Making Department

[0155] 505 Parallax Calculation Model Learning Department

Claims

1. A learning system comprising an in-vehicle information processing device mounted on a vehicle having multiple cameras having a common shooting area with at least partial overlap in field of view, and a server device communicating with the in-vehicle information processing device, the learning system being characterized in that it includes: An image acquisition unit acquires images captured by the plurality of cameras; The learning policy decision-making department determines the learning execution order of the parallax calculation model for multiple jointly captured areas; as well as The disparity calculation model learning unit, according to a predetermined execution order, inputs images acquired by multiple cameras covering a common shooting area of ​​the learning object into the disparity calculation model, and learns the disparity calculation model based on the difference between the obtained disparity and the correct value data. The correct value data includes parallax information of the common shooting areas that were assigned to the order of execution before the common shooting areas of the learning object.

2. The learning system as described in claim 1, characterized in that, The learning policy decision-making department, Based on at least one of the characteristics and configurations of the multiple cameras, the ease of parallax estimation is estimated for multiple shared shooting areas. Based on the estimation results, the learning execution order of the disparity calculation model is determined for multiple shared shooting areas.

3. The learning system as described in claim 1, characterized in that, It also includes a disparity accuracy generation unit that generates the accuracy data based on point cluster data acquired by a LiDAR system configured on the vehicle. The learning policy decision-making unit determines the learning execution order of the parallax calculation model for multiple shared shooting areas based on the positional relationship between the multiple cameras and the LiDAR.

4. The learning system as described in claim 1, characterized in that, It also includes a disparity accuracy generation unit that generates the accuracy data based on point cluster data acquired by a LiDAR system configured on the vehicle. The learning policy decision-making unit determines the execution order of parallax calculation model learning for multiple shared shooting areas based on the number of LiDAR point cluster data contained in each of the multiple shared shooting areas.

5. The learning system as described in claim 1, characterized in that, The correct value data includes at least one of the disparities obtained through rule-based models or feature point matching.

6. The learning system as described in claim 1, characterized in that, When the vehicle-mounted information processing device detects that at least one of the plurality of cameras has been altered, it requests the server device to relearn the disparity calculation model of the common shooting area that includes the shooting area of ​​the altered camera. The server device, A relearning model is generated by relearning the disparity calculation model. The relearning model is distributed to the vehicle information processing device.

7. The learning system as described in claim 1, characterized in that, When the vehicle information processing device detects an anomaly in at least one of the plurality of cameras, it notifies the server device of the abnormal status of the detected camera. The server device, Based on the abnormal state, determine whether to relearn the disparity calculation model for the shared shooting area of ​​the camera that detected the abnormality. The disparity calculation model that is determined to require relearning is relearned to generate a relearned model. The relearning model is distributed to the vehicle information processing device.

8. The learning system as described in claim 1, characterized in that, The vehicle-mounted information processing device sends multiple images captured by the multiple cameras to the server device. The server device, Based on the multiple images, determine whether they contain data that was not used by at least one of the multiple disparity calculation models during learning. In cases where at least one of the multiple disparity calculation models has learned unused data, a relearning model is generated by relearning the disparity calculation model. The relearning model is distributed to the vehicle information processing device.

9. A learning method executed by a computer, characterized in that, include: The steps of acquiring images captured by multiple cameras having a common shooting area with at least partial overlap in field of view; The steps for determining the execution order of learning the disparity inference model by using the images to infer disparity for multiple jointly captured areas; Following the determined execution order, images acquired by multiple cameras covering a common shooting area of ​​the learning target are input into the disparity inference model. The disparity inference model is then learned based on the difference between the obtained disparity and the correct value data. The correct value data includes parallax information of the common shooting areas that were assigned to the order of execution before the common shooting areas of the learning object.

10. The learning method as described in claim 9, characterized in that, This includes the step of estimating the difficulty of parallax calculation for multiple shared shooting areas based on at least one of the characteristics and configurations of the multiple cameras. Based on the estimated results, the learning execution order of the disparity estimation model is determined for multiple shared shooting areas.

11. The learning method as described in claim 9, characterized in that, This includes the step of generating the correct value data based on point cluster data acquired by a LiDAR system configured on a vehicle equipped with the multiple cameras. Based on the positional relationship between the multiple cameras and the LiDAR, the learning execution order of the disparity estimation model is determined for multiple shared shooting areas.

12. The learning method as described in claim 9, characterized in that, The correct value data includes at least one of the disparities obtained from the acquired image through rule-based modeling or feature point matching.

13. The learning method as described in claim 9, characterized in that, This includes the step of generating the correct value data based on point cluster data acquired by a LiDAR system configured on a vehicle equipped with the multiple cameras. The learning execution order of the disparity inference model is determined based on the number of LiDAR point clusters contained in each of the multiple shared shooting areas.

14. The learning method as described in claim 9, characterized in that, The learning execution order of the disparity inference model is determined based on at least one of the changes in sensor configuration, changes in sensor status, and additional data collection conditions, and then the disparity inference model is learned again.

Images