Obstacle recognition methods, devices, equipment, storage media, and software products

Abstract

This application discloses an obstacle recognition method, apparatus, device, storage medium, and program product. The method includes: acquiring a target disparity map, which is obtained by calculating the V-disparity of an image of an obstacle to be recognized; performing obstacle detection on the target disparity map to obtain an obstacle region, the obstacle region including at least one obstacle point; and performing obstacle recognition based on the obstacle region to obtain the corresponding target obstacle. Using this application, the problems of low obstacle recognition accuracy, low precision, and increased computational complexity in existing technologies can be solved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and includes, but is not limited to, an obstacle recognition method, device, equipment, storage medium, and program product. Background Technology

[0002] Obstacle detection and recognition are fundamental to autonomous driving, and improving the speed and accuracy of obstacle recognition has always been a key research area. However, directly detecting obstacles on the road surface can be affected by factors such as the ground and surrounding debris, which can interfere with the detection results.

[0003] Therefore, existing obstacle detection and recognition schemes are mainly based on a combination of vision and LiDAR. However, in practice, it has been found that the effective range and ranging accuracy of vision and LiDAR are easily affected by environmental and weather factors, resulting in low accuracy and low precision in obstacle detection and recognition. Furthermore, existing schemes typically employ grid-based obstacle recognition algorithms that project LiDAR-collected data onto a grid map before performing obstacle detection and recognition on the network map, which increases the computational complexity of obstacle recognition. Summary of the Invention

[0004] In view of this, the obstacle recognition method, apparatus, device, storage medium, and program product provided in this application can solve the problems of low obstacle recognition accuracy, low precision, and increased computational complexity in the prior art.

[0005] In a first aspect, embodiments of this application provide an obstacle recognition method, including:

[0006] Obtain a target disparity map, which is obtained by calculating the V disparity of the image of the obstacle to be identified;

[0007] Obstacle detection is performed on the target disparity map to obtain an obstacle region, wherein the obstacle region includes at least one obstacle point;

[0008] Obstacle identification is performed based on at least one obstacle point in the obstacle region to obtain the corresponding target obstacle.

[0009] In some embodiments, the target disparity map includes a target straight line function obtained by projecting the obstacle image longitudinally, and the obstacle detection on the target disparity map to obtain the obstacle region includes:

[0010] Obtain the x-coordinate and y-coordinate values ​​of each pixel in the target disparity map;

[0011] Based on the target linear function and the x and y coordinates of each pixel in the target disparity map, obstacle detection is performed on the target disparity map to obtain the obstacle region.

[0012] In some embodiments, the obstacle detection process performed on the target disparity map based on the target linear function and the x-coordinate and y-coordinate values ​​of each pixel in the target disparity map to obtain the obstacle region includes:

[0013] The x-coordinate value of each pixel in the target disparity map is substituted into the target linear function to calculate the predicted y-coordinate value of each pixel.

[0014] All pixels in the target disparity map that meet the first preset condition are identified as obstacle points. The first preset condition includes that the predicted ordinate of the pixel is greater than the ordinate value of the pixel, or that the difference between the predicted ordinate of the pixel and the ordinate value of the pixel is greater than a first threshold.

[0015] The obstacle region is obtained based on the obstacle points.

[0016] In some embodiments, the obstacle detection process performed on the target disparity map based on the target linear function and the x-coordinate and y-coordinate values ​​of each pixel in the target disparity map to obtain the obstacle region includes:

[0017] The predicted horizontal value of each pixel is obtained by substituting the ordinate value of each pixel in the target disparity map into the target linear function.

[0018] All pixels in the target disparity map that meet the first preset condition are identified as obstacle points. The first preset condition includes that the predicted horizontal value of the pixel is greater than the horizontal coordinate value of the corresponding pixel, or that the difference between the predicted horizontal value of the pixel and the horizontal coordinate value of the pixel is greater than a first threshold.

[0019] The obstacle region is obtained based on the obstacle points.

[0020] In some embodiments, the obstacle identification based on the obstacle region to obtain the identification result includes:

[0021] The obstacle recognition model is invoked to identify obstacles in the obstacle area, and the recognition results are obtained.

[0022] The obstacle recognition model is obtained by training the model to be trained using a training sample set. The training sample set includes at least one pre-selected anchor box and at least one pre-selected obstacle, with the pre-selected obstacle located within the pre-selected anchor box.

[0023] In some embodiments, the method for generating the obstacle recognition model includes:

[0024] The training sample set is used to perform the current batch training on the model to be trained in order to update the parameters of the model to be trained.

[0025] The average classification accuracy of the current batch training is calculated by performing a classification average accuracy calculation on all pre-selected obstacles used in the current batch training and all previous batch training.

[0026] If the difference between the average accuracy of the current iteration and the average accuracy of the previous iteration is greater than or equal to the third threshold, then the process of performing batch training on the model to be trained using the training sample set is repeated until the difference between the average accuracy of the current iteration and the average accuracy of the previous iteration is less than the third threshold, at which point the obstacle recognition model is output.

[0027] Secondly, embodiments of this application provide an obstacle recognition device, comprising:

[0028] The acquisition module is used to acquire a target disparity map, which is obtained by calculating the V disparity of the image of the obstacle to be identified;

[0029] The processing module is used to perform obstacle detection on the target disparity map to obtain an obstacle region, wherein the obstacle region includes at least one obstacle point;

[0030] The identification module is used to identify obstacles based on at least one obstacle point in the obstacle area to obtain the corresponding target obstacle.

[0031] For any content not introduced or described in the embodiments of this application, please refer to the relevant descriptions in the foregoing method embodiments; they will not be repeated here.

[0032] Thirdly, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement the method described in embodiments of this application.

[0033] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in embodiments of this application.

[0034] Fifthly, embodiments of this application provide a computer program product having instructions stored thereon and a computer program stored thereon, which, when executed by a processor, implements the method described in embodiments of this application.

[0035] Compared with the prior art, this application has at least the following beneficial effects:

[0036] In this embodiment, the computer device acquires a target disparity map, which is obtained by calculating the V-disparity of an image of an obstacle to be identified. Obstacle detection is performed on the target disparity map to obtain an obstacle region, which includes at least one obstacle point. Obstacle identification is then performed based on at least one obstacle point in the obstacle region to obtain the corresponding target obstacle. Therefore, this application can perform obstacle detection and identification based on a target disparity map, significantly improving the speed and accuracy of obstacle identification. Compared with existing technologies, this application also solves the problems of low obstacle identification accuracy, low precision, and increased computational complexity in existing technologies. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0038] Figure 1 This is a flowchart illustrating an obstacle recognition method provided in an embodiment of this application.

[0039] Figure 2 This is a schematic diagram of a road image provided in an embodiment of this application.

[0040] Figure 3 This is a schematic diagram of a target disparity map provided in an embodiment of this application.

[0041] Figure 4 This is a schematic diagram of an obstacle area provided in an embodiment of this application.

[0042] Figure 5 This is a schematic diagram illustrating the effect of obstacle recognition provided in an embodiment of this application.

[0043] Figure 6 This is a schematic diagram of a preset anchor frame and preset obstacles in a training image provided in an embodiment of this application.

[0044] Figure 7 This is a schematic diagram of the curve showing the change of average accuracy as a function of the number of batch training sessions, provided in an embodiment of this application.

[0045] Figure 8 This is a schematic diagram of the structure of an obstacle recognition device provided in an embodiment of this application.

[0046] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0049] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0050] It should be noted that the terms "first, second, third" used in the embodiments of this application are used to distinguish similar or different objects and do not represent a specific order of objects. It can be understood that "first, second, third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0051] In the process of filing this application, the applicant also discovered that existing obstacle detection and recognition schemes based on the fusion of vision and LiDAR involve transforming the obtained radar coordinate system and the camera's pixel coordinate system to match obstacles detected by radar in space to the visual image. Based on this, obstacle data detected by radar, such as motion data, is output. Then, obstacle recognition is performed using a convolutional neural network, for example, identifying candidate regions, and then classifying and recognizing obstacles within these candidate regions. Alternatively, obstacle recognition methods based on LiDAR can also involve using geometric feature-based algorithms to first process the data output by the LiDAR, then employing clustering algorithms to cluster the data and compare it with the geometric features of obstacles to complete obstacle classification and recognition.

[0052] In practice, it has been found that visual (camera) and LiDAR systems, due to their relatively poor operating range and ranging accuracy, are easily affected by factors such as lighting and weather. Obstacle classification schemes based on candidate regions involve adjusting the bounding boxes of candidate regions, requiring repeated calculation of the convolutional features for each candidate region, resulting in slow obstacle recognition speeds. LiDAR-based obstacle recognition schemes primarily use grid-based algorithms to project LiDAR-collected data onto a grid map, then utilize undirected graph correlation methods to process the point cloud data in the grid map. The grid size and structure can be customized. Using a grid distribution image to represent obstacles, higher resolution grids represent more complex obstacles, requiring higher computational complexity and memory.

[0053] To address the aforementioned problems, this application proposes an obstacle recognition method, apparatus, device, storage medium, and program product. Please refer to [link / reference]. Figure 1 This is a flowchart illustrating an obstacle recognition method provided in an embodiment of this application. Figure 1 The methods shown include:

[0054] S101. The computer device acquires a target disparity map, which is obtained by performing V-disparity calculation on the image of the obstacle to be identified.

[0055] This application does not limit the implementation method of obtaining the target disparity map. For example, in one example, this application can obtain an image of an obstacle to be identified. This application does not limit the implementation method of obtaining the obstacle image; for example, it can be obtained by taking a picture using a camera device or by receiving images sent by other devices via a network. Further, V-disparity calculation is performed on the obstacle image, for example, using a V-disparity algorithm to calculate the V-disparity map of the obstacle image, thereby obtaining the aforementioned target disparity map. In another example, this application can obtain the image from other devices (e.g., terminal devices or servers) or websites via a network.

[0056] The V-disparity map of this application mainly consists of vertical and diagonal lines, and is obtained by statistically analyzing the V-rows of the disparity map. It is equivalent to a horizontal projection map, or a side view of the disparity map. In the field of road obstacle detection, the obstacle image to be identified can be a road image, which may include target obstacles to be identified, including but not limited to, buses, cars, trucks, trees, pedestrians, dogs, cats, or other types of obstacles. Points on the road surface are mapped as a sloping straight line in the V-disparity map, referred to below as the target line. For example, please refer to... Figure 2 and Figure 3These are schematic diagrams of a possible road image and the corresponding V-disparity map provided in an embodiment of this application. Figure 2 The road image shown depicts three cars traveling on the road surface; these three cars are the target obstacles to be identified. Figure 3 Show Figure 2 The V-disparity map obtained by performing V-disparity calculation on the road image shown is the target disparity map mentioned above in this application. The white straight line included in this target disparity map is the target straight line mentioned above, which is a straight line obtained by projecting the road surface in the longitudinal direction. This application will not limit or elaborate on it here.

[0057] In this embodiment, the entire portion of the target disparity map above the target straight line is defined as an obstacle; that is, the area above the target straight line is defined as the obstacle region. To a certain extent, this can be understood as the portion protruding from the road surface, typically an obstacle. The target straight line in the target disparity map can be calculated based on the horizontal and vertical coordinates of each pixel in the target disparity map. The specific calculation method is not described in detail here.

[0058] S102. The computer device performs obstacle detection on the target disparity map to obtain an obstacle region, wherein the obstacle region includes at least one obstacle point.

[0059] This application can obtain the coordinate values ​​of each pixel in the target disparity map, which may include the horizontal and vertical coordinate values ​​of the pixel. Then, based on the target line included in the target disparity map and the horizontal and vertical coordinate values ​​of each pixel in the target disparity map, obstacle detection is performed on the target disparity map to obtain the obstacle region. This application does not limit the implementation method of obstacle detection; two possible implementation methods are exemplified below.

[0060] In one possible implementation, since the abscissa value of each pixel in the V-disparity map (i.e., the target disparity map mentioned above) may exhibit projection anomalies or singular values, this application, considering computational precision and accuracy, can use the ordinate value of each pixel in the target disparity map as a reference value for obstacle detection. In implementation, this application can substitute the abscissa value of each pixel in the target disparity map into the target line mentioned above for calculation, thereby obtaining the predicted ordinate value of each pixel, referred to simply as the predicted ordinate value. Then, it is determined whether the predicted ordinate value and ordinate value of each pixel meet a first preset condition. This first preset condition is a system-defined setting, which may include, but is not limited to, for example, the predicted ordinate value of a pixel being greater than its ordinate value, the difference between the predicted ordinate value and its ordinate value being greater than a first threshold, or other custom conditions. The first threshold is also a threshold defined by the system or the user, such as an empirical value set based on user experience.

[0061] Furthermore, this application can identify all pixels in the target disparity image that satisfy the first preset condition as obstacle points, meaning these pixels are all located within the corresponding obstacle region. Finally, the corresponding obstacle region can be obtained based on these obstacle points. This application does not limit the number of obstacle regions; it can be determined according to the actual situation, and there can be one or more. This application does not limit the implementation method for determining obstacle regions. For example, this application can determine whether these obstacle points are located within the same region based on the horizontal and vertical coordinates of each obstacle point and the positional relationship between the obstacle points. If so, the region corresponding to that same region in the obstacle image is then determined as the corresponding obstacle region.

[0062] In practical applications, this application can express the equation function of the target line (referred to as the target line function) as y road =k road x+b road This application can substitute the abscissa value of each pixel in the target disparity map into the aforementioned target linear function to calculate the corresponding output value of that pixel, which is the predicted ordinate value mentioned above, denoted as f. Then, the ordinate value y of the pixel is... road Compare with the predicted ordinate f; if f > y road If the pixel is found to be an obstacle, it can be determined that the pixel is located within an obstacle area. To improve the reliability of obstacle detection, considering the unevenness of the road surface, the difference value of these uneven areas can be denoted as η0. This value can be customized by the system according to the actual situation, and this application does not impose any limitations. Accordingly, this application can determine the predicted ordinate f and ordinate y of the pixel. road Is the difference between them greater than η0? If fy roadIf the predicted ordinate of the pixel is greater than η0, it indicates that the predicted ordinate of the pixel is higher than the actual ordinate of the road surface, and the pixel is an obstacle point, meaning that the pixel is located within the obstacle area. Similarly, this application can traverse every pixel in the target disparity map, filter out all obstacle points, and thus obtain the corresponding obstacle area in the obstacle image.

[0063] For example, combining Figure 2 For the example, please refer to Figure 4 This is a schematic diagram of a possible obstacle area provided in this application. For example... Figure 4 As shown, this application uses the above principle to detect three obstacle regions in the obstacle image, which are shown in the figure with wireframes respectively. This application will not elaborate further here.

[0064] In another possible implementation, without considering computational precision and accuracy, this application can use the abscissa value of each pixel in the target disparity map as a reference value for obstacle detection. In implementation, this application can substitute the ordinate value of each pixel in the target disparity map into the aforementioned target linear function to calculate the predicted abscissa value of each pixel, referred to simply as the predicted abscissa. Then, it is determined whether the predicted abscissa value and abscissa value of each pixel satisfy a second preset condition. This second preset condition is a system-defined setting and may include, but is not limited to, conditions such as the predicted abscissa value being greater than the pixel's abscissa value, the difference between the predicted abscissa value and the pixel's abscissa value being greater than a second threshold, or other custom conditions. The second threshold is also a system- or user-defined threshold, such as an empirical value set based on user experience.

[0065] Furthermore, this application can identify all pixels in the target disparity map that satisfy the second preset condition as obstacle points, meaning these pixels are all located within the corresponding obstacle region. Finally, the corresponding obstacle region can be obtained based on these obstacle points. This application does not limit the number of obstacle regions; it can be determined according to the actual situation, and there can be one or more. This application does not limit the implementation method for determining obstacle regions. For example, this application can determine whether these obstacle points are located within the same region based on the horizontal and vertical coordinates of each obstacle point and the positional relationship between the obstacle points. If so, the region corresponding to that same region in the obstacle image is then determined as the corresponding obstacle region.

[0066] In practical applications, this application can express the equation function of the target line (referred to as the target line function) as y road =k road x+b roadThis application can substitute the ordinate value of each pixel in the target disparity map into the aforementioned target linear function to calculate the corresponding output value of that pixel, which is the predicted horizontal value, denoted as I. Then, the horizontal coordinate value x of that pixel is... road Compare with the predicted horizontal value I; if I > x road If the pixel is found to be an obstacle, it can be determined that the pixel is located within an obstacle area. To improve the reliability of obstacle detection, considering the unevenness of the road surface, the difference value of these uneven areas can be denoted as η1. This value can be customized by the system according to the actual situation, and this application does not impose any limitations. Accordingly, this application can determine the predicted horizontal value I and the horizontal coordinate value x of the pixel. road Is the difference between them greater than η1? If Ix road If the predicted x-coordinate of the pixel is greater than η1, it indicates that the pixel is an obstacle point, meaning it is located within an obstacle region. Similarly, this application can iterate through each pixel in the target disparity map, filter out all obstacle points, and thus obtain the corresponding obstacle region.

[0067] S103. The computer device identifies obstacles based on the obstacle area to obtain the corresponding target obstacle.

[0068] This application does not limit the implementation method of obstacle recognition. For example, this application can call a pre-trained obstacle recognition model to perform obstacle recognition on an obstacle region, thereby identifying the target obstacle included in the obstacle image corresponding to the obstacle region. For example, combined with Figure 2 For the example, please refer to Figure 5 This is a schematic diagram illustrating the effect of obstacle recognition provided in an embodiment of this application. Figure 5 As shown, the solution of this application can identify three target obstacles in the obstacle image, namely three cars.

[0069] The obstacle recognition model of this application is obtained by training a training sample set in advance. The training sample set includes at least one pre-selected anchor box and at least one pre-selected obstacle. Each pre-selected obstacle is located within a pre-selected anchor box. To a certain extent, the pre-selected anchor box of this application can be understood as the object box of the pre-selected obstacle.

[0070] Before using the obstacle recognition model for obstacle recognition, this application needs to train the obstacle recognition model; that is, before performing obstacle recognition, this application needs to generate the aforementioned obstacle recognition model. This application does not limit the entity that performs the generation or training of the obstacle recognition model. For example, a training device can pre-train / generate the obstacle recognition model. This training device includes, but is not limited to, terminal devices, servers, or third-party devices, etc., and this application does not impose any limitations. Optionally, the training device and the computer equipment of this application can be the same or different, and can be determined according to the actual situation; this application does not impose any limitations.

[0071] The process of training / generating the obstacle recognition model using a training device can be as follows: The training device can acquire a training sample set for model training. This training sample set may include at least one pre-selected anchor box and at least one pre-selected obstacle. These pre-selected anchor boxes and pre-selected obstacles may be represented in pairs in the same or more training images, which is not limited in this application. For example, please refer to... Figure 6 This is a schematic diagram of a possible training image containing preset anchor boxes and preset obstacles, provided in an embodiment of this application. Figure 6 The example shown is based on a training image containing a preset obstacle and a pre-selected anchor box, but it is not a limitation and can be determined according to the actual situation.

[0072] This application does not limit the source of the training images (i.e., the training sample set). For example, they can come from website platforms, such as the vehicle classification project on the PaddlePaddle AI Studio open platform; or they can be frame images obtained by a camera, such as video frame images from a video recording. This application does not limit the number of training images. Generally, the more training images, the higher the accuracy of the obstacle recognition model obtained.

[0073] This application does not limit the implementation method for obtaining pre-selected anchor boxes and pre-selected obstacles. For example, this application can use a clustering algorithm (such as K-means) to perform cluster analysis on each training image to obtain the corresponding pre-selected anchor boxes and pre-selected obstacles. This application does not limit the number of pre-selected anchor boxes and pre-selected obstacles, which can be determined according to actual needs. Optionally, all pre-selected anchor boxes selected in this application must satisfy the condition that the average intersection over union (IOU) is greater than or equal to a preset threshold. This preset threshold is set by the system or user, such as a statistical value calculated based on a series of experimental data, and this application does not limit it.

[0074] For example, before model training, this application can first use the K-means clustering algorithm to cluster each training image to select suitable pre-selected anchor boxes corresponding to pre-selected obstacles. Suppose that in this example, five suitable pre-selected anchor boxes are selected, with dimensions of (17, 32), (39, 41), (73, 54), (60, 41), and (168, 143). Furthermore, the average Intersection over Union (IOU) of these five pre-selected anchor boxes is 0.867, which is greater than the preset threshold of 0.7, indicating that all five selected pre-selected anchor boxes are suitable or applicable.

[0075] Furthermore, the execution device can use the training sample set to iteratively train the model to be trained, thereby obtaining the trained obstacle recognition model. In one embodiment, the execution device can repeatedly perform the following steps until the mean average precision (mAP) of the model to be trained tends to stabilize (i.e., does not change with batch training), and output the final obstacle recognition model: This application can use the training sample set to batch train the model to be trained to update the parameters of the model to be trained, and calculate the classification average precision of the pre-selected anchor boxes used in all batch training, thereby obtaining the mean average precision used to evaluate the model to be trained.

[0076] In another embodiment, the execution device can use the training sample set to perform the current batch training of the model to be trained, thereby updating the parameters of the model to be trained; and calculate the classification average accuracy of the pre-selected anchor boxes used in the current and previous batch training sessions, thereby obtaining the average accuracy of the current training session, which can be used to evaluate the quality of the model to be trained obtained in the current training session. If the difference between the average accuracy of the current training session and the average accuracy of the previous training session is greater than or equal to a third threshold, the application can repeat the above-mentioned step of using the training sample set to perform the current batch training of the model to be trained, thereby updating the parameters of the model to be trained, until the difference between the average accuracy of the current training session and the average accuracy of the previous training session is less than the third threshold, at which point the obstacle recognition model can be output. That is, when the difference between the average accuracy obtained in the current batch training session and the average accuracy obtained in the previous batch training session is less than the third threshold, the training process can be terminated, and the model to be trained obtained in the current batch training session can be output as the obstacle recognition model. The third threshold in this application can be a threshold set by the system, such as a value obtained by statistical analysis based on a series of experimental data, and this application does not limit it.

[0077] This application does not limit the number of batches in batch training; it can be determined according to the actual needs of the system. For example, this application can use the training sample set for 100 batches (epochs) of training. During training, this application can use the mAP of pre-selected anchor boxes as an evaluation metric for model training. After each batch training is completed, the mAP of the pre-selected anchor boxes used in all previous batch training needs to be calculated to determine whether the calculated mAP changes with the number of batch training. If so, it indicates that model training is not complete, and the above batch training steps continue. Otherwise, it indicates that the model training area is stable, training can be terminated, and the final obstacle recognition model can be output.

[0078] The training model and obstacle recognition model in this application have the same network structure, which may include, but is not limited to, object detection model YOLOv5, feedforward neural network model, convolutional neural network model or other custom deep learning model, etc., and this application does not limit them.

[0079] For example, the training sample set used in this application may include 1383 training images. Each training image includes four categories of target obstacles: bus, car, truck, and pedestrian. This application uses the above training sample set to train the model for 100 epochs. Please refer to Tables 1 and 2 below for the relevant parameters of the randomly selected 46th and 74th epochs, respectively.

[0080] Table 1

[0081] category accuracy Recall rate mAP the bus 0.914 0.937 0.938 car 0.933 0.947 0.941 truck 0.922 0.946 0.953 pedestrian 0.871 0.926 0.891

[0082] Table 2

[0083] category accuracy Recall rate mAP the bus 0.927 0.952 0.943 car 0.923 0.941 0.948 truck 0.934 0.952 0.957 pedestrian 0.863 0.928 0.898

[0084] Accuracy, recall, and mAP can all be used as evaluation metrics for model training. As shown in Tables 1 and 2 above, the evaluation metrics of the models after the 46th and 74th batch training iterations are not significantly different, indicating that the model training has become relatively stable.

[0085] Understandably, to further confirm the stability of model training, this application needs to evaluate the training sample set used in batch training during the model training process. For example, it may perform classification mAP calculation on the training sample set used in batch training to obtain the mAP variation curve with batch training. Please refer to [link to relevant documentation]. Figure 7 This is a schematic diagram illustrating the change in mean average accuracy (mAP) as a function of the number of batches trained, according to an embodiment of this application. Figure 7As shown, in the early stages of training, mAP increases rapidly with the increase in the number of training batches, and the growth rate gradually slows down. However, in the later stages of training, mAP basically stabilizes and no longer changes with the number of training batches. The fluctuation range of mAP is very small, indicating that the model training is basically complete.

[0086] By implementing the embodiments of this application, obstacle detection can be performed based on a target disparity map, and obstacle regions are divided using target straight lines, thereby greatly improving the speed and accuracy of obstacle recognition. Furthermore, to facilitate visual display and analysis, this application trains the final obstacle recognition model using methods such as setting predetermined anchor frames and batch training. In the embodiments of this application, the computer device acquires a target disparity map, which is obtained by calculating the V-disparity of the obstacle image to be recognized; obstacle detection is performed on the target disparity map to obtain obstacle regions, each including at least one obstacle point; obstacle recognition is performed based on at least one obstacle point in the obstacle region to obtain the corresponding target obstacle. It is evident that this application can perform obstacle detection and recognition based on a target disparity map, greatly improving the speed and accuracy of obstacle recognition. Compared with existing technologies, this application also solves the problems of low obstacle recognition accuracy, low overall accuracy, and increased computational complexity in existing technologies.

[0087] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0088] Based on the foregoing embodiments, this application provides an obstacle recognition device, which includes various modules and units included in each module, and can be implemented by a processor; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP) or field programmable gate array (FPGA), etc.

[0089] Please see Figure 8 This is a schematic diagram of the structure of an obstacle recognition device provided in an embodiment of this application.

[0090] like Figure 8 The device shown includes an acquisition module 801, a processing module 802, and an identification module 803; wherein:

[0091] The acquisition module 801 is used to acquire a target disparity map, which is obtained by calculating the V disparity of the obstacle image to be identified.

[0092] The processing module 802 is used to perform obstacle detection on the target disparity map to obtain an obstacle region, wherein the obstacle region includes at least one obstacle point;

[0093] The identification module 803 is used to identify obstacles based on at least one obstacle point in the obstacle area to obtain the corresponding target obstacle.

[0094] In some embodiments, the target disparity map includes a target straight line function obtained by projecting the obstacle image longitudinally, and the processing module 802 is specifically used for:

[0095] Obtain the x-coordinate and y-coordinate values ​​of each pixel in the target disparity map;

[0096] Based on the target linear function and the x and y coordinates of each pixel in the target disparity map, obstacle detection is performed on the target disparity map to obtain the obstacle region.

[0097] In some embodiments, the processing module 802 is specifically used for:

[0098] The x-coordinate value of each pixel in the target disparity map is substituted into the target linear function to calculate the predicted y-coordinate value of each pixel.

[0099] All pixels in the target disparity map that meet the first preset condition are identified as obstacle points. The first preset condition includes that the predicted ordinate of the pixel is greater than the ordinate value of the pixel, or that the difference between the predicted ordinate of the pixel and the ordinate value of the pixel is greater than a first threshold.

[0100] The obstacle region is obtained based on the obstacle points.

[0101] In some embodiments, the processing module 802 is specifically used for:

[0102] The predicted horizontal value of each pixel is obtained by substituting the ordinate value of each pixel in the target disparity map into the target linear function.

[0103] All pixels in the target disparity map that meet the first preset condition are identified as obstacle points. The first preset condition includes that the predicted horizontal value of the pixel is greater than the horizontal coordinate value of the pixel, or that the difference between the predicted horizontal value of the pixel and the horizontal coordinate value of the pixel is greater than a second threshold.

[0104] The obstacle region is obtained based on the obstacle points.

[0105] In some embodiments, the identification module 803 is specifically used for:

[0106] The obstacle recognition model is invoked to identify obstacles in the obstacle region, thereby obtaining the corresponding target obstacles;

[0107] The obstacle recognition model is obtained by training the model to be trained using a training sample set. The training sample set includes at least one pre-selected anchor box and at least one pre-selected obstacle, with the pre-selected obstacle located within the pre-selected anchor box.

[0108] In some embodiments, the processing module 802 is further configured to:

[0109] The training sample set is used to perform the current batch training on the model to be trained in order to update the parameters of the model to be trained.

[0110] The average classification accuracy of the current batch training is calculated by performing a classification average accuracy calculation on all the pre-selected anchor boxes used in the current batch training and all previous batch training.

[0111] If the difference between the average accuracy of the current iteration and the average accuracy of the previous iteration is greater than or equal to the third threshold, then the process of performing batch training on the model to be trained using the training sample set is repeated until the difference between the average accuracy of the current iteration and the average accuracy of the previous iteration is less than the third threshold, at which point the obstacle recognition model is output.

[0112] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0113] It should be noted that, in the embodiments of this application... Figure 8The obstacle recognition device shown is illustrative of the module division, representing only one logical functional division; in actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical units, or be integrated into one unit with two or more units. The integrated units can be implemented in hardware, as software functional units, or a combination of both.

[0114] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0115] This application provides a computer device, which may be a server or a smartphone, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements an obstacle recognition method.

[0116] The computer equipment covered by this application includes, but is not limited to, smartphones (such as Android phones, iOS phones, etc.), personal computers, tablets, handheld computers, e-readers, mobile internet devices (MIDs), wearable smart devices, or other devices with communication functions.

[0117] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method provided in the above embodiments.

[0118] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the method provided in the above-described method embodiments.

[0119] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0120] In one embodiment, the apparatus provided in this application can be implemented as a computer program, and the computer program can be implemented as follows: Figure 9 It runs on the computer device shown. The computer device's memory can store the various program modules that make up the sampling device, for example, Figure 8 The acquisition module, processing module, and identification module are shown. The computer program, comprised of these modules, causes the processor to execute the steps of the data processing methods in the various embodiments of this application described in this specification.

[0121] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps:

[0122] Obtain a target disparity map, which is obtained by calculating the V disparity of the image of the obstacle to be identified;

[0123] Obstacle detection is performed on the target disparity map to obtain an obstacle region, wherein the obstacle region includes at least one obstacle point;

[0124] Obstacle identification is performed based on at least one obstacle point in the obstacle region to obtain the corresponding target obstacle.

[0125] In one embodiment, the target disparity map includes a target straight line function obtained by projecting the obstacle image longitudinally, and the obstacle detection on the target disparity map to obtain the obstacle region includes:

[0126] Obtain the x-coordinate and y-coordinate values ​​of each pixel in the target disparity map;

[0127] Based on the target linear function and the x and y coordinates of each pixel in the target disparity map, obstacle detection is performed on the target disparity map to obtain the obstacle region.

[0128] In one embodiment, the obstacle detection process based on the target linear function and the x-coordinate and y-coordinate values ​​of each pixel in the target disparity map to obtain the obstacle region includes:

[0129] The x-coordinate value of each pixel in the target disparity map is substituted into the target linear function to calculate the predicted y-coordinate value of each pixel.

[0130] All pixels in the target disparity map that meet the first preset condition are identified as obstacle points. The first preset condition includes that the predicted ordinate of the pixel is greater than the ordinate value of the pixel, or that the difference between the predicted ordinate of the pixel and the ordinate value of the pixel is greater than a first threshold.

[0131] The obstacle region is obtained based on the obstacle points.

[0132] In one embodiment, the obstacle detection process based on the target linear function and the x-coordinate and y-coordinate values ​​of each pixel in the target disparity map to obtain the obstacle region includes:

[0133] The predicted horizontal value of each pixel is obtained by substituting the ordinate value of each pixel in the target disparity map into the target linear function.

[0134] All pixels in the target disparity map that meet the first preset condition are identified as obstacle points. The first preset condition includes that the predicted horizontal value of the pixel is greater than the horizontal coordinate value of the pixel, or that the difference between the predicted horizontal value of the pixel and the horizontal coordinate value of the pixel is greater than a second threshold.

[0135] The obstacle region is obtained based on the obstacle points.

[0136] In one embodiment, the obstacle identification based on the obstacle region, resulting in an identification result, includes:

[0137] The obstacle recognition model is invoked to identify obstacles in the obstacle region, thereby obtaining the corresponding target obstacles;

[0138] The obstacle recognition model is obtained by training the model to be trained using a training sample set. The training sample set includes at least one pre-selected anchor box and at least one pre-selected obstacle, with the pre-selected obstacle located within the pre-selected anchor box.

[0139] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0140] The training sample set is used to perform the current batch training on the model to be trained in order to update the parameters of the model to be trained.

[0141] The average classification accuracy of the current batch training is calculated by performing a classification average accuracy calculation on all the pre-selected anchor boxes used in the current batch training and all previous batch training.

[0142] If the difference between the average accuracy of the current iteration and the average accuracy of the previous iteration is greater than or equal to the third threshold, then the process of performing batch training on the model to be trained using the training sample set is repeated until the difference between the average accuracy of the current iteration and the average accuracy of the previous iteration is less than the third threshold, at which point the obstacle recognition model is output.

[0143] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0144] Obtain a target disparity map, which is obtained by calculating the V disparity of the image of the obstacle to be identified;

[0145] Obstacle detection is performed on the target disparity map to obtain an obstacle region, wherein the obstacle region includes at least one obstacle point;

[0146] Obstacle identification is performed based on at least one obstacle point in the obstacle region to obtain the corresponding target obstacle.

[0147] In one embodiment, the target disparity map includes a target straight line function obtained by projecting the obstacle image longitudinally, and the obstacle detection on the target disparity map to obtain the obstacle region includes:

[0148] Obtain the x-coordinate and y-coordinate values ​​of each pixel in the target disparity map;

[0149] Based on the target linear function and the x and y coordinates of each pixel in the target disparity map, obstacle detection is performed on the target disparity map to obtain the obstacle region.

[0150] In one embodiment, the obstacle detection process based on the target linear function and the x-coordinate and y-coordinate values ​​of each pixel in the target disparity map to obtain the obstacle region includes:

[0151] The x-coordinate value of each pixel in the target disparity map is substituted into the target linear function to calculate the predicted y-coordinate value of each pixel.

[0152] All pixels in the target disparity map that meet the first preset condition are identified as obstacle points. The first preset condition includes that the predicted ordinate of the pixel is greater than the ordinate value of the pixel, or that the difference between the predicted ordinate of the pixel and the ordinate value of the pixel is greater than a first threshold.

[0153] The obstacle region is obtained based on the obstacle points.

[0154] In one embodiment, the obstacle detection process based on the target linear function and the x-coordinate and y-coordinate values ​​of each pixel in the target disparity map to obtain the obstacle region includes:

[0155] The predicted horizontal value of each pixel is obtained by substituting the ordinate value of each pixel in the target disparity map into the target linear function.

[0156] All pixels in the target disparity map that meet the first preset condition are identified as obstacle points. The first preset condition includes that the predicted horizontal value of the pixel is greater than the horizontal coordinate value of the pixel, or that the difference between the predicted horizontal value of the pixel and the horizontal coordinate value of the pixel is greater than a second threshold.

[0157] The obstacle region is obtained based on the obstacle points.

[0158] In one embodiment, the step of identifying the obstacle based on the obstacle region to obtain the corresponding target obstacle includes:

[0159] The obstacle recognition model is invoked to identify obstacles in the obstacle region, thereby obtaining the corresponding target obstacles;

[0160] The obstacle recognition model is obtained by training the model to be trained using a training sample set. The training sample set includes at least one pre-selected anchor box and at least one pre-selected obstacle, with the pre-selected obstacle located within the pre-selected anchor box.

[0161] In one embodiment, iteratively training the model to be trained using the training sample set to obtain the obstacle recognition model includes:

[0162] The training sample set is used to perform batch training on the model to be trained in the current batch, so as to update the parameters of the model to be trained;

[0163] The average classification accuracy of the current batch training is calculated by performing a classification average accuracy calculation on all the pre-selected anchor boxes used in the current batch training and all previous batch training.

[0164] If the difference between the average accuracy of the current iteration and the average accuracy of the previous iteration is greater than or equal to the third threshold, then the process of performing batch training on the model to be trained using the training sample set is repeated until the difference between the average accuracy of the current iteration and the average accuracy of the previous iteration is less than the third threshold, at which point the obstacle recognition model is output.

[0165] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0166] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0167] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0168] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0169] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.

[0170] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0171] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0172] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0173] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0174] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0175] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0176] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0177] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An obstacle recognition method, characterized in that, The method includes: Obtain a target disparity map, which is obtained by calculating the V disparity of the image of the obstacle to be identified; Obstacle detection is performed on the target disparity map to obtain an obstacle region, wherein the obstacle region includes at least one obstacle point; Obstacle identification is performed based on at least one obstacle point in the obstacle region to obtain the corresponding target obstacle; The target disparity map includes a target straight line function obtained by projecting the obstacle image longitudinally; the obstacle detection performed on the target disparity map to obtain the obstacle region includes: Obtain the x-coordinate and y-coordinate values ​​of each pixel in the target disparity map; The x-coordinate value of each pixel in the target disparity map is substituted into the target linear function to calculate the predicted y-coordinate value of each pixel. All pixels in the target disparity map that meet the first preset condition are identified as obstacle points. The first preset condition includes that the predicted ordinate of the pixel is greater than the ordinate value of the corresponding pixel, or that the difference between the predicted ordinate of the pixel and the ordinate value of the pixel is greater than a first threshold. Based on the obstacle points, the obstacle region is obtained; The process of obtaining the obstacle region based on the obstacle points includes: Based on the horizontal and vertical coordinates of the obstacle points and the positional relationship between the obstacle points, determine whether the obstacle points are located in the same area. If the obstacle points are located within the same area, the area in the obstacle image corresponding to the same area is defined as the obstacle area.

2. The method according to claim 1, characterized in that, The obstacle identification based on the obstacle region to obtain the corresponding target obstacle includes: The obstacle recognition model is invoked to identify obstacles in the obstacle region, thereby obtaining the corresponding target obstacles; The obstacle recognition model is obtained by training the model to be trained using a training sample set. The training sample set includes at least one pre-selected anchor frame and at least one pre-selected obstacle, with the pre-selected obstacle located within the pre-selected anchor frame.

3. The method according to claim 2, characterized in that, The method for generating the obstacle recognition model includes: The training sample set is used to perform the current batch training on the model to be trained in order to update the parameters of the model to be trained. The average classification accuracy of the current batch training is calculated by performing a classification average accuracy calculation on all the pre-selected anchor boxes used in the current batch training and all previous batch training. If the difference between the average accuracy of the current iteration and the average accuracy of the previous iteration is greater than or equal to the third threshold, then the process of performing batch training on the model to be trained using the training sample set is repeated until the difference between the average accuracy of the current iteration and the average accuracy of the previous iteration is less than the third threshold, at which point the obstacle recognition model is output.

4. An obstacle recognition device, characterized in that, The device includes: The acquisition module is used to acquire a target disparity map, which is obtained by performing V-disparity calculation on the obstacle image to be identified. The target disparity map includes a target straight line function obtained by projecting the obstacle image in the longitudinal direction. The processing module is used to perform obstacle detection on the target disparity map to obtain an obstacle region, wherein the obstacle region includes at least one obstacle point; The identification module is used to identify obstacles based on at least one obstacle point in the obstacle area to obtain the corresponding target obstacle; The processing module is specifically used to: obtain the x-coordinate and y-coordinate values ​​of each pixel in the target disparity map; substitute the x-coordinate value of each pixel in the target disparity map into the target line function to calculate the predicted y-coordinate value of each pixel; determine all pixels in the target disparity map that satisfy a first preset condition as obstacle points, wherein the first preset condition includes that the predicted y-coordinate value of the pixel is greater than the corresponding y-coordinate value of the pixel, or that the difference between the predicted y-coordinate value and the y-coordinate value of the pixel is greater than a first threshold; and obtain the obstacle region based on the obstacle points. The processing module is specifically used to determine whether the obstacle points are located in the same region based on the horizontal and vertical coordinates of the obstacle points and the positional relationship between the obstacle points; if the obstacle points are located in the same region, the region in the obstacle image corresponding to the same region is determined as the obstacle region.

5. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 3.

7. A computer program product having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 3.

Images