A vehicle scene image processing method, system and electronic device

Abstract

The application discloses a vehicle scene image processing method, system and electronic equipment, which can efficiently and accurately detect and identify a vehicle target. The method comprises the following steps: collecting a basic scene image, and generating training sample data according to the basic scene image; performing feature extraction and target prediction on the training sample data by using a full convolution single-stage neural network to generate a predicted target and corresponding prediction information; calculating and determining a network loss function according to the prediction information, and optimizing the full convolution single-stage neural network according to the network loss function. The system comprises a training sample preparation unit, a sample feature extraction unit, a target information prediction unit and a neural network optimization unit. The electronic equipment comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor to implement the vehicle scene image processing method.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, specifically to a method, system, and electronic device for processing vehicle scene images. Background Technology

[0002] With the continuous development of artificial intelligence technology, related autonomous driving and assisted driving systems have also been widely developed and applied. The most critical component in autonomous driving and assisted driving systems is the perception system, which utilizes various onboard sensors to identify and understand the vehicle's surrounding environment. In vision-based perception systems, it is necessary to accurately and quickly identify and locate vehicle targets around and in front of the vehicle based on scene images, providing a data foundation for path planning.

[0003] Some related technologies employ a two-stage neural network detection framework. This involves first detecting the region of interest (ROI) in the image, then extracting vehicle targets within this region. Following this, a keypoint detection model is applied to predict the vehicle's 3D keypoints, and a post-processing algorithm is used to calculate and complete missing points. However, this approach suffers from high algorithmic complexity, significantly increasing computational costs and time consumption in complex multi-target scenarios. Consequently, the timeliness of this solution fails to meet the deployment requirements for real-time computing. Summary of the Invention

[0004] In view of this, embodiments of this application provide a vehicle scene image processing method, system, and electronic device to solve the problem of poor timeliness of target recognition in perception systems in multi-target scenes.

[0005] In a first aspect, embodiments of this application provide a vehicle scene image processing method, the method comprising:

[0006] Acquire basic scene images, determine the three-dimensional information of vehicle targets in the basic scene images, and combine the basic scene images and the three-dimensional information to generate training sample data;

[0007] The training sample data is used as input to the feature extraction network in a fully convolutional single-stage neural network. The feature extraction network is used to extract features from the training sample data to generate a feature image.

[0008] The feature image is used as the input to the prediction network in the fully convolutional single-stage neural network. The prediction network is used to predict the target based on the feature image, determine the prediction target, and output the corresponding prediction information. The prediction information includes the prediction category information and prediction three-dimensional information of the prediction target.

[0009] The predicted information is compared with the training sample data to calculate and determine the network loss function of the fully convolutional single-stage neural network, and the fully convolutional single-stage neural network is optimized based on the network loss function.

[0010] Optionally, the training sample data may also include regression box information of the vehicle target;

[0011] The prediction network is equipped with a category output channel, a three-dimensional output channel, a regression output channel, and a center output channel;

[0012] The step of using the prediction network to predict the target based on the feature image, determining the predicted target, and outputting corresponding prediction information includes:

[0013] The predicted category information of the predicted target is output using the category output channel;

[0014] The predicted three-dimensional information of the predicted target is output using the three-dimensional output channel;

[0015] The predicted regression box information of the predicted target is output using the regression output channel;

[0016] The prediction centrality information of the prediction target is output using the central output channel;

[0017] The prediction information also includes the prediction regression box information and the prediction centrality information.

[0018] Optionally, the predicted 3D information includes the relative distances between multiple predicted key points and the corresponding feature points of the vehicle target in the feature image.

[0019] Optionally, comparing the predicted information with the training sample data to calculate and determine the network loss function of the fully convolutional single-stage neural network includes:

[0020] Based on the training sample data, the corresponding category loss function, 3D loss function, regression loss function and center loss function are calculated and determined for the predicted category information, the predicted 3D information, the predicted regression box information and the predicted centrality information, respectively.

[0021] The network loss function is calculated and determined by combining the category loss function, the center loss function, the three-dimensional loss function, and the regression loss function.

[0022] Optionally, the step of calculating and determining the corresponding category loss function, 3D loss function, regression loss function, and center loss function based on the training sample data for the predicted category information, the predicted 3D information, the predicted regression box information, and the predicted centrality information includes:

[0023] The category loss function corresponding to the predicted category information is calculated using the focus loss function;

[0024] The smooth distance loss function is used to calculate the 3D loss function and the regression loss function corresponding to the predicted 3D information and the predicted regression box information, respectively.

[0025] The centrality loss function corresponding to the predicted centrality information is calculated using the binary cross-entropy loss function.

[0026] Optionally, optimizing the fully convolutional single-stage neural network based on the network loss function includes:

[0027] The network parameters of the fully convolutional single-stage neural network are iteratively optimized and adjusted using gradient backpropagation based on the network loss function.

[0028] Optionally, the method further includes:

[0029] Acquire the scene image to be tested and input the scene image to be tested into the optimized fully convolutional single-stage neural network;

[0030] Multiple candidate targets and corresponding prediction information are determined based on the output of the fully convolutional single-stage neural network.

[0031] Based on the predicted information, multiple candidate targets are filtered to determine multiple vehicle targets and their corresponding three-dimensional information in the scene image to be tested.

[0032] Optionally, the step of filtering the multiple candidate targets based on the prediction information includes:

[0033] Based on the predicted category information, the multiple candidate targets are classified, and non-vehicle category targets among the multiple candidate targets are filtered out.

[0034] The class confidence of multiple candidate targets is calculated and determined based on the predicted class information and the predicted centrality information;

[0035] Based on the category confidence, a non-maximum suppression algorithm is used to filter out duplicates from multiple candidate targets.

[0036] In a second aspect, embodiments of this application also provide a vehicle scene image processing system, the system comprising:

[0037] The training sample preparation unit is used to acquire basic scene images, determine the three-dimensional information of vehicle targets in the basic scene images, and combine the basic scene images and the three-dimensional information to generate training sample data.

[0038] The sample feature extraction unit is used to take the training sample data as input to the feature extraction network in the fully convolutional single-stage neural network, and use the feature extraction network to extract features from the training sample data to generate a feature image.

[0039] A target information prediction unit is configured to use the feature image as input to the prediction network in the fully convolutional single-stage neural network, utilize the prediction network to predict the target based on the feature image, determine the predicted target, and output corresponding prediction information. The prediction information includes the predicted category information and predicted 3D information of the predicted target.

[0040] The neural network optimization unit is used to compare the prediction information with the training sample data, calculate and determine the network loss function of the fully convolutional single-stage neural network, and optimize the fully convolutional single-stage neural network according to the network loss function.

[0041] The system is used to perform the vehicle scene image processing method as described in the first aspect.

[0042] In a third aspect, embodiments of this application also provide a vehicle scene image processing electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the vehicle scene image processing method as described in the first aspect.

[0043] As can be seen from the above, the vehicle scene image processing method, system, and electronic device provided in this application have the following beneficial technical effects:

[0044] (1) A fully convolutional single-stage neural network is used as the detection framework. During the model training phase, the basic scene image and corresponding 3D information are used as input data, and the corresponding prediction target and corresponding prediction 3D information are set as outputs to train and optimize the fully convolutional single-stage neural network. The end-to-end method of training the neural network model directly outputs the 3D information of the vehicle target, which can avoid the suboptimal problem and achieve effective and stable optimization of the neural network model. Thus, when using the optimized fully convolutional single-stage neural network for target recognition in the target recognition phase, the accuracy of the vehicle target recognition result can be guaranteed. In the target recognition phase, when recognizing the vehicle target, only a two-dimensional vehicle scene image is needed as the input data of the fully convolutional single-stage neural network, which can directly output the 3D information of the predicted target. This method eliminates the intermediate process and post-processing steps of converting from 2D information to 3D information, which can greatly simplify the complexity of the method and still ensure the execution efficiency of the method when facing complex multi-target scenes, meeting the timeliness requirements.

[0045] (2) The training sample data also includes the regression box information of the vehicle target, and the corresponding output prediction information also includes the prediction regression box information of the predicted target. Adding regression box information to the training sample data can increase the amount of input information of the feature extraction network, making the feature image containing richer feature information. When the prediction network makes predictions based on the feature image, its prediction results are more accurate. Adding the prediction regression box information to the prediction information output by the prediction network, the prediction regression box information corresponds to the regression box information in the training sample data, and the comparison result between the two can serve as another indicator to measure the optimization effect of the fully convolutional single-stage neural network. Therefore, by adding the prediction regression box information to the prediction information, the training optimization effect of the fully convolutional single-stage neural network can be improved, making the target recognition result of the fully convolutional single-stage neural network more accurate.

[0046] Furthermore, prediction centrality information is added to the prediction information. This prediction centrality information can effectively suppress the occurrence of low-quality predicted targets, ensuring the target prediction performance of the prediction network and improving the target recognition performance of the fully convolutional single-stage neural network. Similar to the effect of the prediction regression box information, the prediction centrality information can also serve as another indicator to measure the optimization effect of the fully convolutional single-stage neural network, further improving the training optimization effect of the fully convolutional single-stage neural network.

[0047] (3) The prediction network uses the relative distances between multiple predicted keypoints and the corresponding feature points of the vehicle target in the feature image as the predicted 3D information. This is then combined with the coordinates of the corresponding feature points of the vehicle target to determine the coordinate information of the multiple predicted keypoints, instead of directly outputting the coordinate data of the multiple predicted keypoints. The prediction network avoids directly outputting the absolute coordinates of the multiple predicted keypoints by outputting the relative distances from the predicted keypoints to the feature points. This approach allows for effective training and optimization, ensuring that the determined coordinates of the predicted keypoints are more stable.

[0048] (4) The network parameters of the fully convolutional single-stage neural network are iteratively optimized and adjusted using gradient backpropagation based on the network loss function. Specifically, when calculating the network loss function, appropriate function algorithms are selected to calculate the loss function for the predicted category information, the predicted 3D information, the predicted bounding box information, and the predicted centrality information, respectively, and then the overall network loss function of the fully convolutional single-stage neural network is calculated. This method of iteratively optimizing the network parameters of the fully convolutional single-stage neural network provides iterative efficiency while ensuring the accuracy of target recognition by the optimized fully convolutional single-stage neural network.

[0049] (5) The optimized fully convolutional single-stage neural network is used to process the scene image under test, and multiple candidate targets in the scene image under test are determined. Further, the multiple candidate targets are filtered according to the corresponding prediction information to ensure the accuracy of the target category and perform deduplication operation. Finally, multiple independent vehicle targets and their corresponding three-dimensional information can be accurately determined. Attached Figure Description

[0050] The features and advantages of this application will be more clearly understood by referring to the accompanying drawings, which are illustrative and should not be construed as limiting the application in any way. In the drawings:

[0051] Figure 1 This illustration shows a schematic diagram of a vehicle scene image processing method provided by one or more optional embodiments of this application;

[0052] Figure 2 This illustration shows a schematic diagram of multiple three-dimensional key points of a vehicle target in a vehicle scene image processing method provided by one or more optional embodiments of this application;

[0053] Figure 3 This illustration shows a method for determining a prediction target and outputting corresponding prediction information in a vehicle scene image processing method provided by one or more optional embodiments of this application;

[0054] Figure 4 This illustration shows a method for calculating and determining a network loss function in a vehicle scene image processing method provided by one or more optional embodiments of this application;

[0055] Figure 5 This illustration shows a schematic diagram of a method for target detection and recognition in a vehicle scene image processing method provided by one or more optional embodiments of this application;

[0056] Figure 6 This illustration shows a method for filtering multiple candidate targets in a vehicle scene image processing method provided by one or more optional embodiments of this application;

[0057] Figure 7 This invention illustrates a schematic diagram of the structure of a vehicle scene image processing system provided by one or more optional embodiments of this application;

[0058] Figure 8 The diagram illustrates a structural schematic of a vehicle scene image processing electronic device provided by one or more optional embodiments of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] With the continuous development of artificial intelligence technology, related autonomous driving and assisted driving systems have also been widely developed and applied. The most critical component in these systems is the perception system, which utilizes various onboard sensors to identify and understand the vehicle's surrounding environment. In vision-based perception systems, it is necessary to accurately and quickly identify and locate vehicle targets around and in front of the vehicle based on scene images, plan a safe and feasible driving route, and avoid collisions with other vehicles. Therefore, how to perform real-time and accurate 3D recognition and detection of vehicle targets is a crucial issue in autonomous driving and assisted driving systems.

[0061] Currently, among methods for vehicle target detection using in-vehicle equipment, the method based on a 2D forward-looking monocular camera is the most widely used due to its low cost and rich information content. Some related technologies employ a two-stage detection framework: first, the region of interest (ROI) of the vehicle target is detected in the image; then, the vehicle target is extracted from this region; a keypoint detection model is applied to the extracted target to predict the vehicle's 3D keypoints; finally, a post-processing algorithm is used to calculate and complete missing points. This approach has high algorithmic complexity, and its computational consumption and time consumption increase significantly when dealing with complex multi-target scenes, making it difficult to meet the deployment requirements for real-time computing. Other related technologies rely on LiDAR point cloud data. These methods require preprocessing of the point cloud data and extraction of target vehicle information to achieve 3D vehicle detection. Therefore, manual annotation of the point cloud data collected by LiDAR is necessary, which is costly and time-consuming. These methods have high execution costs, large computational loads, and poor practicality.

[0062] To address the aforementioned issues, this application proposes a vehicle scene image processing method, system, and electronic device. It employs a fully convolutional single-stage neural network as the detection model. During the model training and optimization phase, a two-dimensional vehicle scene image and corresponding three-dimensional information are used as input, and the predicted three-dimensional information of the vehicle target is used as the output. In the target recognition phase, only the two-dimensional vehicle scene image is needed as input data to the fully convolutional single-stage neural network, which can directly output the predicted three-dimensional information of the target. This method is more direct, has lower hardware costs, and can achieve accurate and efficient recognition of vehicle targets, facilitating widespread deployment and application.

[0063] The technical solution of this application will be described below with reference to specific embodiments.

[0064] In a first aspect, embodiments of this application provide a method for processing vehicle scene images.

[0065] like Figure 1 As shown, one or more optional embodiments of this application provide a vehicle scene image processing method, including:

[0066] S1: Acquire basic scene images, determine the three-dimensional information of vehicle targets in the basic scene images, and combine the basic scene images and the three-dimensional information to generate training sample data.

[0067] The base scene image can be an image of the environment surrounding the vehicle from any direction and angle. It can be collected from images captured by onboard cameras located at different parts of the vehicle, and the base scene image is a two-dimensional planar image. Taking the scene in front of the vehicle as an example, images captured by the onboard forward-looking monocular camera are collected as the base scene image. Generally, onboard cameras capture high frame rate video, and multiple images containing the vehicle target can be periodically selected from the video stream data at certain frame intervals as the base scene image. This approach balances the total amount of data with data diversity.

[0068] For each of the aforementioned basic scene images, multiple three-dimensional key points (V1, V2, V3, ..., V) of the vehicle target can be annotated in the two-dimensional image using a key point annotation method. n ) as the three-dimensional information. For example, Figure 2 As shown, in some optional embodiments, eight vertices of the cubic space corresponding to the vehicle target can be selected as eight three-dimensional key points (V1, V2, V3, V4, ..., V8) of the vehicle target. The cubic space exactly encloses the vehicle target, and multiple faces of the cubic space are tangent to multiple sides of the vehicle target. The length, width, and height of the vehicle target are the length, width, and height of the cubic space. Optionally, the center points of multiple edges of the cubic space can also be selected as the three-dimensional key points.

[0069] In some alternative embodiments, multiple sides of the vehicle target can be identified and determined. For each side, multiple points are selected at equal intervals along its contour edge as the three-dimensional key points. These multiple three-dimensional key points from the multiple sides are then used as the three-dimensional information.

[0070] S2: The training sample data is used as the input to the feature extraction network in the fully convolutional single-stage neural network. The feature extraction network is used to extract features from the training sample data to generate a feature image.

[0071] The training sample data includes the base scene image and the corresponding 3D information. The 3D information, as a label for the base scene image, is input into the feature extraction network backbone. Feature extraction is performed through the feedforward of the feature extraction network backbone to generate the corresponding feature image.

[0072] The feature extraction network backbone can be a ResNet network, MobileNet network, or ShuffleNet network, etc. The feature extraction network backbone has multiple feature extraction channels, and the feature image F is finally output through these channels. The feature image F includes H × W feature points, where H and W represent the height and width of the feature image, respectively.

[0073] S3: The feature image is used as the input to the prediction network in the fully convolutional single-stage neural network. The prediction network is used to predict the target based on the feature image, determine the prediction target, and output the corresponding prediction information. The prediction information includes the prediction category information and prediction three-dimensional information of the prediction target.

[0074] The prediction network head predicts a target based on multiple feature points on the feature image, determines whether the predicted target is located at the positions of the H×W feature points, and outputs the predicted 3D information of the target accordingly. Corresponding to the 3D information in the training sample data, the predicted 3D information may include multiple predicted 3D keypoints.

[0075] S4: Compare the predicted information with the training sample data, calculate and determine the network loss function of the fully convolutional single-stage neural network, and optimize the fully convolutional single-stage neural network according to the network loss function.

[0076] The predicted information is compared with the training sample data, and the network loss function of the fully convolutional single-stage neural network is calculated based on the difference between the two. The network loss function is used to measure the accuracy of the prediction result of the fully convolutional single-stage neural network. Multiple training sample data are sequentially input into the fully convolutional single-stage neural network, and the corresponding network loss function is calculated based on its output. Then, the network parameters of the fully convolutional single-stage neural network are continuously adjusted and optimized based on the network loss function until the network loss function meets the preset threshold requirement.

[0077] When optimizing the fully convolutional single-stage neural network according to the network loss function, the network parameters of the fully convolutional single-stage neural network can be iteratively optimized and adjusted by gradient backpropagation.

[0078] After the fully convolutional single-stage neural network is optimized, it can be used to process the test scene image. Using only the two-dimensional test scene image as the neural network input, it can quickly and accurately identify multiple vehicle targets and their corresponding three-dimensional information in the test scene image.

[0079] The described vehicle scene image processing method employs a fully convolutional single-stage neural network (LCN) as the detection framework. During model training, the basic scene image and corresponding 3D information are used as input data, and the predicted target and corresponding predicted 3D information are set as outputs to train and optimize the LCN. This end-to-end training of the neural network model directly outputs the 3D information of the vehicle target, avoiding secondary optimization problems and achieving effective and stable optimization of the neural network model. Therefore, in the target recognition stage, when using the optimized LCN for target recognition, the accuracy of the vehicle target recognition results is guaranteed. In the target recognition stage, only a 2D vehicle scene image is needed as input data for the LCN, which can directly output the predicted target's 3D information. This approach eliminates the intermediate process and post-processing steps of converting 2D information to 3D information, greatly simplifying the method's complexity. Even in complex multi-target scenes, the method maintains execution efficiency and meets timeliness requirements.

[0080] In a vehicle scene image processing method provided in one or more optional embodiments of this application, the training sample data further includes the regression box information of the vehicle target.

[0081] To achieve accurate localization and detection of vehicle targets, a bounding box for the vehicle target can be determined in the base scene image. The requirement is that the generated bounding box should completely encompass the vehicle target. In some optional embodiments, the bounding box for the vehicle target can be determined based on the coordinate data of multiple 3D key points of the vehicle target.

[0082]

[0083] Wherein, B represents the regression box information. These represent the horizontal and vertical coordinates of the i-th 3D key point, respectively. Based on this, the coordinates of the lower left and upper right corners of the regression box can be determined, and the regression box information is obtained using the coordinates of its lower left and upper right corners.

[0084] In some alternative embodiments, the prediction network is provided with a category output channel, a 3D output channel, a regression output channel, and a center output channel.

[0085] like Figure 3 As shown, the step of using the prediction network to predict the target based on the feature image, determining the predicted target, and outputting corresponding prediction information includes:

[0086] S301: Output the predicted category information of the predicted target using the category output channel.

[0087] S302: Output the predicted three-dimensional information of the predicted target using the three-dimensional output channel.

[0088] S303: Output the predicted regression box information of the predicted target using the regression output channel.

[0089] S304: Output the prediction centrality information of the prediction target using the central output channel.

[0090] The prediction information also includes the prediction regression box information and the prediction centrality information.

[0091] The centrality information defines the positional relationship between the feature point location on the feature image and the corresponding vehicle target bounding box. The centrality value ranges from 0 to 1, where 1 represents that the feature point is located at the exact center of the vehicle target bounding box, and 0 represents that the feature point is located at the edge of the vehicle target bounding box.

[0092] The prediction network includes four convolutional layers, which correspond to the category output channel, the 3D output channel, the regression output channel, and the center output channel, respectively. The predicted category information, the predicted 3D information, the predicted regression box information, and the predicted centrality information are output through the category output channel, the 3D output channel, the regression output channel, and the center output channel.

[0093] In the vehicle scene image processing method, the training sample data also includes the bounding box information of the vehicle target, and the corresponding output prediction information also includes the predicted bounding box information of the predicted target. Adding bounding box information to the training sample data increases the amount of input information to the feature extraction network, making the feature image richer in feature information. When the prediction network makes predictions based on the feature image, its prediction results are more accurate. Adding the predicted bounding box information to the prediction information output by the prediction network, which corresponds to the bounding box information in the training sample data, allows for a comparison that can serve as another indicator of the optimization effect of the fully convolutional single-stage neural network. Therefore, by adding the predicted bounding box information to the prediction information, the training optimization effect of the fully convolutional single-stage neural network can be improved, making the target recognition results of the fully convolutional single-stage neural network more accurate.

[0094] The prediction centrality information is further added to the prediction information. This prediction centrality information can effectively suppress the occurrence of low-quality prediction bounding boxes, ensuring the target prediction performance of the prediction network and improving the target recognition performance of the fully convolutional single-stage neural network. Similar to the effect of the prediction bounding box information, the prediction centrality information can also serve as another indicator to measure the optimization effect of the fully convolutional single-stage neural network, further improving the training and optimization effect of the fully convolutional single-stage neural network.

[0095] To make the predicted 3D keypoint coordinates more stable, the prediction network can be set to predict the relative distance between the 3D keypoints and feature points to avoid directly predicting the absolute coordinates of the 3D keypoints.

[0096] In a vehicle scene image processing method provided in one or more optional embodiments of this application, the predicted three-dimensional information includes the relative distances between multiple predicted key points and the corresponding feature points of the vehicle target in the feature image.

[0097] For each 3D key point of the target vehicle, we use:

[0098]

[0099] To learn the relative distance from 3D keypoints to the corresponding feature point coordinates of the target vehicle on the feature image, where x center y center This refers to the feature point on the feature image F that is closest to the center of the vehicle target. The 3D output channel outputs the relative distances (Δ) between multiple 3D key points. x ,Δ y Finally, add the coordinates (x, y) of the feature point corresponding to the target vehicle. center ,y center This allows us to determine the position of the predicted 3D keypoints in the image coordinate system.

[0100] The prediction network uses the relative distances between multiple predicted keypoints and their corresponding feature points in the feature image as the predicted 3D information. This is then combined with the coordinates of the corresponding feature points of the vehicle target to determine the coordinates of the multiple predicted keypoints, rather than directly outputting the coordinate data of the multiple predicted keypoints. The prediction network avoids directly outputting the absolute coordinates of the multiple predicted keypoints by outputting the relative distances from the predicted keypoints to the feature points. This approach allows for effective training and optimization, ensuring that the determined coordinates of the predicted keypoints are more stable.

[0101] like Figure 4As shown, in a vehicle scene image processing method provided by one or more optional embodiments of this application, the step of comparing the predicted information with the training sample data and calculating and determining the network loss function of the fully convolutional single-stage neural network includes:

[0102] S401: Based on the training sample data, calculate and determine the corresponding category loss function, 3D loss function, regression loss function, and center loss function for the predicted category information, the predicted 3D information, the predicted regression box information, and the predicted centrality information, respectively.

[0103] For the predicted category information, the corresponding category loss function can be calculated using the Focal loss (FL) function.

[0104]

[0105] Wherein, FL(p) represents the category loss function, p∈[0,1] represents the probability that the predicted target output by the prediction network is a vehicle, and y∈{1,-1} represents the vehicle target category and the background category. α and γ are adjustment parameters of the loss function. The specific data of the adjustment parameters can be flexibly adjusted according to the actual situation. In some optional embodiments, α can be set to 0.25 and γ to 2.

[0106] For the predicted 3D information, the corresponding 3D loss function can be calculated using the Smooth L1 loss function.

[0107]

[0108] Among them, L reg (p r ,t1) represents the three-dimensional loss function, p r t1 represents the relative distance between the vehicle's 3D key points and corresponding feature points determined by the predicted 3D information output by the 3D output channel, and t1 represents the relative distance between the 3D key points and feature points determined by the training sample data.

[0109] For the predicted regression box information, the corresponding regression loss function can be calculated using the smooth distance loss function.

[0110]

[0111] Among them, L reg (p q t2) represents the regression loss function, p q t1 represents the regression quantity determined based on the predicted regression box information output by the regression output channel, and t2 represents the regression quantity determined based on the training sample data.

[0112] For the predicted centrality information, the corresponding centrality loss function can be calculated using the binary cross-entropy loss function.

[0113] L cls (c, p) c ) = E c [-c*log(p c )+(1-c)*log(1-p c )]

[0114] Among them, L cls (c, p) c ) represents the centrality loss function, c represents the centrality determined based on feature point coordinates and bounding box information, and E c It is the mathematical expectation of centrality c, p c This represents the prediction centrality information output by the prediction network.

[0115] S402: Calculate and determine the network loss function by combining the category loss function, the center loss function, the three-dimensional loss function, and the regression loss function.

[0116] The network loss function:

[0117] LOSS=β1FL(p)+β2L reg (p r ,t1)+β3L reg (p q ,t2)+β4L cls (c, p) c )

[0118] Wherein, β1, β2, β3, and β4 represent the adjustment parameters of the network loss function LOSS, which can be flexibly adjusted according to actual conditions. In some optional embodiments, the values ​​of β1, β2, β3, and β4 can all be set to 1, that is, the network loss function is obtained by directly adding the category loss function, the three-dimensional loss function, the regression loss function, and the center loss function.

[0119] In the vehicle scene image processing method, the network parameters of the fully convolutional single-stage neural network are iteratively optimized and adjusted using gradient backpropagation based on the network loss function. Specifically, when calculating the network loss function, appropriate function algorithms are selected to calculate the loss function for the predicted category information, the predicted 3D information, the predicted bounding box information, and the predicted centrality information, respectively, and then the overall network loss function of the fully convolutional single-stage neural network is calculated. This method of iteratively optimizing the network parameters of the fully convolutional single-stage neural network provides iterative efficiency while ensuring the accuracy of target recognition by the optimized fully convolutional single-stage neural network.

[0120] Figure 5 A schematic diagram of a method for target recognition and detection in a scene image under test is shown. Figure 5 As shown, a vehicle scene image processing method provided by one or more optional embodiments of this application further includes:

[0121] S501: Obtain the scene image to be tested, and input the scene image to be tested into the optimized fully convolutional single-stage neural network.

[0122] S502: Determine multiple candidate targets and the corresponding prediction information based on the output of the fully convolutional single-stage neural network.

[0123] S503: Based on the prediction information, multiple candidate targets are screened and filtered to determine multiple vehicle targets and their corresponding three-dimensional information in the scene image to be tested.

[0124] Considering that among the multiple candidate targets output by the fully convolutional single-stage neural network, there may be targets that are not in the vehicle category, they can be filtered based on the corresponding predicted category information. However, for multiple candidate targets belonging to the vehicle category, there may be cases of duplicate identification, requiring further deduplication processing of the multiple candidate targets.

[0125] like Figure 6 In a vehicle scene image processing method provided in one or more optional embodiments of this application, the step of filtering multiple candidate targets based on the prediction information includes:

[0126] S601: Based on the predicted category information, classify the multiple candidate targets and filter out non-vehicle category targets among the multiple candidate targets.

[0127] The prediction category information and prediction centrality information of the candidate target are processed by the nonlinear function Sigmoid, and the output values ​​are all in the range of [0,1].

[0128] The formula for calculating the nonlinear function Sigmoid is as follows:

[0129]

[0130] The corresponding output value S of the predicted category information cls The candidate target is compared with a preset threshold to determine whether it belongs to the vehicle category. Targets that do not belong to the vehicle category are filtered out.

[0131] S602: Calculate and determine the category confidence of multiple candidate targets based on the predicted category information and the predicted centrality information.

[0132] Sig = S cls *ctn

[0133] Among them, S cls ctn and ctn represent the output values ​​of the predicted category information and the predicted centrality information after processing by the nonlinear function sigmoid, respectively. Sig represents the category confidence of the candidate target.

[0134] S603: Based on the category confidence, a non-maximum suppression algorithm is used to filter out duplicates from multiple candidate targets.

[0135] A set W can be constructed using multiple candidate targets after filtering out non-vehicle categories, and a non-maximum suppression algorithm can be used to filter this set. Specific filtering methods include:

[0136] Step 1: Sort the candidate targets in descending order according to their category confidence scores in set W.

[0137] Step 2: Select the candidate target T with the highest confidence level in the category of set W, and filter out targets whose overlap rate with candidate target T is greater than or equal to the preset overlap threshold.

[0138] Step 3: When the overlap rate between other candidate targets in set W and candidate target T is less than the preset overlap rate, remove candidate target T from set W and add it to set W. fin l .

[0139] Step 4: Repeat Steps 2-3 until set W is empty. final The multiple targets included in the data are the vehicle targets that are ultimately identified and determined.

[0140] In the vehicle scene image processing method, the optimized fully convolutional single-stage neural network is used to process the scene image under test, determine multiple candidate targets in the scene image under test, and further filter the multiple candidate targets according to the corresponding prediction information to ensure the accuracy of the target category and perform deduplication operation, so as to accurately determine multiple independent vehicle targets and their corresponding three-dimensional information.

[0141] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0142] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0143] Based on the same inventive objective as the first aspect, in a second aspect, embodiments of this application provide a vehicle scene image processing system.

[0144] like Figure 7 As shown, one or more optional embodiments of this application provide a vehicle scene image processing system, comprising:

[0145] The training sample preparation unit 701 is used to acquire basic scene images, determine the three-dimensional information of vehicle targets in the basic scene images, and combine the basic scene images and the three-dimensional information to generate training sample data.

[0146] The sample feature extraction unit 702 is used to take the training sample data as input to the feature extraction network in the fully convolutional single-stage neural network, and use the feature extraction network to extract features from the training sample data to generate a feature image.

[0147] The target information prediction unit 703 is used to take the feature image as input to the prediction network in the fully convolutional single-stage neural network, use the prediction network to predict the target based on the feature image, determine the predicted target, and output corresponding prediction information. The prediction information includes the predicted category information and predicted 3D information of the predicted target.

[0148] The neural network optimization unit 704 is used to compare the prediction information with the training sample data, calculate and determine the network loss function of the fully convolutional single-stage neural network, and optimize the fully convolutional single-stage neural network according to the network loss function.

[0149] In a vehicle scene image processing system provided in one or more optional embodiments of this application, the training sample data further includes the regression box information of the vehicle target;

[0150] The prediction network is equipped with a category output channel, a three-dimensional output channel, a regression output channel, and a center output channel;

[0151] The target information prediction unit 703 is further configured to output the predicted category information of the predicted target using the category output channel; output the predicted three-dimensional information of the predicted target using the three-dimensional output channel; output the predicted regression box information of the predicted target using the regression output channel; and output the predicted centrality information of the predicted target using the center output channel. The prediction information also includes the predicted regression box information and the predicted centrality information.

[0152] In a vehicle scene image processing system provided by one or more optional embodiments of this application, the predicted 3D information includes the relative distances between multiple predicted key points and the corresponding feature points of the vehicle target in the feature image.

[0153] In a vehicle scene image processing system provided in one or more optional embodiments of this application, the neural network optimization unit 704 is further configured to calculate and determine corresponding category loss functions, 3D loss functions, regression loss functions, and center loss functions based on the training sample data for the predicted category information, the predicted 3D information, the predicted regression box information, and the predicted centrality information; and to calculate and determine the network loss function by combining the category loss function, the center loss function, the 3D loss function, and the regression loss function.

[0154] In a vehicle scene image processing system provided in one or more optional embodiments of this application, the neural network optimization unit 704 is further configured to calculate the category loss function corresponding to the predicted category information using the focus loss function; calculate the 3D loss function and the regression loss function corresponding to the predicted 3D information and the predicted regression box information respectively using the smoothing distance loss function; and calculate the center loss function corresponding to the predicted centrality information using the binary classification cross-entropy loss function.

[0155] In a vehicle scene image processing system provided in one or more optional embodiments of this application, the neural network optimization unit 704 is further configured to iteratively optimize and adjust the network parameters of the fully convolutional single-stage neural network according to the network loss function using gradient backpropagation.

[0156] This application provides a vehicle scene image processing system in one or more optional embodiments, which further includes a vehicle target recognition unit. The vehicle target recognition unit is configured to acquire a scene image to be tested, input the scene image to be tested into an optimized fully convolutional single-stage neural network; determine multiple candidate targets and corresponding prediction information based on the output of the fully convolutional single-stage neural network; and filter the multiple candidate targets based on the prediction information to determine multiple vehicle targets and their corresponding three-dimensional information in the scene image to be tested.

[0157] In a vehicle scene image processing system provided in one or more optional embodiments of this application, the vehicle target recognition unit is further configured to: determine the category of a plurality of candidate targets based on the predicted category information; filter out non-vehicle category targets among the plurality of candidate targets; calculate and determine the category confidence of the plurality of candidate targets based on the predicted category information and the predicted centrality information; and perform deduplication filtering on the plurality of candidate targets based on the category confidence using a non-maximum suppression algorithm.

[0158] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0159] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0160] The system described above is used to implement the corresponding vehicle scene image recognition method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0161] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the vehicle scene image recognition method described in any of the above embodiments.

[0162] Figure 8 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0163] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the vehicle scene image recognition method provided in the method embodiments of this specification.

[0164] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0165] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0166] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0167] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0168] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0169] The electronic devices described above are used to implement the corresponding vehicle scene image recognition methods in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0170] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the vehicle scene image recognition method as described in any of the above embodiments.

[0171] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0172] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the electric field shielding effectiveness calculation method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0173] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0174] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0175] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0176] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A method for processing vehicle scene images, characterized in that, include: Acquire basic scene images, determine the three-dimensional information of vehicle targets in the basic scene images, and combine the basic scene images and the three-dimensional information to generate training sample data; The training sample data is used as input to the feature extraction network in a fully convolutional single-stage neural network. The feature extraction network is used to extract features from the training sample data to generate a feature image. The feature image is used as the input to the prediction network in the fully convolutional single-stage neural network. The prediction network is used to predict the target based on the feature image, determine the prediction target, and output the corresponding prediction information. The prediction information includes the prediction category information and prediction three-dimensional information of the prediction target. The predicted information is compared with the training sample data, the network loss function of the fully convolutional single-stage neural network is calculated and determined, and the fully convolutional single-stage neural network is optimized according to the network loss function. The training sample data also includes the regression bounding box information of the vehicle target; The prediction network is equipped with a category output channel, a three-dimensional output channel, a regression output channel, and a center output channel; The step of using the prediction network to predict the target based on the feature image, determining the predicted target, and outputting corresponding prediction information includes: The predicted category information of the predicted target is output using the category output channel; The predicted three-dimensional information of the predicted target is output using the three-dimensional output channel; The predicted regression box information of the predicted target is output using the regression output channel; The prediction centrality information of the prediction target is output using the central output channel; The prediction information also includes the prediction regression box information and the prediction centrality information; The step of comparing the predicted information with the training sample data and calculating and determining the network loss function of the fully convolutional single-stage neural network includes: Based on the training sample data, the corresponding category loss function, 3D loss function, regression loss function and center loss function are calculated and determined for the predicted category information, the predicted 3D information, the predicted regression box information and the predicted centrality information, respectively. The network loss function is calculated and determined by combining the category loss function, the center loss function, the three-dimensional loss function, and the regression loss function.

2. The method according to claim 1, characterized in that, The predicted 3D information includes the relative distances between multiple predicted key points and the corresponding feature points of the vehicle target in the feature image.

3. The method according to claim 1, characterized in that, The step of calculating and determining the corresponding category loss function, 3D loss function, regression loss function, and center loss function based on the training sample data for the predicted category information, the predicted 3D information, the predicted regression box information, and the predicted centrality information includes: The category loss function corresponding to the predicted category information is calculated using the focus loss function; The smooth distance loss function is used to calculate the 3D loss function and the regression loss function corresponding to the predicted 3D information and the predicted regression box information, respectively. The centrality loss function corresponding to the predicted centrality information is calculated using the binary cross-entropy loss function.

4. The method according to claim 1, characterized in that, The optimization of the fully convolutional single-stage neural network based on the network loss function includes: The network parameters of the fully convolutional single-stage neural network are iteratively optimized and adjusted using gradient backpropagation based on the network loss function.

5. The method according to claim 1, characterized in that, The method further includes: Acquire the scene image to be tested and input the scene image to be tested into the optimized fully convolutional single-stage neural network; Multiple candidate targets and corresponding prediction information are determined based on the output of the fully convolutional single-stage neural network. Based on the predicted information, multiple candidate targets are filtered to determine multiple vehicle targets and their corresponding three-dimensional information in the scene image to be tested.

6. The method according to claim 5, characterized in that, The step of filtering multiple candidate targets based on the prediction information includes: Based on the predicted category information, the multiple candidate targets are classified, and non-vehicle category targets among the multiple candidate targets are filtered out. The class confidence of multiple candidate targets is calculated and determined based on the predicted class information and the predicted centrality information; Based on the category confidence, a non-maximum suppression algorithm is used to filter out duplicates from multiple candidate targets.

7. A vehicle scene image processing system, characterized in that, include: The training sample preparation unit is used to acquire basic scene images, determine the three-dimensional information of vehicle targets in the basic scene images, and combine the basic scene images and the three-dimensional information to generate training sample data. The sample feature extraction unit is used to take the training sample data as input to the feature extraction network in the fully convolutional single-stage neural network, and use the feature extraction network to extract features from the training sample data to generate a feature image. A target information prediction unit is configured to use the feature image as input to the prediction network in the fully convolutional single-stage neural network, utilize the prediction network to predict the target based on the feature image, determine the predicted target, and output corresponding prediction information. The prediction information includes the predicted category information and predicted 3D information of the predicted target. The neural network optimization unit is used to compare the prediction information with the training sample data, calculate and determine the network loss function of the fully convolutional single-stage neural network, and optimize the fully convolutional single-stage neural network according to the network loss function. The training sample data also includes the regression bounding box information of the vehicle target; The prediction network is equipped with a category output channel, a three-dimensional output channel, a regression output channel, and a center output channel; The step of using the prediction network to predict the target based on the feature image, determining the predicted target, and outputting corresponding prediction information includes: The predicted category information of the predicted target is output using the category output channel; The predicted three-dimensional information of the predicted target is output using the three-dimensional output channel; The predicted regression box information of the predicted target is output using the regression output channel; The prediction centrality information of the prediction target is output using the central output channel; The prediction information also includes the prediction regression box information and the prediction centrality information; The step of comparing the predicted information with the training sample data and calculating and determining the network loss function of the fully convolutional single-stage neural network includes: Based on the training sample data, the corresponding category loss function, 3D loss function, regression loss function and center loss function are calculated and determined for the predicted category information, the predicted 3D information, the predicted regression box information and the predicted centrality information, respectively. The network loss function is calculated and determined by combining the category loss function, the center loss function, the three-dimensional loss function, and the regression loss function.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 6.

Images