Road scene target detection method, device, program product and storage medium

Abstract

The present application discloses a target detection method, device, program product and storage medium for road scenes, which are applied to the field of image recognition technology, including: extracting features from a training image through an image feature extraction network of a target detection model, obtaining feature extraction results and inputting them into a height distribution estimation network and a depth distribution estimation network in the target detection model respectively; converting the obtained height distribution estimation result from a height distribution to a depth distribution based on a preset reference image plane with a fixed depth, and fusing the converted first depth distribution result with the depth distribution estimation result; performing post-processing on the fused second depth distribution result based on a post-processing network in the target detection model, obtaining a target detection result for the training image and updating the target detection model parameters after performing loss function calculation. Applying the solution of the present application can reduce the computational burden and increase reliability.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method, device, program product and storage medium for detecting targets in road scenes. Background Art

[0002] Roadside perception technology uses roadside cameras to collect image data of road scenes, and then detect and identify targets such as vehicles and pedestrians. Due to the high installation height, wide perception range and small blind spots of roadside cameras, they play an increasingly important role in the field of smart transportation.

[0003] The accuracy of current roadside perception methods needs to be improved, especially because the size of distant objects in the image will be significantly reduced, so the detection error of distant objects is greater. One current method uses a solution that combines the depth estimation and height estimation branches, which can be seen in Figure 1 , which is a schematic diagram of the BEV (Bird's-Eye View) feature generation process of an existing roadside visual target detection method. After the image feature extraction network performs image feature extraction, the height distribution estimation network and the depth distribution estimation network perform height distribution estimation and depth distribution estimation respectively, and then process them to obtain height branch bird's-eye view features and depth branch bird's-eye view features respectively. Finally, an additional bird's-eye view feature fusion module is used to fuse the two BEV features to obtain fused BEV features. The computation and storage burden of the feature view conversion operation is very large. For a h (Height of image feature) × w (width of image feature)× c (the number of channels of image features) image features, first need to process the depth distribution into h × w × D (the number of predefined depth values) × 1, processing the height distribution into h × w × H (the number of predefined height values) × 1, and then multiply the depth distribution and height distribution with the image features respectively to obtain the shape of h × w × D × c and h × w × H × c The depth distribution and height distribution are then processed into BEV features using the voxel pooling method. Finally, an additional bird's-eye view feature fusion module is used to fuse the two BEV features. The above process requires h × w × c × (H + D ) multiplications, required during voxel pooling h × w × ( H + D ) addition calculations.

[0004] In summary, how to effectively realize target detection in road scenes and reduce the computational burden is a technical problem that those skilled in the art urgently need to solve. Summary of the Invention

[0005] The purpose of the present invention is to provide a method, device, program product and storage medium for detecting objects in road scenes, so as to effectively realize object detection in road scenes and reduce the computational burden.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a method for detecting an object in a road scene, comprising:

[0008] The input training image is subjected to feature extraction through the image feature extraction network of the preset target detection model to obtain the feature extraction result;

[0009] Based on the feature extraction results, a height distribution estimation result and a depth distribution estimation result are obtained using a height distribution estimation network and a depth distribution estimation network respectively;

[0010] Based on a preset reference image plane of a fixed depth, reference information for converting a height distribution into a depth distribution is obtained, and based on the reference information, the height distribution estimation result is converted from a height distribution into a depth distribution using a triangle similarity principle to obtain a converted first depth distribution result;

[0011] fusing the first depth distribution result with the depth distribution estimation result to obtain a fused second depth distribution result;

[0012] Based on the post-processing network in the target detection model, the second depth distribution result is post-processed to obtain the target detection result of the training image, and after the loss function of the target detection model is calculated based on the target detection result, the parameters of the target detection model are updated.

[0013] On the other hand, it also includes:

[0014] Inputting the true height data into a height truth encoding network for estimating height distribution, and obtaining a first height distribution result output by the height truth encoding network;

[0015] Based on the first height distribution result and the height distribution estimation result, obtaining a first loss function for reflecting the difference between the first height distribution result and the height distribution estimation result;

[0016] The height truth data is height truth data used to reflect parameter information of different targets in the training image, and the reflected parameter information includes the height of the center point of the bounding box of the target;

[0017] Accordingly, after calculating the loss function of the target detection model based on the target detection result, updating the parameters of the target detection model includes:

[0018] Calculating a loss function of the target detection model based on the target detection result;

[0019] Based on the loss function of the target detection model and the first loss function, parameters of the target detection model are updated.

[0020] The depth truth data is input into a depth truth encoding network for performing depth distribution estimation, to obtain a third depth distribution result output by the depth truth encoding network;

[0021] obtaining, based on the third depth distribution result and the depth distribution estimation result, a second loss function for reflecting a difference between the third depth distribution result and the depth distribution estimation result;

[0022] The true depth data is used to reflect parameter information of different targets in the training image, and the reflected parameter information includes the depth of the center point of the bounding box of the target;

[0023] Accordingly, after calculating the loss function of the target detection model based on the target detection result, updating the parameters of the target detection model includes:

[0024] Calculating a loss function of the target detection model based on the target detection result;

[0025] Based on the loss function of the target detection model, the first loss function and the second loss function, the parameters of the target detection model are updated.

[0026] On the other hand, it also includes:

[0027] Based on the first loss function, updating the parameters of the high-level truth encoding network;

[0028] Based on the second loss function, updating the parameters of the deep truth encoding network;

[0029] Among them, the high truth value coding network is a high truth value coding network including multiple layers of fully connected layers, and the deep truth value coding network is a deep truth value coding network including multiple layers of fully connected layers.

[0030] On the other hand, the high truth data includes high truth sub-data in the form of multiple groups (z1, z2, z3, z4, z5);

[0031] Among them, for any set of height truth sub-data in the form of (z1, z2, z3, z4, z5), z1 represents the height of the center point of the bounding box of the target corresponding to the set of height truth sub-data, z2 represents the length of the bounding box of the target, z3 represents the width of the bounding box of the target, z4 represents the height of the bounding box of the target, and z5 represents the labeled category of the target;

[0032] The depth truth data includes multiple groups of depth truth sub-data in the form of (z6, z2, z3, z4, z5);

[0033] Among them, for any set of depth truth sub-data in the form of (z6, z2, z3, z4, z5), z6 represents the depth of the center point of the bounding box of the target corresponding to the set of depth truth sub-data.

[0034] On the other hand, the first loss function is expressed as:

[0035] ;

[0036] The second loss function is expressed as:

[0037] ;

[0038] in, L 1 represents the first loss function, L 2 represents the second loss function, n represents the number of targets in the training image, H It represents the number of predefined height values ​​in the height distribution. i and j are all positive integers, P i h ( j ) represents that in the height distribution estimation result, i The height distribution of the target j The probability of the height value, Q i he ( j ) means that in the first height distribution result, i The height distribution of the targetj Probability of height value;

[0039] D It represents the number of predefined depth values ​​in the depth distribution. P i d ( j ) represents that in the depth distribution estimation result, i The depth distribution of the target j The probability of depth value, Q i de ( j ) represents that in the third depth distribution result, i The depth distribution of the target j The probability of depth value.

[0040] On the other hand, based on the loss function of the target detection model, the first loss function and the second loss function, updating the parameters of the target detection model includes:

[0041] Based on the loss function of the target detection model, the first loss function and the second loss function, the total loss is obtained Lc , and based on the total loss Lc Updating parameters of the target detection model;

[0042] in, Lc = L b + k 1× L 1+ k 2× L 2, L b It represents the loss function of the target detection model. k 1 represents the weight coefficient set for the first loss function, k 2 represents the weight coefficient set for the second loss function.

[0043] On the other hand, it also includes:

[0044] Inputting the height distribution estimation result into a height distribution decoding network for performing height value decoding to obtain height value data output by the height distribution decoding network;

[0045] Based on the height value data and predetermined true height data, a third loss function is obtained to reflect the difference between the height value data and the true height data;

[0046] Accordingly, after calculating the loss function of the target detection model based on the target detection result, updating the parameters of the target detection model includes:

[0047] Calculating a loss function of the target detection model based on the target detection result;

[0048] Based on the loss function of the target detection model and the third loss function, parameters of the target detection model are updated.

[0049] On the other hand, it also includes:

[0050] Inputting the depth distribution estimation result into a depth distribution decoding network for performing depth value decoding to obtain depth value data output by the depth distribution decoding network;

[0051] Based on the depth value data and predetermined true depth data, obtaining a fourth loss function for reflecting a difference between the depth value data and the true depth data;

[0052] Accordingly, after calculating the loss function of the target detection model based on the target detection result, updating the parameters of the target detection model includes:

[0053] Calculating a loss function of the target detection model based on the target detection result;

[0054] Based on the loss function of the target detection model, the third loss function and the fourth loss function, the parameters of the target detection model are updated.

[0055] On the other hand, the third loss function is expressed as:

[0056] ;

[0057] The fourth loss function is expressed as:

[0058] ;

[0059] in, L 3 represents the third loss function, L 4 represents the fourth loss function, n represents the number of targets in the training image, h i It represents the first value obtained from the real height data. i The true height of the center point of the target's bounding box, h i ' represents the height value data output by the height distribution decoding network iThe height of the center point of the target's bounding box, d i It represents the first i The true depth of the center point of the target's bounding box, d i ' represents the depth value data output by the depth distribution decoding network i The depth of the object's bounding box center point.

[0060] On the other hand, it also includes:

[0061] Classify each target pointed to by the height distribution estimation result according to the target's labeled category;

[0062] For any one category, determine whether the number of targets in the category exceeds a preset first threshold;

[0063] If not, discard the category;

[0064] If yes, retain the classification;

[0065] For each retained category, The loss corresponding to the classification is determined in this way;

[0066] according to The fifth loss function is obtained by:

[0067] in, L 5 represents the fifth loss function, C It represents the total number of retained categories. a is a positive integer;

[0068] L a Indicates the corresponding a Classification loss, i and j as well as k are all positive integers, H It represents the number of predefined height values ​​in the height distribution. w ik It means the a The first in the classification i The goal and the first k The weights between the targets, M i h ( j ) represents that in the height distribution estimation result, a The first in the classification i The height distribution of the target jThe probability of the height value, M k h ( j ) represents that in the height distribution estimation result, a The first in the classification k The height distribution of the target j Probability of height value;

[0069] and, , d i It means the i The depth value of the target, d k It means the k The depth value of the target, d max It represents the predefined maximum depth value;

[0070] Accordingly, after calculating the loss function of the target detection model based on the target detection result, updating the parameters of the target detection model includes:

[0071] Calculating a loss function of the target detection model based on the target detection result;

[0072] Based on the loss function of the target detection model and the fifth loss function, parameters of the target detection model are updated.

[0073] On the other hand, fusing the first depth distribution result with the depth distribution estimation result to obtain a fused second depth distribution result, including:

[0074] Based on the first depth distribution result and the depth distribution estimation result, a fusion is performed according to a fusion rule of taking an average value to obtain a fused second depth distribution result.

[0075] On the other hand, fusing the first depth distribution result with the depth distribution estimation result to obtain a fused second depth distribution result, including:

[0076] Based on the first depth distribution result and the depth distribution estimation result, for any one target, according to Determine the first confidence level and the second confidence level of the target in a manner;

[0077] Based on the first confidence level and the second confidence level, The fusion weight is obtained by

[0078] based on The first depth distribution result and the depth distribution estimation result are fused according to a fusion rule to obtain a fused second depth distribution result;

[0079] in, C a It represents the first confidence of the target. C b It represents the second confidence of the target. D It represents the number of predefined depth values ​​in the depth distribution. p a ( j ) represents the first depth distribution of the target in the first depth distribution result. j The probability of the height value, p b ( j ) represents the depth distribution of the target in the depth distribution estimation result. j Probability of height value; represents the fusion weight, p ( j ) represents the depth distribution of the target in the second depth distribution result obtained. j The probability of the height value.

[0080] On the other hand, based on the post-processing network in the target detection model, post-processing the second depth distribution result to obtain the target detection result of the training image, and after calculating the loss function of the target detection model based on the target detection result, updating the parameters of the target detection model, including:

[0081] Performing point cloud feature extraction on the second depth distribution result based on a point cloud feature extraction network in a post-processing network in the target detection model to obtain a point cloud feature extraction result;

[0082] Based on the bird's-eye view feature extraction network in the post-processing network, performing bird's-eye view feature extraction on the point cloud feature extraction result to obtain a bird's-eye view feature extraction result;

[0083] Based on the detection network in the post-processing network, target detection is performed on the bird's-eye view feature extraction result to obtain the target detection result of the training image, and after the loss function of the target detection model is calculated based on the target detection result, the parameters of the target detection model are updated.

[0084] On the other hand, based on a preset reference image plane of a fixed depth, reference information for converting a height distribution into a depth distribution is obtained, and based on the reference information, the height distribution estimation result is converted from a height distribution into a depth distribution using a triangle similarity principle to obtain a converted first depth distribution result, including:

[0085] Based on the preset fixed depth reference image plane, reference information for converting height distribution to depth distribution is obtained.

[0086] For each height value in the height distribution of each target in the height distribution estimation result, obtain an initial depth conversion value corresponding to the height value in the height distribution of the target based on the reference information;

[0087] For each target in the height distribution estimation result, each of the initial depth conversion values ​​of the target is aligned with a corresponding depth value predefined in the depth distribution to obtain a converted first depth distribution result.

[0088] On the other hand, for each target in the height distribution estimation result, aligning each of the initial depth conversion values ​​of the target with a corresponding depth value predefined in the depth distribution to obtain a converted first depth distribution result, including:

[0089] For each target in the height distribution estimation result, according to According to an alignment rule, each of the initial depth conversion values ​​of the target is aligned with a corresponding depth value predefined in the depth distribution to obtain a converted first depth distribution result;

[0090] in, H It represents the number of predefined height values ​​in the height distribution. j is a positive integer, p i Represents the predefined first i Depth value, the probability obtained after alignment;

[0091] d set-i Represents the predefined first i Depth value, d max Indicates the predefined maximum depth value. d h j It represents the first of the initial depth conversion values ​​of the target. j initial depth conversion value, pd h j It represents the first of the initial depth conversion values ​​of the target.j The probability of the initial depth conversion value.

[0092] In a second aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the object detection method for road scenes as described above.

[0093] In a third aspect, the present invention provides an object detection device for a road scene, comprising:

[0094] memory for storing computer programs;

[0095] A processor is used to execute the computer program to implement the steps of the object detection method for road scenes as described above.

[0096] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the target detection method for road scenes as described above.

[0097] In a fifth aspect, the present invention provides a method for detecting an object in a road scene, comprising:

[0098] After the sensors in the roadside edge platform collect images, they send the collected images to the cloud data center through the communication device;

[0099] The cloud data center annotates the received image to obtain a training image and inputs the training image into the artificial intelligence processor;

[0100] The artificial intelligence processor is used for training a target detection model, and the training process includes: performing feature extraction on an input training image through an image feature extraction network of a preset target detection model to obtain a feature extraction result; based on the feature extraction result, respectively using a height distribution estimation network and a depth distribution estimation network to obtain a height distribution estimation result and a depth distribution estimation result; based on a preset reference image plane of a fixed depth, obtaining reference information for converting the height distribution to the depth distribution, and based on the reference information, using the triangle similarity principle, converting the height distribution estimation result from the height distribution to the depth distribution to obtain a converted first depth distribution result; fusing the first depth distribution result with the depth distribution estimation result to obtain a fused second depth distribution result; performing post-processing on the second depth distribution result based on the post-processing network in the target detection model to obtain a target detection result of the training image, and after calculating the loss function of the target detection model based on the target detection result, updating the parameters of the target detection model;

[0101] After the target detection model is trained, the cloud data center sends the target detection model to the roadside edge platform and deploys it, so that when the roadside edge platform receives the road scene image to be detected, the target detection result output by the target detection model is obtained based on the target detection model.

[0102] By applying the technical solution provided by the embodiments of the present invention, the height distribution and depth distribution are directly merged into one distribution, and then post-processing is performed, which helps to significantly reduce the computational and storage burden. Specifically, the input training image is subjected to feature extraction by the image feature extraction network of the preset target detection model. After obtaining the feature extraction results, the feature extraction results can be respectively input into the height distribution estimation network for height distribution estimation in the target detection model and the depth distribution estimation network for depth distribution estimation in the target detection model, to obtain the height distribution estimation results output by the height distribution estimation network and the depth distribution estimation results output by the depth distribution estimation network. Subsequently, the present application solution does not perform BEV feature extraction on the height distribution estimation results and the depth distribution estimation results separately, but instead converts the height distribution estimation results from height distribution to depth distribution, and the converted result is called the first depth distribution result. During the conversion, reference information for converting the height distribution to the depth distribution is obtained based on a preset fixed-depth reference image plane, and based on the reference information, the height distribution estimation results are converted from height distribution to depth distribution using the triangle similarity principle to obtain the converted first depth distribution result. After the conversion, the first depth distribution result needs to be fused with the depth distribution estimation result to obtain a fused second depth distribution result. Due to the fusion, when using the post-processing network in the target detection model for processing, it is not necessary to process the height distribution estimation result and the depth distribution estimation result separately. Instead, only the second depth distribution result needs to be post-processed to obtain the target detection result of the training image. After the loss function of the target detection model is calculated based on the target detection result, the parameters of the target detection model are updated. After the target detection model is trained, the road scene image to be detected can be input into the target detection model to obtain the target detection result output by the target detection model.

[0103] In addition, it can be understood that since the first depth distribution result and the depth distribution estimation result are directly integrated in the solution of the present application, in the actual use of the target detection model, when any one of the height distribution estimation result and the depth distribution estimation result is missing, the target detection model can still work normally, that is, when one of the height distribution estimation result and the depth distribution estimation result is missing, the second depth distribution result is equal to the first depth distribution result or the depth distribution estimation result, which does not affect the post-processing network in the target detection model to continue processing the second depth distribution result, and the target detection model can still output the target detection result. In traditional roadside visual target detection methods, since independent height branches and depth branches are required to obtain BEV features respectively, when one of the distributions is missing, the target detection will not be performed. For example, when the depth distribution is missing, the depth branch will not be able to obtain the BEV feature, resulting in the inability to detect the target.

[0104] In summary, the solution of the present application can effectively realize target detection in road scenes, reduce the computational burden, and has high reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0106] Figure 1 A schematic diagram of the BEV feature generation process for an existing roadside visual target detection method;

[0107] Figure 2 A flowchart of an implementation method for detecting an object in a road scene provided by a specific embodiment of the present invention;

[0108] Figure 3 A schematic diagram of the structure of a target detection model in a specific embodiment of the present invention;

[0109] Figure 4 Schematic diagram of the principle of converting height to depth;

[0110] Figure 5 This is a schematic diagram of the fusion principle in a specific embodiment of the present invention;

[0111] Figure 6 A schematic diagram of the structure of an object detection model in another specific embodiment of the present invention;

[0112] Figure 7A schematic diagram of the principle of improving the accuracy of the target detection model in a specific embodiment of the present invention;

[0113] Figure 8 A schematic diagram of optimization based on a high truth value coding network and a deep truth value coding network in a specific embodiment of the present invention;

[0114] Figure 9 A schematic diagram of optimization based on a highly distributed decoding network and a deeply distributed decoding network in a specific embodiment of the present invention;

[0115] Figure 10 A schematic diagram of the principle of a correction method using target height distribution consistency in a specific embodiment of the present invention;

[0116] Figure 11 A schematic diagram of the principle of achieving acceleration through row processing during fusion in a specific embodiment of the present invention

[0117] Figure 12 A schematic diagram of the device structure of a roadside edge platform or a cloud data center computing unit in a specific implementation manner;

[0118] Figure 13 A schematic diagram of the structure of a target detection device for a road scene provided in a specific embodiment of the present invention;

[0119] Figure 14 A schematic structural diagram of a computer-readable storage medium according to the present invention;

[0120] Figure 15 The figure is a schematic diagram of the data interaction process during implementation of the target detection method for road scenes in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0121] The core of the present invention is to provide a method, device, program product and storage medium for target detection in road scenes, which can effectively realize target detection in road scenes, reduce the computational burden and have high reliability.

[0122] In order to enable those skilled in the art to better understand the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0123] Please refer to Figure 2 , Figure 2This is a flowchart of an implementation method of a road scene object detection method provided in a specific embodiment of the present invention. The road scene object detection method may include the following steps:

[0124] Step S201: extracting features from an input training image through an image feature extraction network of a preset target detection model to obtain a feature extraction result.

[0125] In practical applications, roadside cameras can capture images of road scenes. Each captured image, after being annotated, can be used as a training image for training a target detection model. Furthermore, roadside cameras are typically deployed on roadside edge platforms, while target detection model training is typically performed in a cloud data center. Therefore, the roadside edge platform needs to transmit images captured by the roadside cameras to the cloud data center via its own communication unit. Annotation of images captured by roadside cameras can be performed manually, automatically using relevant models, or using a combination of multiple annotation methods, all without affecting the implementation of the present invention. Furthermore, when annotating images captured by roadside cameras, the specific annotation items can be set based on actual needs. For example, in one scenario, the targets to be annotated are pedestrians, cars, bicycles, trucks, buses, and motorcycles in a road scene. For each target, not only the target category needs to be specifically annotated, but also the target's location information, the size of the target's bounding box, and the target's orientation information. Of course, other specific implementations may also include more annotation content.

[0126] See Figure 3 , which is a schematic diagram of the structure of an object detection model in a specific embodiment. The image feature extraction network can receive input training images and perform feature extraction to obtain feature extraction results. The specific feature extraction method of the image feature extraction network can be set and adjusted according to actual needs and does not affect the implementation of the present invention.

[0127] Step S202: Based on the feature extraction result, a height distribution estimation result and a depth distribution estimation result are obtained using a height distribution estimation network and a depth distribution estimation network respectively.

[0128] Specifically, the feature extraction results can be input into the height distribution estimation network in the target detection model for height distribution estimation and the depth distribution estimation network in the target detection model for depth distribution estimation, respectively, to obtain the height distribution estimation results output by the height distribution estimation network and the depth distribution estimation results output by the depth distribution estimation network.

[0129] After the feature extraction results are input into the height distribution estimation network in the target detection model, the height distribution estimation network will perform height distribution estimation and output the height distribution estimation result. Specifically, the height distribution estimation result will include the height distribution of one or more different targets. For the height distribution of any target, the height distribution reflects the respective probabilities of the target at different height values ​​within the predefined height distribution range.

[0130] Similarly, after the feature extraction results are input into the depth distribution estimation network in the target detection model, the depth distribution estimation network will perform depth distribution estimation and output the depth distribution estimation result. Specifically, the depth distribution estimation result will include the depth distribution of one or more different targets. For the depth distribution of any target, the depth distribution reflects the respective probabilities of the target at different depth values ​​within the predefined depth distribution range.

[0131] Step S203: Based on a preset fixed-depth reference image plane, reference information for converting the height distribution to the depth distribution is obtained, and based on the reference information, the height distribution estimation result is converted from the height distribution to the depth distribution using the triangle similarity principle to obtain a converted first depth distribution result.

[0132] Specifically, in the solution of the present application, it is necessary to convert the height distribution estimation result from height distribution to depth distribution to obtain a converted first depth distribution result.

[0133] See Figure 3 Since the present application solution needs to realize the fusion of height distribution and depth distribution, and the present application takes into account that height distribution and depth distribution cannot be directly fused, the present application solution specifically converts the height distribution estimation result from height distribution to depth distribution, obtains the converted first depth distribution result, and then fuses it with the depth distribution estimation result. The fused result is called the second depth distribution result.

[0134] See Figure 4 , which is a schematic diagram of the principle of converting height to depth. Figure 4 Where Q is the roadside camera position, A and B represent the positions of the two targets in the 3D space corresponding to the pixel points in the image. Taking point B as an example, the height distribution estimation network will predict its height distribution within the predefined height distribution range. That is, through the height distribution estimation network, the probability of the target at different height values ​​within the predefined height distribution range can be obtained. In this example, the predefined height distribution range is Figure 3MN distance in. Assuming that there is no error in the prediction of the height distribution estimation network, the height distribution at point B should be a very steep curve, that is, when the horizontal coordinate is equal to BT, the vertical coordinate is close to 1, indicating that the height at point B is BT. Set a reference image plane with a fixed depth, and project the point V2 on the camera ray QB into 3D space, so that the depth L2V2 and the length QL2 of point V2 can be obtained. According to the properties of similar triangles, QL2 / QC =L2V2 / BC, where QL2, QC and L2V2 are all known, and the depth BC of point B can be calculated. For targets at other locations, such as point A, repeat the above calculation process to convert the height distribution of all feature points into depth distribution.

[0135] In a specific embodiment of the present invention, step S203 may specifically include:

[0136] Based on the preset fixed depth reference image plane, reference information for converting height distribution to depth distribution is obtained.

[0137] For each height value in the height distribution of each target in the height distribution estimation result, obtain an initial depth conversion value corresponding to the height value in the height distribution of the target based on the reference information;

[0138] For each target in the height distribution estimation result, each initial depth conversion value of the target is aligned with a corresponding depth value predefined in the depth distribution to obtain a converted first depth distribution result.

[0139] In this implementation, the properties of similar triangles are used to realize the conversion of height distribution to depth distribution. Specifically, it is necessary to preset a reference image plane of a fixed depth to obtain the reference information required for the conversion of height distribution to depth distribution, and then combine the height distribution estimation results to realize the conversion of height distribution to depth distribution. For example, in the above example, for the height value BT in the height distribution of target B, the required parameter information is QL2, QC and L2V2, and then the conversion of height distribution to depth distribution is realized according to the properties of the triangle. Similarly, for other height values ​​in the height distribution of target B, according to this principle, they can all be converted into corresponding depths, thereby converting the entire height distribution of target B into a depth distribution.

[0140] And in this embodiment, it is further considered that the depth value converted from the height value may not match the predefined depth point in the depth distribution. Therefore, the depth distribution obtained by converting the height distribution needs to be aligned to the predefined depth point. For ease of understanding, take a specific numerical value as an example to illustrate. In the depth distribution, for example, the predefined depth distribution range is 20 meters, and 1 depth value is divided every 0.5 meters, that is, the depth values ​​of the predefined depth distribution range are 0 meters, 0.5 meters, 1 meter, 1.5 meters, and so on until 20 meters. And for example, in the height distribution, the predefined height distribution range is 4 meters, and 1 height value is divided every 0.2 meters, that is, the height values ​​of the predefined height distribution range are 0 meters, 0.2 meters, 0.4 meters, 0.6 meters, and so on until 4 meters. And for example, for the height value of 0.6 meters, the depth value obtained after conversion to depth is 15.3 meters. It can be seen that there is no depth point 15.3 in the depth distribution, so alignment is needed.

[0141] In this regard, in this embodiment, after converting each height value in the height distribution of each target in the height distribution estimation result to a depth based on the reference information, the depth value obtained is called the initial depth conversion value corresponding to the height value in the height distribution of the target. Thereafter, for each target in the height distribution estimation result, it is necessary to align the initial depth conversion values ​​of the target with the corresponding depth values ​​predefined in the depth distribution to obtain the converted first depth distribution result.

[0142] There are many specific implementations for achieving alignment. For example, in one specific embodiment of the present invention, for each target in the height distribution estimation result, each initial depth conversion value of the target is aligned with the corresponding depth value predefined in the depth distribution to obtain a converted first depth distribution result, which may specifically include:

[0143] For each target in the height distribution estimation result, according to According to the alignment rule, each initial depth conversion value of the target is aligned with the corresponding depth value predefined in the depth distribution to obtain a converted first depth distribution result;

[0144] in, H It represents the number of predefined height values ​​in the height distribution. j is a positive integer, p i Represents the predefined first i Depth value, the probability obtained after alignment. d set-i Represents the predefined first i Depth value, dmax Indicates the predefined maximum depth value. d h j Indicates the initial depth conversion value of the target. j initial depth conversion value, pd h j Indicates the initial depth conversion value of the target. j The probability of the initial depth conversion value.

[0145] In this embodiment, p i Represents the predefined first i Depth value, the probability obtained after alignment, that is, for any target, can be calculated according to i From 1 to D in the order of p i The calculation of , thus obtaining the depth distribution of the target in the first depth distribution result. Taking the probability calculation of the first depth value in the depth distribution of the target as an example, it can be seen that for each initial depth conversion value of the target, it is necessary to compare with the predefined first depth value in the depth distribution and perform a weighted sum. The weight is the respective probability of each initial depth conversion value of the target. It can be seen that when d set-i and d h j When it is closer, it will make p i The value of is higher.

[0146] Step S204: fusing the first depth distribution result with the depth distribution estimation result to obtain a fused second depth distribution result.

[0147] After obtaining the first depth distribution result through conversion, it is necessary to fuse the first depth distribution result with the depth distribution estimation result to obtain a fused second depth distribution result.

[0148] During fusion, the fusion can be performed directly by taking the average value, or a more complex weighted fusion method can be set as needed.

[0149] For example, in a specific embodiment of the present invention, step S204 may specifically include:

[0150] Based on the first depth distribution result and the depth distribution estimation result, they are fused according to a fusion rule of taking an average value to obtain a fused second depth distribution result.

[0151] In this implementation, fusion is performed according to a fusion rule of taking an average value, which is simple to implement and helps to simplify calculations.

[0152] In a specific embodiment of the present invention, step S204 may specifically include:

[0153] Based on the first depth distribution result and the depth distribution estimation result, for any one target, according to Determine the first confidence level and the second confidence level of the target in a manner;

[0154] Based on the first confidence level and the second confidence level, The fusion weight is obtained by

[0155] based on The first depth distribution result and the depth distribution estimation result are fused according to a fusion rule to obtain a fused second depth distribution result;

[0156] in, C a It represents the first confidence of the target. C b It represents the second confidence of the target. D It represents the number of predefined depth values ​​in the depth distribution. p a ( j ) represents the depth distribution of the target in the first depth distribution result. j The probability of the height value, p b ( j ) represents the depth distribution of the target in the depth distribution estimation result. j Probability of height value; represents the fusion weight, p ( j ) represents the depth distribution of the target in the second depth distribution result obtained. j The probability of the height value.

[0157] In this implementation, a weighted fusion method is set, which can be found in Figure 5 , which is a schematic diagram of the fusion principle of this implementation. It is necessary to obtain a first confidence level and a second confidence level based on the first depth distribution result and the depth distribution estimation result, respectively, and then obtain a fusion weight, and finally fuse the first depth distribution result and the depth distribution estimation result.

[0158] Specifically, from the formulas of the first confidence level and the second confidence level, it can be seen that for a certain target, when the target depth distribution in the first depth distribution result is relatively concentrated, the first confidence level is Ca The value of will be relatively small, and the value of the fusion weight will be large. That is, at this time, the first depth distribution result is more reliable. Therefore, for this target, the first depth distribution result has a greater impact on the second depth distribution result, that is, p ( j ) are more affected p a ( j On the contrary, when the target depth distribution in the first depth distribution result is more dispersed, the first confidence level will be increased. C a The value of makes the fusion weight value decrease, thus weakening p a ( j )for p ( j )’s impact.

[0159] Similarly, for a certain target, when the depth distribution of the target in the depth distribution estimation result is relatively concentrated, the second confidence C b The value of will be relatively small, and the value of the fusion weight will be small. Therefore, for this target, when the depth distribution estimation result is more reliable, the depth distribution estimation result has a greater impact on the second depth distribution result, that is, p ( j ) are more affected p b ( j On the contrary, when the target depth distribution in the depth distribution estimation result is more dispersed, the second confidence level will be increased. C b The value of The value of p b ( j )for p ( j )’s impact.

[0160] In addition, it can be understood that for the above fusion rule of taking the average value, it is equivalent to the fusion weight being always fixed at 0.5, that is, in the above implementation method, for any target, it is based on The fusion rule is used to fuse the first depth distribution result with the depth distribution estimation result to obtain a fused second depth distribution result, which is computationally simpler. In this embodiment, the fusion weights will dynamically change for different targets, which is beneficial to improving the accuracy of the obtained second depth distribution result.

[0161] Step S205: Based on the post-processing network in the target detection model, the second depth distribution result is post-processed to obtain the target detection result of the training image, and after calculating the loss function of the target detection model based on the target detection result, the parameters of the target detection model are updated.

[0162] Depend on Figure 3 It can be seen that after the fusion is performed and the second depth distribution result is obtained, the second depth distribution result can be post-processed based on the post-processing network in the target detection model. The specific processing method can be set according to actual needs, and the target detection result of the training image can be effectively obtained based on the second depth distribution result. For example, the target detection result of the training image is usually obtained through BEV feature extraction.

[0163] For example, in a specific embodiment of the present invention, step S205 may specifically include:

[0164] Based on the point cloud feature extraction network in the post-processing network in the target detection model, point cloud feature extraction is performed on the second depth distribution result to obtain a point cloud feature extraction result;

[0165] Based on the bird's-eye view feature extraction network in the post-processing network, bird's-eye view feature extraction is performed on the point cloud feature extraction result to obtain the bird's-eye view feature extraction result;

[0166] Based on the detection network in the post-processing network, target detection is performed on the bird's-eye view feature extraction results to obtain the target detection results of the training image. After the loss function of the target detection model is calculated based on the target detection results, the parameters of the target detection model are updated.

[0167] See Figure 6 , is a schematic diagram of the structure of a target detection model in a specific embodiment of the present invention. The post-processing network of the target detection model in this embodiment specifically includes a point cloud feature extraction network, a bird's-eye view feature extraction network, and a detection network. The point cloud feature extraction network can be used to extract point cloud features from the second depth distribution result to achieve 2D to 3D conversion. In addition, it should be noted that when the point cloud feature extraction network obtains the point cloud feature extraction result, it is usually necessary to use the feature extraction result of the image feature extraction network. Therefore Figure 6 In the example, the image feature extraction network is connected to the point cloud feature extraction network.

[0168] The bird's-eye view feature extraction network can perform BEV feature extraction, that is, perform bird's-eye view feature extraction on the point cloud feature extraction results to obtain the bird's-eye view feature extraction results. Finally, the detection network performs target detection on the bird's-eye view feature extraction results to obtain the target detection results of the training image. For example, in a specific scenario, the target detection results obtained include the target category of each detected target, the target location information, the size information of the target's bounding box, the target orientation information, etc.

[0169] After obtaining the target detection results, it is necessary to calculate the loss function of the target detection model based on the target detection results, so as to perform reverse update, that is, update the parameters of the target detection model.

[0170] In practical applications, after the target detection model is trained, the road scene image to be detected is input into the target detection model to obtain the target detection result output by the target detection model.

[0171] Specifically, after multiple rounds of training, the object detection model can be deployed. For example, training is typically performed in a cloud data center. After training is complete, the object detection model is deployed on a roadside edge platform. Later, during operation, road scene images to be detected are input to the object detection model, which then outputs object detection results.

[0172] As described above, the process of the traditional solution requires h × w × c × ( H + D ) multiplications, required during voxel pooling h × w × ( H + D ) times of addition calculation, and in the solution of this application, after the height distribution and the depth distribution are integrated, only h × w × c × D Multiplication calculations are required in the process of voxel pooling to obtain BEV features. h × w × D This significantly reduces the computational and storage burden.

[0173] In the above Figure 4 The principle of converting height to depth is explained in the example of Figure 4The above calculations are explained under the assumption that the height distribution estimation network has no errors. When the height distribution estimation network has errors, the converted depth will also have errors with the actual depth of the target. Figure 4 Taking the target point B in the figure as an example, it can be seen that the maximum value of its height estimation is P1P3, and the calculated depth at this time is NP3. The minimum value of its height estimation is 0, and the calculated depth at this time is NP2. In other words, the height-to-depth conversion of target point B will be within the range of NP3 to NP2. Similarly, for the more distant target point A, its height-to-depth conversion will be within the range of NP6 to NP5. Obviously, when the target distance is farther and farther, the range of height-to-depth conversion corresponding to the same height distribution range will become larger and larger. In addition, the pixel ratio of distant targets in the image is low, and height estimation is difficult. The combination of these two factors leads to an increase in the depth estimation error for distant targets in actual applications, that is, the positioning accuracy for distant targets will be low.

[0174] In the solution of this application, it is considered that a combination of one or more optimization methods can be used to further improve the detection accuracy.

[0175] Specifically, in one embodiment of the present invention, the method may further include:

[0176] Inputting the true height data into a height truth encoding network for estimating height distribution, and obtaining a first height distribution result output by the height truth encoding network;

[0177] Based on the first height distribution result and the height distribution estimation result, obtaining a first loss function for reflecting the difference between the first height distribution result and the height distribution estimation result;

[0178] The height truth data is height truth data used to reflect parameter information of different targets in the training image, and the reflected parameter information includes the height of the center point of the bounding box of the target;

[0179] Accordingly, after calculating the loss function of the target detection model based on the target detection result described in step S205, updating the parameters of the target detection model may specifically include:

[0180] Calculate the loss function of the target detection model based on the target detection results;

[0181] Based on the loss function of the target detection model and the first loss function, the parameters of the target detection model are updated.

[0182] In this embodiment, please refer to Figure 7 , a height truth encoding network is set up to improve the accuracy of the height distribution estimation network.

[0183] Specifically, the height truth encoding network can estimate height distribution. The input to the height truth encoding network is height truth data, a set of height truth data that reflects the parameter information of different objects in the training image. This parameter information must include the height of the center point of the object's bounding box. It is also understood that the height truth data should be relatively accurate.

[0184] For example, in a specific implementation, the high truth data includes high truth sub-data in the form of multiple groups (z1, z2, z3, z4, z5).

[0185] Among them, for any set of height truth sub-data in the form of (z1, z2, z3, z4, z5), z1 represents the height of the center point of the bounding box of the target corresponding to the set of height truth sub-data, z2 represents the length of the bounding box of the target, z3 represents the width of the bounding box of the target, z4 represents the height of the bounding box of the target, and z5 represents the labeled category of the target;

[0186] It can be seen that the height truth sub-data in this embodiment includes the height of the center point of the target's bounding box, the length and width of the bounding box, and the target's labeled category, which can more comprehensively and accurately reflect the target's height information.

[0187] After each set of height truth sub-data in the height truth data is input into the height truth encoding network, the height truth encoding network can perform height distribution estimation to obtain a first height distribution result.

[0188] The first loss function is the first loss function used to reflect the difference between the first height distribution result and the height distribution estimation result. It can be understood that the larger the first loss function is, the greater the difference between the first height distribution result and the height distribution estimation result is, and the more necessary it is to update the parameters of the height distribution estimation network. Ideally, the height distribution estimation result obtained by the height distribution estimation network should be close to the first height distribution result.

[0189] The specific structure of the high truth value encoding network in this embodiment can be set and adjusted as needed, for example, it can be implemented using multiple layers of fully connected layers or convolutional layers. Figure 8 In the example, the high truth value encoding network is a high truth value encoding network implemented by multi-layer fully connected layers. Figure 8 H in i Represents the dimension of the input height truth data, W h-enc represents the number of neurons in the hidden layer, H Indicates the number of predefined height values ​​in the height distribution.

[0190] In addition, it should be noted that, since the first loss function is obtained in this implementation, when the parameters of the target detection model are subsequently updated, the parameters of the target detection model can be updated based on the loss function of the target detection model and the first loss function. That is, when this implementation is adopted, the parameters of the target detection model are updated, and the loss function used is = L b + k 1× L 1. Here L b It represents the loss function of the target detection model, that is, the main loss. k 1 represents the first loss function L The weight coefficient set by 1.

[0191] The first loss function reflects the difference between the first height distribution result and the height distribution estimation result. There are many specific implementation forms. For example, in a specific embodiment of the present invention, the first loss function can be specifically expressed as:

[0192] ;

[0193] in, L 1 represents the first loss function, n represents the number of objects in the training image. In one embodiment above, n That is, the number of groups of high-truth sub-data included in the high-truth data. H It represents the number of predefined height values ​​in the height distribution. i and j are all positive integers, P i h ( j ) means that in the height distribution estimation result, i The height distribution of the target j The probability of the height value, Q i he ( j ) means that in the first height distribution result, i The height distribution of the target j The probability of the height value.

[0194] It can be seen that the closer the height distribution estimation result is to the first height distribution result, the smaller the first loss function is. Therefore, through continuous training, the first loss function can be reduced, which means that through training, the height distribution estimation result is made as close as possible to the first height distribution result, which is conducive to improving the accuracy of the height distribution estimation network.

[0195] In the above, the height truth encoding network is used to optimize the height distribution estimation network. According to the same principle, the depth truth encoding network can be used to optimize the depth distribution estimation network. In practical applications, one of the two optimizations can be selected or used at the same time, depending on the actual needs. For example, the following uses the simultaneous use as an example. That is, in a specific embodiment of the present invention, it can also include:

[0196] Inputting the true depth data into a depth truth encoding network for depth distribution estimation to obtain a third depth distribution result output by the depth truth encoding network;

[0197] Based on the third depth distribution result and the depth distribution estimation result, obtaining a second loss function for reflecting the difference between the third depth distribution result and the depth distribution estimation result;

[0198] The depth truth data is depth truth data used to reflect parameter information of different targets in the training image, and the reflected parameter information includes the depth of the center point of the bounding box of the target;

[0199] Accordingly, after calculating the loss function of the target detection model based on the target detection result described in step S205, updating the parameters of the target detection model may specifically include:

[0200] Calculate the loss function of the target detection model based on the target detection results;

[0201] Based on the loss function of the target detection model, the first loss function and the second loss function, the parameters of the target detection model are updated.

[0202] See Figure 7 , not only a height truth encoding network is set up to improve the accuracy of the height distribution estimation network, but also a deep truth encoding network is set up to improve the accuracy of the depth distribution estimation network.

[0203] Specifically, the depth ground truth encoding network can also estimate depth distribution. The input to the depth ground truth encoding network is depth ground truth data, a set of depth ground truth data that reflects the parameter information of different objects in the training image. This parameter information must include the depth of the center point of the object's bounding box. Furthermore, it is understood that the depth ground truth data should be relatively accurate.

[0204] For example, in a specific implementation, the depth truth data includes multiple groups of depth truth sub-data in the form of (z6, z2, z3, z4, z5).

[0205] For any set of depth truth sub-data in the form of (z6, z2, z3, z4, z5), z6 represents the depth of the center point of the bounding box of the target corresponding to this set of depth truth sub-data, z2 represents the length of the bounding box of the target, z3 represents the width of the bounding box of the target, z4 represents the height of the bounding box of the target, and z5 represents the labeled category of the target. It can be seen that the depth truth sub-data in this embodiment includes the depth of the center point of the target's bounding box, the length and width of the bounding box, and the labeled category of the target, which can more comprehensively and accurately reflect the depth information of the target.

[0206] After each set of depth truth sub-data in the depth truth data is input into the depth truth encoding network, the depth truth encoding network can perform height distribution estimation to obtain a third depth distribution result.

[0207] The second loss function reflects the difference between the third depth distribution result and the depth distribution estimation result. The larger the second loss function, the greater the difference between the third depth distribution result and the depth distribution estimation result, and the more necessary it is to update the parameters of the depth distribution estimation network. Ideally, the depth distribution estimation result obtained by the depth distribution estimation network should be close to the third depth distribution result.

[0208] Similar to the above, the specific structure of the deep truth encoding network in this embodiment can be set and adjusted as needed, for example, it can be implemented using multiple fully connected layers or convolutional layers. Figure 8 In the example, the deep truth encoding network is a deep truth encoding network implemented with multiple layers of fully connected layers. Figure 8 D in i Represents the dimension of the input depth truth data, W d-enc represents the number of neurons in the hidden layer, D Indicates the number of predefined depth values ​​in the depth distribution.

[0209] In addition, it should be noted that, since the first loss function and the second loss function are obtained in this embodiment, when the parameters of the target detection model are subsequently updated, the parameters of the target detection model can be updated based on the loss function of the target detection model, the first loss function and the second loss function. For example, in a specific embodiment, the total loss function is obtained based on the loss function of the target detection model, the first loss function and the second loss function. Lc , and based on the total loss Lc Update the parameters of the target detection model.

[0210] At this time Lc = L b + k 1×L 1+ k 2× L 2, L b It represents the loss function of the target detection model. k 1 represents the weight coefficient set for the first loss function, k 2 represents the weight coefficient set for the second loss function.

[0211] In addition, the second loss function reflects the difference between the third depth distribution result and the depth distribution estimation result. There may be multiple specific implementation forms. For example, in a specific embodiment of the present invention, the second loss function may be specifically expressed as:

[0212] The second loss function is expressed as:

[0213] ;

[0214] in, L 2 represents the second loss function, n represents the number of objects in the training image. i and j are all positive integers, D It represents the number of predefined depth values ​​in the depth distribution. P i d ( j ) means that in the depth distribution estimation result, i The depth distribution of the target j The probability of depth value, Q i de ( j ) means that in the third depth distribution result, i The depth distribution of the target j The probability of depth value.

[0215] It can be seen that the closer the depth distribution estimation result is to the third depth distribution result, the smaller the second loss function is. Therefore, through continuous training, the second loss function can be reduced, which means that through training, the depth distribution estimation result is made as close as possible to the third depth distribution result, which is conducive to improving the accuracy of the depth distribution estimation network.

[0216] In a specific embodiment of the present invention, it may further include:

[0217] Based on the first loss function, the parameters of the high-level truth encoding network are updated;

[0218] Based on the second loss function, the parameters of the deep truth encoding network are updated;

[0219] Among them, the high truth value coding network is a high truth value coding network including multiple layers of fully connected layers, and the deep truth value coding network is a deep truth value coding network including multiple layers of fully connected layers.

[0220] This implementation method takes into account that although the high truth value coding network and the deep truth value coding network are usually pre-set and relatively accurate networks, they are not necessarily the most ideal state. Moreover, as the training progresses, the accuracy of the deep distribution estimation network and the high distribution estimation network is constantly improving. Therefore, the parameters of the high truth value coding network can also be updated based on the first loss function to further improve the accuracy of the high truth value coding network, and the parameters of the deep truth value coding network can be updated based on the second loss function to further improve the accuracy of the deep truth value coding network. In addition, in this implementation method, considering the ease of updating, both the high truth value coding network and the deep truth value coding network can adopt a multi-layer fully connected layer structure.

[0221] In a specific embodiment of the present invention, it may further include:

[0222] Inputting the height distribution estimation result into a height distribution decoding network for decoding the height value to obtain the height value data output by the height distribution decoding network;

[0223] Based on the height value data and the predetermined true height data, a third loss function is obtained to reflect the difference between the height value data and the true height data;

[0224] Accordingly, after calculating the loss function of the target detection model based on the target detection results, the parameters of the target detection model are updated, including:

[0225] Calculate the loss function of the target detection model based on the target detection results;

[0226] Based on the loss function of the target detection model and the third loss function, the parameters of the target detection model are updated.

[0227] In this implementation, it is considered that the accuracy of the height distribution estimation network can be further improved based on the height distribution decoding network, and it should be noted that the optimization scheme of this implementation and the above-mentioned optimization schemes based on the height truth value encoding network and the depth truth value encoding network can be partially or fully adopted, which does not affect the implementation of the present invention.

[0228] For example Figure 7In the example, not only a height truth encoding network is set to improve the accuracy of the height distribution estimation network, but also a deep truth encoding network is set to improve the accuracy of the depth distribution estimation network, and a height distribution decoding network is set to improve the accuracy of the height distribution estimation network.

[0229] The input of the height distribution decoding network is the height distribution estimation result, which the height distribution decoding network will parse to obtain the height value data. Then, based on the height value data and the predetermined true height data, it is necessary to obtain a third loss function that reflects the difference between the height value data and the true height data.

[0230] It can be understood that the smaller the third loss function is, the closer the height value data output by the height distribution decoding network is to the predetermined true height data, and the height value data output by the height distribution decoding network will be affected by the height distribution estimation result input to the height distribution decoding network. Therefore, continuous training is beneficial to improve the accuracy of the height distribution estimation network.

[0231] Similarly, in this implementation, since the third loss function is obtained, it is necessary to update the parameters of the target detection model based on the loss function of the target detection model and the third loss function.

[0232] In a specific embodiment of the present invention, it may further include:

[0233] Inputting the depth distribution estimation result into a depth distribution decoding network for performing depth value decoding to obtain depth value data output by the depth distribution decoding network;

[0234] Based on the depth value data and the predetermined true depth data, a fourth loss function is obtained to reflect the difference between the depth value data and the true depth data;

[0235] Accordingly, after calculating the loss function of the target detection model based on the target detection results, the parameters of the target detection model are updated, including:

[0236] Calculate the loss function of the target detection model based on the target detection results;

[0237] Based on the loss function of the target detection model, the third loss function and the fourth loss function, the parameters of the target detection model are updated.

[0238] In this implementation, considering that the accuracy of the depth distribution estimation network can be further improved based on the depth distribution decoding network, similarly, the optimization scheme of this implementation and the other optimization schemes above can be adopted in part or in full, which does not affect the implementation of the present invention.

[0239] For example Figure 7 In the example, not only a height truth encoding network is set up to improve the accuracy of the height distribution estimation network, but also a depth truth encoding network is set up to improve the accuracy of the depth distribution estimation network. A height distribution decoding network is set up to improve the accuracy of the height distribution estimation network, and a depth distribution decoding network is set up to improve the accuracy of the depth distribution estimation network.

[0240] The input of the depth distribution decoding network is the depth distribution estimation result, which the depth distribution decoding network will parse to obtain depth value data. Then, based on the depth value data and the predetermined true depth data, a fourth loss function is obtained to reflect the difference between the depth value data and the true depth data.

[0241] It can be understood that the smaller the fourth loss function is, the closer the depth value data output by the depth distribution decoding network is to the predetermined true depth data. The depth value data output by the depth distribution decoding network will be affected by the depth distribution estimation result input to the depth distribution decoding network. Therefore, continuous training is beneficial to improve the accuracy of the depth distribution estimation network.

[0242] The specific forms of the third loss function and the fourth loss function can be set according to actual needs. For example, in a specific implementation, the third loss function is expressed as:

[0243] ;

[0244] The fourth loss function is expressed as:

[0245] ;

[0246] in, L 3 represents the third loss function, L 4 represents the fourth loss function, n represents the number of objects in the training image. h i It represents the first i The true height of the center point of the target's bounding box, h i ' represents the height value data output by the height distribution decoding network i The height of the center point of the target's bounding box, d i It represents the first i The true depth of the center point of the target's bounding box, d i ' represents the depth value data output by the depth distribution decoding network iThe depth of the object's bounding box center point.

[0247] It can be seen that the third loss function in this embodiment can accurately and effectively reflect the difference between the height value data and the true height data, while the fourth loss function in this embodiment can accurately and effectively reflect the difference between the depth value data and the true depth data.

[0248] The specific structures of the height distribution decoding network and the depth distribution decoding network in the above embodiments can be set and adjusted as needed, for example, they can both be implemented using multiple layers of fully connected layers or convolutional layers. Figure 9 In the example, both the highly distributed decoding network and the deeply distributed decoding network are implemented using multiple layers of fully connected layers. Figure 9 in H represents the number of predefined height values ​​in the height distribution, D represents the number of predefined depth values ​​in the depth distribution, W d-enc represents the number of neurons in the hidden layer, W h-enc represents the number of neurons in the hidden layer.

[0249] In a specific embodiment of the present invention, it may further include:

[0250] Based on the third loss function, the parameters of the highly distributed decoding network are updated;

[0251] Based on the fourth loss function, the parameters of the deep distribution decoding network are updated;

[0252] Among them, the highly distributed decoding network is a highly distributed decoding network including multiple layers of fully connected layers, and the deeply distributed decoding network is a deeply distributed decoding network including multiple layers of fully connected layers.

[0253] This implementation method takes into account that although the height distribution decoding network and the depth distribution decoding network are usually pre-set and relatively accurate networks, they are not necessarily the most ideal state. Moreover, as the training progresses, the accuracy of the depth distribution estimation network and the height distribution estimation network is constantly improving. Therefore, the parameters of the height distribution decoding network can also be updated based on the third loss function to further improve the accuracy of the height distribution decoding network, and the parameters of the depth distribution decoding network can be updated based on the fourth loss function to further improve the accuracy of the depth distribution decoding network. In addition, in this implementation method, considering the ease of updating, both the height distribution decoding network and the depth distribution decoding network can adopt a multi-layer fully connected layer structure.

[0254] In a specific embodiment of the present invention, it may further include:

[0255] For each target pointed by the height distribution estimation result, classify it according to the target's labeled category;

[0256] For any one category, determine whether the number of targets in the category exceeds a preset first threshold;

[0257] If not, discard the category;

[0258] If yes, retain the classification;

[0259] For each retained category, The loss corresponding to the classification is determined in this way;

[0260] according to The fifth loss function is obtained by:

[0261] in, L 5 represents the fifth loss function, C It represents the total number of retained categories. a is a positive integer;

[0262] L a Indicates the corresponding a Classification loss, i and j as well as k are all positive integers, H It represents the number of predefined height values ​​in the height distribution. w ik It means the a The first in the classification i The goal and the first k The weights between the targets, M i h ( j ) means that in the height distribution estimation result, a The first in the classification i The height distribution of the target j The probability of the height value, M k h ( j ) means that in the height distribution estimation result, a The first in the classification k The height distribution of the target j Probability of height value;

[0263] and, , d i It means the i The depth value of the target,d k It means the k The depth value of the target, d max It represents the predefined maximum depth value;

[0264] Accordingly, after calculating the loss function of the target detection model based on the target detection results, the parameters of the target detection model are updated, including:

[0265] Calculate the loss function of the target detection model based on the target detection results;

[0266] Based on the loss function of the target detection model and the fifth loss function, the parameters of the target detection model are updated.

[0267] In this implementation, a correction method for target height distribution consistency is adopted to further improve the accuracy of the height distribution estimation network.

[0268] Specifically, this implementation method takes into account that for targets of the same labeled category, the heights should be similar. For example, if the labeled category of multiple different targets is motorcycle, the heights of these targets should be similar. Based on this idea, in this implementation method, each target pointed to by the height distribution estimation result will be classified according to the labeled category of the target.

[0269] For any category, it is necessary to determine whether the number of targets in the category exceeds the preset first threshold. If it does not exceed the first threshold, it means that the number of targets in the category is too small and does not have much analytical value. The category can be discarded, and the rest can be retained.

[0270] In this implementation, for each retained category, This formula is based on the first a Let’s take classification as an example to illustrate, Figure 10 In the example, a1, a2 and a3 represent the a There are three targets under the category, b1 and b2 represent the b 2 goals under the category.

[0271] By calculating a Losses by category L a From the formula, we can see that in this implementation, the a For each target under the classification, the height distribution difference is calculated pairwise. And it should be emphasized that the formula also sets the a The first in the classification i The goal and the first kWeights between targets w ik .

[0272] Specifically, , d i It means the i The depth value of the target, d k It means the k The depth value of the target, d max Indicates the predefined maximum depth value.

[0273] This setting is based on the consideration that when optimizing the height distribution of different types of targets using this implementation method, the height distribution of distant targets should be guided by close targets, rather than by original distance targets, because the height distribution of close targets is more accurate. d i ≤ d k When the original distance target cannot be used to guide the height distribution of the close distance target, that is, w ik Equal to 0. On the contrary, when d i > d k At this time w ik It takes effect only when the value is not equal to 0.

[0274] Since this implementation method obtains the fifth loss function, the parameters of the target detection model can be updated based on the loss function of the target detection model and the fifth loss function.

[0275] The total loss at this time Lc = L b + k 1× L 1+ k 2× L 2+ k 3× L 3+ k 4× L 4+ k 5× L 5. L b It represents the loss function of the target detection model. k 1 to k5 represents the weight coefficients set for the first to fifth loss functions, which determines the degree of influence of each of the five optimization methods on the total loss. k 1 to k The value is set to 5.

[0276] That is to say, when any one of the first to fifth loss functions is obtained, the loss function can be weighted superimposed when calculating the total loss. Anyway, if no optimization method is adopted, that is, if there is no need to calculate a certain loss function from the first to fifth loss functions, then there is no need to perform weighted superposition of the corresponding loss function when calculating the total loss.

[0277] In addition, it should be pointed out that no matter which of the above optimization methods is selected, they are all used during training. After the training is completed, when deploying the target detection model, only the target detection model itself needs to be deployed. Figure 7 For example, after the training is completed, the trained target detection model can be deployed on the roadside edge platform, that is, Figure 7 The height truth encoding network, the depth truth encoding network, the height distribution decoding network, the depth distribution decoding network and the correction method for target height distribution consistency shown in the figure do not need to be deployed on the roadside edge platform, so Figure 7 In the figure, these optimization schemes are connected to the target detection model with dotted lines to indicate that these optimization schemes only need to be used in the training phase.

[0278] Furthermore, in a specific embodiment of the present invention, whether in the training process or in the running process after deployment, multi-threaded parallel acceleration can be used to achieve parallel processing to improve processing efficiency.

[0279] Specifically, for the size h (Height of image feature) × w (width of image feature)× D (predefined number of depth values) and the depth distribution estimation results of size h (Height of image feature) × w (width of image feature)× H The height distribution estimation result of (a predefined number of height values) can be h and w The dimension is divided into t partitions, where t is a preset parameter. The length of each partition is h × w / t. See Figure 11, is a schematic diagram of the principle of achieving acceleration through a row processing scheme during fusion in a specific implementation method, and the depth distribution estimation results and the height distribution estimation results are both calculated according to h and w After the dimension is divided into t partitions, for example, multiple threads of an AI processor can process data from different partitions simultaneously to achieve conversion from height to depth and fusion operations.

[0280] Similarly, after training is completed and deployed, partitioning can also be performed according to the above principles. Figure 12 Figure 2 shows the device architecture of a roadside edge platform or cloud data center computing unit in a specific implementation. Taking the roadside edge platform as an example, each partition can be assigned to a corresponding thread of the AI ​​processor, enabling parallel computing to achieve multi-threaded parallel acceleration of deep-to-height conversion and fusion operations.

[0281] By applying the technical solution provided by the embodiments of the present invention, the height distribution and depth distribution are directly merged into one distribution, and then post-processing is performed, which helps to significantly reduce the computational and storage burden. Specifically, the input training image is subjected to feature extraction by the image feature extraction network of the preset target detection model. After obtaining the feature extraction results, the feature extraction results can be respectively input into the height distribution estimation network for height distribution estimation in the target detection model and the depth distribution estimation network for depth distribution estimation in the target detection model, to obtain the height distribution estimation results output by the height distribution estimation network and the depth distribution estimation results output by the depth distribution estimation network. Subsequently, the present application solution does not perform BEV feature extraction on the height distribution estimation results and the depth distribution estimation results separately, but instead converts the height distribution estimation results from height distribution to depth distribution, and the converted result is called the first depth distribution result. During the conversion, reference information for converting the height distribution to the depth distribution is obtained based on a preset fixed-depth reference image plane, and based on the reference information, the height distribution estimation results are converted from height distribution to depth distribution using the triangle similarity principle to obtain the converted first depth distribution result. After the conversion, the first depth distribution result needs to be fused with the depth distribution estimation result to obtain a fused second depth distribution result. Due to the fusion, when using the post-processing network in the target detection model for processing, it is not necessary to process the height distribution estimation result and the depth distribution estimation result separately. Instead, only the second depth distribution result needs to be post-processed to obtain the target detection result of the training image. After the loss function of the target detection model is calculated based on the target detection result, the parameters of the target detection model are updated. After the target detection model is trained, the road scene image to be detected can be input into the target detection model to obtain the target detection result output by the target detection model.

[0282] In addition, it can be understood that since the first depth distribution result and the depth distribution estimation result are directly integrated in the solution of the present application, in the actual use of the target detection model, when any one of the height distribution estimation result and the depth distribution estimation result is missing, the target detection model can still work normally, that is, when one of the height distribution estimation result and the depth distribution estimation result is missing, the second depth distribution result is equal to the first depth distribution result or the depth distribution estimation result, which does not affect the post-processing network in the target detection model to continue processing the second depth distribution result, and the target detection model can still output the target detection result. In traditional roadside visual target detection methods, since independent height branches and depth branches are required to obtain BEV features respectively, when one of the distributions is missing, the target detection will not be performed. For example, when the depth distribution is missing, the depth branch will not be able to obtain the BEV feature, resulting in the inability to detect the target.

[0283] In summary, the solution of the present application can effectively realize target detection in road scenes, reduce the computational burden, and has high reliability.

[0284] Corresponding to the above method embodiments, embodiments of the present invention further provide a target detection device for a road scene, a computer-readable storage medium, and a computer program product, which can be referenced in correspondence with the above.

[0285] See also Figure 13 As shown, the device may include:

[0286] Memory 301, used for storing computer programs;

[0287] The processor 302 is configured to execute a computer program to implement the steps of the method for detecting an object in a road scene in any of the above embodiments.

[0288] The computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the object detection method for road scenes in any of the above embodiments.

[0289] See Figure 14The computer-readable storage medium 40 stores a computer program 41. When executed by a processor, the computer program 41 implements the steps of the method for detecting an object in a road scene as described in any of the above embodiments. The computer-readable storage medium 40 herein includes a RAM (Random Access Memory), internal memory, a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read Only Memory), a register, a hard disk, a removable disk, or any other form of storage medium known in the art.

[0290] An embodiment of the present invention further provides a method for detecting an object in a road scene, comprising:

[0291] After the sensors in the roadside edge platform collect images, they send the collected images to the cloud data center through the communication device;

[0292] The cloud data center annotates the received images to obtain training images and inputs the training images into the artificial intelligence processor;

[0293] The artificial intelligence processor is used for training the target detection model, and the training process includes: extracting features from the input training image through the image feature extraction network of the preset target detection model to obtain feature extraction results; based on the feature extraction results, respectively using the height distribution estimation network and the depth distribution estimation network to obtain height distribution estimation results and depth distribution estimation results; based on the preset fixed-depth reference image plane, obtaining reference information for converting the height distribution to the depth distribution, and based on the reference information, using the triangle similarity principle to convert the height distribution estimation result from the height distribution to the depth distribution to obtain a converted first depth distribution result; fusing the first depth distribution result with the depth distribution estimation result to obtain a fused second depth distribution result; based on the post-processing network in the target detection model, post-processing the second depth distribution result to obtain the target detection result of the training image, and after calculating the loss function of the target detection model based on the target detection result, updating the parameters of the target detection model;

[0294] After the target detection model is trained, the cloud data center sends the target detection model to the roadside edge platform and deploys it, so that when the roadside edge platform receives the road scene image to be detected, it obtains the target detection result output by the target detection model based on the target detection model.

[0295] For easier understanding, please refer to Figure 15, which is a schematic diagram of the data interaction process when implementing this target detection method for road scenes. In the roadside edge platform, sensors are used to collect images, and then the collected images can be sent to the cloud data center through communication devices. After the cloud data center annotates the received images, it can train the target detection model based on the artificial intelligence processor, i.e., the AI ​​processor. After training, the target detection model is sent to the roadside edge platform for deployment and can be run. That is, when the roadside edge platform receives the road scene image to be detected, it can obtain the target detection result output by the target detection model based on the target detection model.

[0296] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0297] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0298] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the technical solution and core ideas of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A method for detecting an object in a road scene, characterized in that: include: The input training image is subjected to feature extraction through the image feature extraction network of the preset target detection model to obtain the feature extraction result; Based on the feature extraction results, a height distribution estimation result and a depth distribution estimation result are obtained using a height distribution estimation network and a depth distribution estimation network respectively; Based on a preset reference image plane of a fixed depth, reference information for converting a height distribution into a depth distribution is obtained, and based on the reference information, the height distribution estimation result is converted from a height distribution into a depth distribution using a triangle similarity principle to obtain a converted first depth distribution result; fusing the first depth distribution result with the depth distribution estimation result to obtain a fused second depth distribution result; Based on the post-processing network in the target detection model, the second depth distribution result is post-processed to obtain the target detection result of the training image, and after the loss function of the target detection model is calculated based on the target detection result, the parameters of the target detection model are updated.

2. The target detection method for road scenes according to claim 1, characterized in that: Also includes: Inputting the true height data into a height truth encoding network for estimating height distribution, and obtaining a first height distribution result output by the height truth encoding network; Based on the first height distribution result and the height distribution estimation result, obtaining a first loss function for reflecting the difference between the first height distribution result and the height distribution estimation result; The height truth data is height truth data used to reflect parameter information of different targets in the training image, and the reflected parameter information includes the height of the center point of the bounding box of the target; Accordingly, after calculating the loss function of the target detection model based on the target detection result, updating the parameters of the target detection model includes: Calculating a loss function of the target detection model based on the target detection result; Based on the loss function of the target detection model and the first loss function, parameters of the target detection model are updated.

3. The target detection method for road scenes according to claim 2, characterized in that: Also includes: Inputting the true depth data into a depth truth encoding network for depth distribution estimation to obtain a third depth distribution result output by the depth truth encoding network; obtaining, based on the third depth distribution result and the depth distribution estimation result, a second loss function for reflecting a difference between the third depth distribution result and the depth distribution estimation result; The true depth data is used to reflect parameter information of different targets in the training image, and the reflected parameter information includes the depth of the center point of the bounding box of the target; Accordingly, after calculating the loss function of the target detection model based on the target detection result, updating the parameters of the target detection model includes: Calculating a loss function of the target detection model based on the target detection result; Based on the loss function of the target detection model, the first loss function and the second loss function, the parameters of the target detection model are updated.

4. The target detection method for road scenes according to claim 3, characterized in that: Also includes: Based on the first loss function, updating the parameters of the high-level truth encoding network; Based on the second loss function, updating the parameters of the deep truth encoding network; Among them, the high truth value coding network is a high truth value coding network including multiple layers of fully connected layers, and the deep truth value coding network is a deep truth value coding network including multiple layers of fully connected layers.

5. The target detection method for road scenes according to claim 3, characterized in that: The high truth value data includes high truth value sub-data in the form of multiple groups (z1, z2, z3, z4, z5); Among them, for any set of height truth sub-data in the form of (z1, z2, z3, z4, z5), z1 represents the height of the center point of the bounding box of the target corresponding to the set of height truth sub-data, z2 represents the length of the bounding box of the target, z3 represents the width of the bounding box of the target, z4 represents the height of the bounding box of the target, and z5 represents the labeled category of the target; The depth truth data includes multiple groups of depth truth sub-data in the form of (z6, z2, z3, z4, z5); Among them, for any set of depth truth sub-data in the form of (z6, z2, z3, z4, z5), z6 represents the depth of the center point of the bounding box of the target corresponding to the set of depth truth sub-data.

6. The target detection method for road scenes according to claim 3, characterized in that: The first loss function is expressed as: ； The second loss function is expressed as: ； in, L 1 represents the first loss function, L 2 represents the second loss function, n represents the number of targets in the training image, H It represents the number of predefined height values ​​in the height distribution. i and j are all positive integers, P i h ( j ) represents that in the height distribution estimation result, i The height distribution of the target j The probability of the height value, Q i he ( j ) means that in the first height distribution result, i The height distribution of the target j Probability of height value; D It represents the number of predefined depth values ​​in the depth distribution. P i d ( j ) represents that in the depth distribution estimation result, i The depth distribution of the target j The probability of depth value, Q i de ( j ) represents that in the third depth distribution result, i The depth distribution of the target j The probability of depth value.

7. The target detection method for road scenes according to claim 6, characterized in that: Based on the loss function of the target detection model, the first loss function, and the second loss function, updating parameters of the target detection model includes: Based on the loss function of the target detection model, the first loss function and the second loss function, the total loss is obtained Lc , and based on the total loss Lc Updating parameters of the target detection model; in, Lc = L b + k 1× L 1+ k 2× L 2, L b It represents the loss function of the target detection model. k 1 represents the weight coefficient set for the first loss function, k 2 represents the weight coefficient set for the second loss function.

8. The target detection method for road scenes according to claim 1, characterized in that: Also includes: Inputting the height distribution estimation result into a height distribution decoding network for performing height value decoding to obtain height value data output by the height distribution decoding network; Based on the height value data and predetermined true height data, a third loss function is obtained to reflect the difference between the height value data and the true height data; Accordingly, after calculating the loss function of the target detection model based on the target detection result, updating the parameters of the target detection model includes: Calculating a loss function of the target detection model based on the target detection result; Based on the loss function of the target detection model and the third loss function, parameters of the target detection model are updated.

9. The target detection method for road scenes according to claim 8, characterized in that: Also includes: Inputting the depth distribution estimation result into a depth distribution decoding network for performing depth value decoding to obtain depth value data output by the depth distribution decoding network; Based on the depth value data and predetermined true depth data, obtaining a fourth loss function for reflecting a difference between the depth value data and the true depth data; Accordingly, after calculating the loss function of the target detection model based on the target detection result, updating the parameters of the target detection model includes: Calculating a loss function of the target detection model based on the target detection result; Based on the loss function of the target detection model, the third loss function and the fourth loss function, the parameters of the target detection model are updated.

10. The target detection method for road scenes according to claim 9, characterized in that: The third loss function is expressed as: ； The fourth loss function is expressed as: ； in, L 3 represents the third loss function, L 4 represents the fourth loss function, n represents the number of targets in the training image, h i It represents the first value obtained from the real height data. i The true height of the center point of the target's bounding box, h i ' represents the height value data output by the height distribution decoding network i The height of the center point of the target's bounding box, d i It represents the first i The true depth of the center point of the target's bounding box, d i ' represents the depth value data output by the depth distribution decoding network i The depth of the object's bounding box center point.

11. The target detection method for road scenes according to claim 1, characterized in that: Also includes: Classify each target pointed to by the height distribution estimation result according to the target's labeled category; For any one category, determine whether the number of targets in the category exceeds a preset first threshold; If not, discard the category; If yes, retain the classification; For each retained category, The loss corresponding to the classification is determined in this way; according to The fifth loss function is obtained by: in, L 5 represents the fifth loss function, C It represents the total number of retained categories. a is a positive integer; L a Indicates the corresponding a Classification loss, i and j as well as k are all positive integers, H It represents the number of predefined height values ​​in the height distribution. w ik It means the a The first in the classification i The goal and the first k The weights between the targets, M i h ( j ) represents that in the height distribution estimation result, a The first in the classification i The height distribution of the target j The probability of the height value, M k h ( j ) represents that in the height distribution estimation result, a The first in the classification k The height distribution of the target j Probability of height value; and, , d i It means the i The depth value of the target, d k It means the k The depth value of the target, d max It represents the predefined maximum depth value; Accordingly, after calculating the loss function of the target detection model based on the target detection result, updating the parameters of the target detection model includes: Calculating a loss function of the target detection model based on the target detection result; Based on the loss function of the target detection model and the fifth loss function, parameters of the target detection model are updated.

12. The method for detecting a target in a road scene according to claim 1, wherein: Fusing the first depth distribution result with the depth distribution estimation result to obtain a fused second depth distribution result, including: Based on the first depth distribution result and the depth distribution estimation result, a fusion is performed according to a fusion rule of taking an average value to obtain a fused second depth distribution result.

13. The method for detecting objects in a road scene according to claim 1, wherein: Fusing the first depth distribution result with the depth distribution estimation result to obtain a fused second depth distribution result, including: Based on the first depth distribution result and the depth distribution estimation result, for any one target, according to Determine the first confidence level and the second confidence level of the target in a manner; Based on the first confidence level and the second confidence level, The fusion weight is obtained by based on The first depth distribution result and the depth distribution estimation result are fused according to a fusion rule to obtain a fused second depth distribution result; in, C a It represents the first confidence of the target. C b It represents the second confidence of the target. D It represents the number of predefined depth values ​​in the depth distribution. p a ( j ) represents the first depth distribution of the target in the first depth distribution result. j The probability of the height value, p b ( j ) represents the depth distribution of the target in the depth distribution estimation result. j Probability of height value; represents the fusion weight, p ( j ) represents the depth distribution of the target in the second depth distribution result obtained. j The probability of the height value.

14. The method for detecting a target in a road scene according to claim 1, wherein: The method further comprises: performing post-processing on the second depth distribution result based on a post-processing network in the target detection model to obtain a target detection result of the training image; and performing parameter updating on the target detection model after calculating a loss function of the target detection model based on the target detection result, including: Performing point cloud feature extraction on the second depth distribution result based on a point cloud feature extraction network in a post-processing network in the target detection model to obtain a point cloud feature extraction result; Based on the bird's-eye view feature extraction network in the post-processing network, performing bird's-eye view feature extraction on the point cloud feature extraction result to obtain a bird's-eye view feature extraction result; Based on the detection network in the post-processing network, target detection is performed on the bird's-eye view feature extraction result to obtain the target detection result of the training image, and after the loss function of the target detection model is calculated based on the target detection result, the parameters of the target detection model are updated.

15. The method for detecting an object in a road scene according to any one of claims 1 to 14, characterized in that: Based on a preset reference image plane of a fixed depth, reference information for converting a height distribution into a depth distribution is obtained, and based on the reference information, the height distribution estimation result is converted from a height distribution into a depth distribution using a triangle similarity principle to obtain a converted first depth distribution result, including: Based on the preset fixed depth reference image plane, reference information for converting height distribution to depth distribution is obtained. For each height value in the height distribution of each target in the height distribution estimation result, obtaining, based on the reference information, an initial depth conversion value corresponding to the height value in the height distribution of the target; For each target in the height distribution estimation result, each of the initial depth conversion values ​​of the target is aligned with a corresponding depth value predefined in the depth distribution to obtain a converted first depth distribution result.

16. The method for detecting a target in a road scene according to claim 15, wherein: For each target in the height distribution estimation result, aligning each of the initial depth conversion values ​​of the target with a corresponding depth value predefined in the depth distribution to obtain a converted first depth distribution result, including: For each target in the height distribution estimation result, according to According to an alignment rule, each of the initial depth conversion values ​​of the target is aligned with a corresponding depth value predefined in the depth distribution to obtain a converted first depth distribution result; in, H It represents the number of predefined height values ​​in the height distribution. j is a positive integer, p i Represents the predefined first i Depth value, the probability obtained after alignment; d set-i Represents the predefined first i Depth value, d max Indicates the predefined maximum depth value. d h j It represents the first of the initial depth conversion values ​​of the target. j initial depth conversion value, pd h j It represents the first of the initial depth conversion values ​​of the target. j The probability of the initial depth conversion value.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the object detection method for road scenes described in any one of claims 1 to 16 are implemented.

18. A target detection device for a road scene, characterized in that: include: memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the object detection method for a road scene as claimed in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the object detection method for road scenes according to any one of claims 1 to 16.

20. A method for detecting an object in a road scene, characterized in that: include: After the sensors in the roadside edge platform collect images, they send the collected images to the cloud data center through the communication device; The cloud data center annotates the received image to obtain a training image and inputs the training image into the artificial intelligence processor; The artificial intelligence processor is used for training a target detection model, and the training process includes: performing feature extraction on an input training image through an image feature extraction network of a preset target detection model to obtain a feature extraction result; based on the feature extraction result, respectively using a height distribution estimation network and a depth distribution estimation network to obtain a height distribution estimation result and a depth distribution estimation result; based on a preset reference image plane of a fixed depth, obtaining reference information for converting the height distribution to the depth distribution, and based on the reference information, using the triangle similarity principle, converting the height distribution estimation result from the height distribution to the depth distribution to obtain a converted first depth distribution result; fusing the first depth distribution result with the depth distribution estimation result to obtain a fused second depth distribution result; performing post-processing on the second depth distribution result based on the post-processing network in the target detection model to obtain a target detection result of the training image, and after calculating the loss function of the target detection model based on the target detection result, updating the parameters of the target detection model; After the target detection model is trained, the cloud data center sends the target detection model to the roadside edge platform and deploys it, so that when the roadside edge platform receives the road scene image to be detected, the target detection result output by the target detection model is obtained based on the target detection model.

Images