Target detection method and device, equipment and storage medium

Abstract

The invention provides a target detection method and device, equipment and a storage medium, and relates to the technical field of target detection. The method comprises the following steps: mapping each collected image in a sample image set to a Riemannian space to obtain a Riemannian mean value corresponding to each type of road target; and taking the sample image set and each Riemannian mean value as training data, inputting the training data into a convolutional neural network for training, and taking the trained convolutional neural network as a target detection model. In the real-time image processing process, inputting the real-time image into the backbone network after weighted training in the target detection model to obtain target feature information of the real-time image, and obtaining the target feature information of the real-time image based on the target feature information; and the head network outputs the type and position information used for indicating each target in the real-time image after the soft non-maximum suppression processing based on the Riemannian distance. The feature extraction capability of the target features is improved by weighting the convolution kernel in the backbone network and improving soft non-maximum suppression through the Riemannian distance, the target detection precision is improved, and meanwhile, the detection time is not increased.

Description

Technical Field

[0001] The present application relates to the field of identification technology, and in particular to a target detection method, device, equipment and storage medium. Background Art

[0002] In the field of intelligent driving, accurate perception of the driving environment is crucial. Given that there are many types of targets in driving scenes, such as vehicles, pedestrians, and traffic lights, and these targets have different shapes and sizes, existing solutions generally use deep learning solutions to achieve road target detection.

[0003] The target detection models based on deep learning can be mainly divided into two categories, including single-stage detection networks based on Euclidean space and two-stage detection networks. The single-stage target detection algorithm uses regression to directly predict the classification probability and position coordinates of the target through the convolutional network. This method does not need to generate candidate areas, so the detection speed is fast, but the detection accuracy is insufficient. In contrast, the two-stage detection algorithm first extracts a large number of anchor boxes on the feature map generated by the feature extraction network, and locks the anchor box containing the target location information. However, due to the large number of anchor boxes, although its detection accuracy is significantly improved compared with the single-stage algorithm, the detection speed is relatively reduced.

[0004] Therefore, the existing technology is still lacking in detection accuracy and detection speed when detecting road targets. Summary of the invention

[0005] The present application provides a target detection method, device, equipment and storage medium to solve the problem that the prior art is still lacking in detection accuracy and detection speed when detecting road targets.

[0006] In a first aspect, the present application provides a target detection method, comprising:

[0007] Acquire a sample image set, the sample image set comprising a plurality of collected images, the collected images being used to indicate various types of road targets pre-collected by a vehicle sensor, and map each of the collected images to a Riemann space to obtain a Riemann mean corresponding to each type of road target, and use the sample image set and each of the Riemann means as training data;

[0008] Constructing a convolutional neural network, inputting the training data into the convolutional neural network for training, and using the trained convolutional neural network as a target detection model, wherein the target detection model includes a backbone network, a neck network, and a head network, wherein the target weight of each convolution kernel of the weighted feature layer in the backbone network is obtained according to each of the Riemann means;

[0009] A real-time image is acquired and input into the backbone network, and target feature information of the real-time image is collected through the target weights of each convolution kernel. Based on the target feature information, an anchor box and a confidence level are output through the head network, wherein the anchor box is used to indicate the type and position of each target in the real-time image.

[0010] In a possible design, mapping each of the acquired images to a Riemann space includes:

[0011] Extracting target areas in each collected image, scaling target areas of the same type to the same size, wherein the scaled size is determined according to the cluster center of the scale of each target area;

[0012] Acquire a covariance matrix of the target area in each of the scaled acquired images, wherein the covariance matrix is ​​used to indicate feature information of the target area in each of the acquired images;

[0013] Regularizing each of the covariance matrices to obtain a corresponding symmetric positive definite matrix, and mapping each of the symmetric positive definite matrices to the Riemann space to obtain a plurality of different types of road target sample point groups in the Riemann space;

[0014] The road target sample point group includes a plurality of road target Riemann mapping points of the same type, and the corresponding Riemann mean is obtained according to each of the road target sample point groups.

[0015] In a possible design, obtaining the corresponding Riemann mean according to each of the road target sample point groups includes:

[0016] By clustering each of the road target sample point groups to obtain a Riemann mean, the Riemann mean indicates a sample center point corresponding to the road target sample point group;

[0017] In the process of training the convolutional neural network, an image in a feature channel of a feature layer of a backbone network is obtained, and a target weight is calculated by calculating the Riemann distance between a target region extracted by a feature channel in each feature layer of the backbone network and a Riemann mean of the same type, and the feature channel is weighted;

[0018] Among them, each of the target weights corresponds to the road target type indicated by each of the road target sample point groups, and the target weight is positively correlated with the ability of the feature channel in each feature layer of the backbone network to extract the target area.

[0019] In a possible design, obtaining the target weight value includes:

[0020] In the process of training the convolutional neural network, a target area in a feature map extracted by a feature channel of a feature layer of a backbone network is obtained through the scale invariance of the convolutional network, a target Riemann distance between the extracted target area and a Riemann mean of the same type is calculated, and the feature channel is weighted, wherein the target Riemann distance is used to indicate the similarity between the extracted target area and the Riemann mean of the same type;

[0021] According to the target Riemann distance, the weight of each feature channel in the feature layer of the backbone network is calculated, and the weight is applied to each feature channel to obtain the weighted adjusted feature channel parameters.

[0022] In a possible design, the step of calculating the weights of feature channels in each feature layer of the backbone network and acting on the feature channels includes:

[0023] Normalizing each of the target Riemann distances to obtain a plurality of normalized Riemann distances, obtaining corresponding weights according to each of the normalized Riemann distances, and using the weights of the corresponding feature channels in the feature layer of the backbone network;

[0024] In a possible design, outputting an anchor box and a confidence level through the head network based on the target feature information includes:

[0025] The neck network is used to fuse the feature information of each target to obtain feature fusion information. Based on the feature fusion information, the head network generates a candidate frame set for each road target and gives a confidence level. Each candidate frame set refers to a road target in the image and gives multiple candidate frames.

[0026] Through the soft non-maximum suppression algorithm, the candidate boxes belonging to the same target are screened in each candidate box set to obtain the anchor box of each road target.

[0027] In a possible design, obtaining the anchor frame of each road target in each candidate frame set by a soft non-maximum suppression algorithm includes:

[0028] Obtaining the confidence of each candidate frame in the candidate frame set, and obtaining a target candidate frame and a candidate frame to be processed according to each confidence, wherein the target candidate frame is used to indicate a candidate frame with the highest confidence, and the candidate frame to be processed is used to indicate a candidate frame whose intersection-over-union ratio with the target candidate frame is greater than a threshold;

[0029] Mapping the target candidate box and the candidate box to be processed to the Riemann space to obtain a candidate Riemann distance, where the candidate Riemann distance is used to indicate the distance between the target candidate box and the candidate box to be processed and each corresponding mapping point in the Riemann space, and obtaining an adjusted confidence according to the candidate Riemann distance, and using the adjusted confidence as a new confidence of the candidate box to be processed, so as to update the confidence of each candidate box in the candidate box set;

[0030] Based on the iteratively updated candidate box set, the candidate box corresponding to the highest confidence is used as the anchor box.

[0031] In a second aspect, the present application provides a target detection device, comprising:

[0032] An acquisition module is used to acquire a sample image set, wherein the sample image set includes a plurality of collected images, wherein the collected images are used to indicate various types of road targets pre-collected by a vehicle sensor, and each of the collected images is mapped to a Riemann space to obtain a Riemann mean corresponding to each type of road target, and the sample image set and each of the Riemann means are used as training data;

[0033] A processing module, used to construct a convolutional neural network, input the training data into the convolutional neural network for training, and use the trained convolutional neural network as a target detection model, wherein the target detection model includes a backbone network, a neck network, and a head network, wherein the target weight of each convolution kernel of the weighted feature layer in the backbone network is obtained according to each of the Riemann means;

[0034] An execution module is used to acquire a real-time image, input the real-time image into the backbone network, and collect target feature information of the real-time image through the target weights of each convolution kernel, and output an anchor box and confidence through the head network based on the target feature information, wherein the anchor box is used to indicate the type and position of each target in the real-time image.

[0035] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0036] The memory stores computer-executable instructions;

[0037] The processor executes the computer-executable instructions stored in the memory to implement a target detection method.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to implement a target detection method when executed by a processor.

[0039] The target detection method, device, equipment and storage medium provided by the present application map each collected image in the sample image set to the Riemann space to obtain the Riemann mean corresponding to each type of road target, use the sample image set and each Riemann mean as training data, input it into a preset convolutional neural network for training, and use the trained convolutional neural network as a target detection model. In the real-time image processing process, the real-time image is input into the backbone network of the target detection model, and the target feature information of the real-time image is collected through the weighted trained backbone network. Based on the target feature information, the head network outputs the type and position information of each target in the real-time image after the soft non-maximum suppression processing based on the Riemann distance. By calculating the Riemann distance to weight some convolution kernels in the backbone network and improving the soft non-maximum suppression, the feature extraction capability of the target feature is improved, and the target detection accuracy is improved, while not increasing the detection time. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] Figure 1 Schematic diagram of the target detection method provided in the embodiment of the present application Figure 1 ;

[0042] Figure 2 Schematic diagram of the target detection method provided in the embodiment of the present application Figure 2 ;

[0043] Figure 3 A schematic diagram of the geometric relationship of the image Riemann space provided in the embodiment of the present application;

[0044] Figure 4 A schematic diagram of the structure of a target detection device provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0046] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0048] In driving scenarios, existing solutions generally detect road targets through deep learning-based target detection models, and target detection models include single-stage detection networks and two-stage detection networks. The single-stage detection network is represented by YOLO, and the two-stage detection network is based on the Faster RCNN paradigm.

[0049] Existing solutions usually focus on Euclidean space when detecting road targets through deep learning networks. Single-stage detection networks and two-stage detection algorithms based on Euclidean space detect road targets on feature maps generated by feature extraction networks and extract corresponding anchor frames containing target location information. However, there are trade-offs in detection accuracy and detection speed. The description of image data in non-Euclidean space such as Riemann space also contains rich useful information. When mapping the image to Riemann space to obtain the corresponding Riemann manifold, the Riemann manifold can represent the feature structure of the image as a geometric structure in the Riemann space, and based on the geometric structure in the Riemann space, the corresponding Riemann distance is obtained, and the spatial characteristics of the image are represented by the geometric relationship indicated by the Riemann distance.

[0050] Therefore, if based on the Riemann manifold, by calculating the Riemann distance between the target area extracted by the feature channel in each feature layer of the backbone network and the Riemann mean of the same type, and weighting the convolution kernel corresponding to each feature channel, the ability to extract target features in the input image can be enhanced without increasing the detection time. Soft non-maximum suppression based on the Riemann metric is used in the candidate box screening process, and the Riemann distance is applied as a parameter to the function of adjusting the confidence of soft non-maximum suppression, which can improve the situation where the detection effect of different overlapping targets is not ideal, and at the same time improve the detection accuracy.

[0051] The present application provides a target detection method, which maps each collected image in a sample image set to a Riemann space to obtain the Riemann mean corresponding to each type of road target, uses the sample image set and each Riemann mean as training data, inputs them into a preset convolutional neural network for training, and uses the trained convolutional neural network as a target detection model. In the real-time image processing process, the real-time image is input into the backbone network of the target detection model, and the target feature information of the real-time image is collected through the weighted trained backbone network. Based on the target feature information, the head network outputs the type and position information of each target in the real-time image after the soft non-maximum suppression processing based on the Riemann distance. By calculating the Riemann distance to weight some convolution kernels in the backbone network and improving the soft non-maximum suppression, the feature extraction capability of the target feature is improved, and the target detection accuracy is improved without increasing the detection time.

[0052] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0053] Embodiment 1

[0054] Figure 1 Schematic diagram of the target detection method provided in the embodiment of the present application Figure 1 .like Figure 1 As shown, the method includes:

[0055] S101, obtaining a sample image set, wherein the sample image set includes a plurality of collected images, wherein the collected images are used to indicate various types of road targets pre-collected by a vehicle sensor, and each of the collected images is mapped to a Riemann space to obtain a Riemann mean corresponding to each type of road target, and the sample image set and each of the Riemann means are used as training data;

[0056] Specifically, the original image is acquired through the vehicle's image capture device, that is, the vehicle sensor such as a high-definition camera and / or an infrared sensor, and the original image is preprocessed such as denoising, enhancement and normalization to improve the image quality, and the preprocessed original image is used as the acquired image, thereby obtaining a sample image set including multiple acquired images, wherein each acquired image contains various types of road targets such as pedestrians, vehicles, road signs, etc., and annotation information of each type of road target, and the type and position coordinates of each road target are indicated by the annotation information. The vehicle's image capture device can be installed on the front, rear and side of the vehicle to provide an all-round field of view.

[0057] S102, constructing a convolutional neural network, inputting the training data into the convolutional neural network for training, and using the trained convolutional neural network as a target detection model, wherein the target detection model includes a backbone network, a neck network, and a head network, wherein the target weight of each convolution kernel of the weighted feature layer in the backbone network is obtained according to each of the Riemann means;

[0058] Specifically, after mapping each type of road target in each collected image to the Riemann space and obtaining the Riemann mean corresponding to each type of road target, each Riemann mean and the sample image set are used as training data to train the convolutional neural network to obtain a target detection model, wherein when training the convolutional neural network, a target weight set including multiple target weights is obtained;

[0059] Among them, the backbone network is used to extract the features of the input image, and is composed of a convolution layer, an activation layer, a normalization layer, a pooling layer, etc. Through operations such as convolution and pooling, the features extracted by the backbone network include low-level features such as edges, textures, etc., as well as high-level features such as shapes. The backbone network is divided into different feature layers according to the hierarchy of features and the needs of multi-scale feature fusion. Each feature layer contains multiple feature channels for extracting the features of the input image. The target features extracted by the feature channel are extracted, and the Riemann distance between the extracted target features and the corresponding Riemann mean is calculated, and the feature channels are weighted. The Riemann distance is used to indicate the distance between the Riemann mapping point of the road target extracted by the backbone network in the Riemann space and the Riemann mean of the corresponding type;

[0060] Furthermore, each feature layer extracts a feature image, and the convolution kernel slides on the feature image and performs a convolution operation to generate a new feature channel. Each channel corresponds to the features learned by a specific convolution kernel. The parameters of each convolution kernel are weighted through the target weight to enhance the contribution of the feature channel for extracting the road target of interest in the network, reduce the response of the feature channel other than the features of the road target, and promote the convergence of the network.

[0061] S103, acquiring a real-time image, and inputting the real-time image into the backbone network, and collecting target feature information of the real-time image through the target weights of each convolution kernel, and outputting an anchor box and a confidence level through the head network based on the target feature information, wherein the anchor box is used to indicate the type and position of each target in the real-time image.

[0062] Specifically, after obtaining the target detection model, the current image is collected through the vehicle sensor, and the preprocessed image is used as the real-time image, and the real-time image is input into the backbone network in the target detection model, and the corresponding convolution kernel is used to extract the corresponding type of road target feature information in the real-time image. The neck network performs multi-scale feature fusion on the multiple road target feature information extracted by the backbone network, and in the head network, the soft non-maximum suppression algorithm based on the Riemann distance is used to output the anchor box of the corresponding road target to avoid the problem of targets being ignored due to the overlap of similar road targets such as pedestrians and pedestrians, or vehicles and vehicles during detection.

[0063] The method provided in this embodiment maps the target area of ​​each collected image in the sample image set to the Riemann space to obtain the Riemann mean corresponding to each type of road target, uses the sample image set and each Riemann mean as training data, inputs them into a preset convolutional neural network for training, and uses the trained convolutional neural network as a target detection model. In the real-time image processing process, the real-time image is input into the backbone network of the target detection model, and the target feature information of the real-time image is collected through the weighted trained backbone network. Based on the target feature information, the head network outputs the type and position information of each target in the real-time image after the soft non-maximum suppression processing based on the Riemann distance. By calculating the Riemann distance to weight some convolution kernels in the backbone network and improving the soft non-maximum suppression, the feature extraction capability of the target feature is improved, and the target detection accuracy is improved, while not increasing the detection time.

[0064] The target detection method of the present application is described in detail using a specific embodiment below.

[0065] Embodiment 2

[0066] Figure 2 Schematic diagram of the target detection method provided in the embodiment of the present application Figure 2 . Figure 3 A schematic diagram of the geometric relationship of the image Riemann space provided in the embodiment of the present application. Figure 2 and Figure 3 As shown, the method includes:

[0067] S201, obtaining a sample image set, the sample image set including a plurality of collected images, obtaining a covariance matrix of a target area in each of the collected images, the covariance matrix being used to indicate feature information of each of the collected images;

[0068] Specifically, after acquiring the sample image set through the vehicle's image capture device, before mapping the sample image set to the Riemann space, the target area needs to be scaled to the same size. The sample image to be mapped to the Riemann space is the target area in each image in the sample image set, referred to as the sample image. The sample images are clustered in size, and the scale of the cluster center is selected as the benchmark, and the sample images are scaled to the scale of the cluster center. The covariance matrix corresponding to each sample image is obtained. For images of size c×d, the covariance matrix of each sample image is obtained by the following formula:

[0069]

[0070] Among them, P R is the covariance matrix of the sample image, is the mean of the feature image corresponding to the sample image, {z i} i＝1...d For each column of data in the sample image, Obtained through the following formula:

[0071]

[0072] Among them, {z i} i＝1...d is each column of data in the sample image, and d is the number of columns in the sample image.

[0073] S202, performing regularization processing on each of the covariance matrices to obtain a corresponding symmetric positive definite matrix, and mapping each of the symmetric positive definite matrices to the Riemann space to obtain a plurality of different types of road target sample point groups in the Riemann space;

[0074] Specifically, after obtaining the covariance matrix of each sample image, it is also necessary to regularize the covariance matrix of each sample image to meet the symmetric positive definite conditions, so that the sample image can be obtained in the Riemann space. On the corresponding symmetric positive definite matrix Riemann manifold, the covariance matrix of the sample image becomes a c×c symmetric positive definite matrix after regularization, which is obtained by the following formula:

[0075] P R =P R +λ t trace(P R )A

[0076] Among them, λ tis the regularization parameter, A is the identity matrix, and trace() is the trace of the covariance matrix. Based on this transformation, the sample image can be regarded as a point on the symmetric positive definite matrix Riemann manifold after the covariance transformation. Therefore, the relationship between the feature information contained in the image can be represented by the geometric relationship of its covariance matrix on the Riemann manifold.

[0077] Furthermore, after each symmetric positive definite matrix is ​​mapped to the Riemann space, multiple different types of road target sample point groups are mapped in the Riemann space, and the road target sample point groups include multiple road target Riemann mapping points of the same type, that is, sample points of the same type, such as the first type of sample points.

[0078] S203, obtaining the corresponding Riemann mean according to each of the road target sample point groups, using the sample image set and each of the Riemann means as training data, and inputting the training data into a convolutional neural network for training;

[0079] Specifically, after obtaining a plurality of different types of road target sample point groups, in each road target sample point group, each road target sample point group is clustered to obtain the Riemann mean corresponding to each road target, and the Riemann mean indicates the sample center point of the corresponding road target sample point group. In the process of training the convolutional neural network, based on each Riemann mean, a target weight value set is obtained, and the target weight value set includes a plurality of target weights, and each target weight acts on the corresponding feature channel convolution kernel;

[0080] Among them, the target weight is positively correlated with the ability of the feature channel in the corresponding backbone network feature layer to extract the target feature. For the Riemann mean, it is obtained by the following formula:

[0081]

[0082] in, is the Riemann mean of the k-th detection target, with a size of c p ×c p , n k is the number of k-th type of road targets, δ R is the Riemann distance between any two points in the Riemann mapping points of the same type of road targets. The Riemann mean indicates the cluster center corresponding to each road target sample point group, that is, the sample center point of each road target sample point group.

[0083] S204, based on each of the road target sample point groups, obtaining a plurality of target Riemann distances, the target Riemann distance being used to indicate the Riemann distance between the target feature extracted by the feature channel and the Riemann mean;

[0084] Specifically, after obtaining the Riemann mean corresponding to each road target sample point group, during the training process, the target features extracted from each feature channel in the feature layer of the backbone network are obtained according to scale invariance and annotation files, the target features are scaled to the scale of the same type of Riemann mean, and mapped to the Riemann space, and the Riemann distance between the target features and the same type of Riemann mean is calculated. Among them, the smaller the target Riemann distance, that is, the closer the distance between the target feature and the corresponding type of Riemann mean, the more similar the target features extracted by the corresponding feature channel in the feature layer of the backbone network are to the feature information of the corresponding type of road target;

[0085] Furthermore, the target Riemann distance is obtained by the following formula:

[0086]

[0087] Among them, δ R (X,Y) is the target Riemann distance between the road target Riemann mapping point X and the Riemann mean Y, c is the number of columns of the symmetric positive definite matrix, and λ is the -1 The eigenvalues ​​of Y.

[0088] S205, normalizing each of the target Riemann distances to obtain a plurality of normalized Riemann distances, obtaining corresponding weights according to each of the normalized Riemann distances, and applying the weight values ​​to feature channels corresponding to the target features;

[0089] Specifically, after obtaining the Riemann distance of each target, each target Riemann distance is normalized to obtain multiple normalized Riemann distances. The normalized Riemann distance is obtained by the following formula:

[0090]

[0091] Among them, δ Rm is the target Riemann distance, min(δ Rm ) is the minimum value in the target Riemann distance, max(δ Rm ) is the maximum value of the target Riemann distance, is the normalized Riemann distance. After obtaining each normalized Riemann distance, the target weight W is obtained by the following formula:

[0092]

[0093] in, The normalized Riemann distance is used so that when the distance between the target feature extracted from the feature channel and the target Riemann mean of the same type is small, a larger weight is added to the convolution kernel corresponding to this feature channel, and when the Riemann distance is large, a smaller weight is added. By adding weights, the contribution of the feature channel that extracts the target feature of interest in the network is enhanced, the response of the feature channel other than the target feature is reduced, the convergence of the network is promoted, and it also has a positive effect on the subsequent feature fusion.

[0094] S206, using the trained convolutional neural network as a target detection model;

[0095] Specifically, when constructing a convolutional neural network, for example, when selecting YOLO V5 as the basic framework for training to obtain a target detection model, the target detection model includes a backbone network, a neck network, and a head network. The backbone network is used to extract target features, i.e., a feature extraction network; the neck network is used to perform multi-scale feature fusion on multiple feature layers extracted by the backbone network; and the head network is used to predict the type and location of road targets, i.e., to obtain the anchor box of the road target.

[0096] Furthermore, when ResNet50 is selected as the backbone network, the following discrimination method can be used to obtain which layer of ResNet50 the weighted feature layer is set to. N images are randomly selected from each target image and sent to the feature extraction network to obtain the feature maps extracted from each feature layer. The average and discreteness of the Riemann distance between the target feature and the corresponding Riemann mean in the feature channel of each layer are calculated, and the appropriate weighted layer is selected according to the size of the Riemann distance and discreteness. The higher the number of layers, the larger the Riemann distance and the larger the discreteness. The features learned in the high-level feature map are relatively abstract, integrating all the feature information in the image, and some small target feature information is lost after pooling, resulting in a large Riemann distance. When determining the weighting mechanism during the training process, the feature image with a higher number of layers will have a larger scale when scaled to the same scale as the Riemann mean of the same target. Therefore, affected by the scaling process, its standard deviation is relatively large and is not suitable for weighting. In addition, using it in the low layer can also play a positive role in adjusting the parameters of the subsequent convolutional layer to a large extent. Therefore, the C2 layer is used as the weighted feature layer;

[0097] S207, acquiring a real-time image, and inputting the real-time image into the backbone network, and collecting target feature information of the real-time image through each weighted trained backbone network;

[0098] Specifically, after obtaining the trained target detection model, the real-time image collected by the current vehicle is obtained, and the feature information in the real-time image is extracted in the backbone network of the target detection model. The weighting mechanism based on Riemann distance plays a key role in the training process. It is used to fine-tune the parameters of the feature channel in the backbone network and enhance the ability to extract target features in the input image without increasing the detection time, thereby improving the detection performance of the target detection model.

[0099] S208, fusing feature information of each target through the neck network to obtain feature fusion information, and generating a candidate frame set of each road target through the head network according to the feature fusion information, each candidate frame set including multiple candidate frames;

[0100] Specifically, after obtaining the feature information of each target through the backbone network, the feature information of each target is fused through the neck network in the target detection model, and the fused feature fusion information is passed through the head network to generate corresponding anchor boxes for each type of road target. When the head network generates the anchor box of each road target, it usually generates multiple candidate boxes at each road target position, and uses the non-maximum suppression algorithm (NMS) to exclude candidate boxes belonging to the same road target.

[0101] Among them, the processing method of NMS is to sort each candidate box according to the confidence, calculate the ratio of the intersection and union (IoU) of the remaining candidate boxes and the candidate box with the highest confidence, and if it is greater than the set threshold, its confidence will be set to zero, and the corresponding area will be regarded as having no road target. The problem of overlapping detection targets such as pedestrians and vehicles often occurs in road target detection, and the zeroing operation will cause the overlapping targets to be ignored;

[0102] Furthermore, in order to avoid ignoring overlapping targets, soft non-maximum suppression (Soft-NMS) is used to improve this situation. Soft-NMS does not directly set the confidence of candidate boxes greater than the set threshold to zero, but uses a function about IoU to adjust it so that the more overlapping candidate boxes, the lower the confidence, and the candidate boxes with low overlap have almost no effect. The soft non-maximum suppression function is shown in the following formula:

[0103]

[0104] Among them, g h is the candidate box with the highest confidence, g b For h Overlapping candidate boxes, σ is the scale factor, S i is the confidence of the candidate box, Adjusted confidence for the soft non-maximum suppression method.

[0105] S209, obtaining the confidence of each candidate box in the candidate box set, and obtaining a target candidate box and a candidate box to be processed according to each confidence;

[0106] Specifically, the above Soft-NMS does not have feature information to distinguish candidate boxes for overlapping areas, and only adjusts the confidence based on IoU. In view of this, a soft non-maximum suppression method based on Riemann distance is proposed. The Riemann distance is used to judge the correlation degree of overlapping areas to distinguish the overlap of different targets. b The size of the target candidate box g is scaled to h Corresponding scales, map the two regions onto the Riemann manifold and obtain their Riemann distance as the candidate Riemann distance;

[0107] Furthermore, the target candidate box is used to indicate the candidate box with the highest confidence, and the candidate box to be processed is used to indicate the candidate box whose intersection-over-union ratio with the target candidate box is greater than a threshold. The target candidate box and the candidate box to be processed are mapped to the Riemann space to obtain the candidate Riemann distance, and the candidate Riemann distance is used to indicate the distance between the target candidate box and the candidate box to be processed, and each corresponding mapping point in the Riemann space.

[0108] S210, obtaining an adjusted confidence according to the candidate Riemann distance, and using the adjusted confidence as a new confidence of the candidate frame to be processed, so as to update the confidence of each candidate frame in the candidate frame set, and then obtaining an anchor frame based on the updated candidate frame set;

[0109] Specifically, after obtaining the candidate Riemann distance, the new confidence of the candidate box to be processed is obtained based on the adjustment function of the candidate Riemann distance and the soft non-maximum suppression function. The adjustment function is shown in the following formula;

[0110]

[0111] Among them, η is the Riemann distance proportional coefficient, is the candidate Riemann distance, g h is the candidate box with the highest confidence, g b For h Overlapping candidate boxes, i.e. candidate boxes to be processed, The confidence adjusted by the soft non-maximum suppression method based on the candidate Riemann distance is the new confidence of the candidate frame to be processed. After obtaining the new confidence of each candidate frame to be processed in the candidate frame set, the confidence of each candidate frame in the candidate frame set is updated. Based on the updated candidate frame set, the candidate frame corresponding to the highest confidence is used as the anchor frame through the maximum suppression method.

[0112] Among them, the proportional coefficient σ and the Riemann distance proportional coefficient η, the relationship curve between the proportional coefficient σ and the detection accuracy is obtained through experiments, σ∈[0.1,1]. As the proportional coefficient σ increases, the detection effect is better. After reaching a certain threshold, as the proportional coefficient σ increases, the detection effect decreases. The proportional coefficient σ corresponding to the highest detection accuracy is obtained. The Riemann distance proportional coefficient η is used to adjust the Riemann distance to a suitable order of magnitude. According to the Riemann distance, the value range of the Riemann distance proportional coefficient η is set, such as η∈{10 0 , 10 1 , 10 2 , 10 3 , 10 4 , 10 5}, respectively conduct experiments on the influence of the Riemann distance ratio coefficient η on the detection accuracy. When η is larger, the coefficient of the confidence obtained by the soft non-maximum suppression function based on the candidate Riemann distance is closer to 1, resulting in more candidate boxes that are not adjusted, causing the maximum suppression to not work and reducing the detection accuracy; when it is smaller, the adjustment item containing the Riemann distance is closer to 1, the Riemann distance cannot play a regulatory role, and the detection effect is closer to the soft non-maximum suppression. According to the experimental results, the value of the Riemann distance ratio coefficient η corresponding to the best detection effect is obtained;

[0113] Furthermore, through the soft non-maximum suppression function based on the candidate Riemann distance, if the Riemann distance between two overlapping candidate box areas is small, it means that the features of the two candidate box areas are similar, that is, the two candidate box areas are likely to belong to the same target, and the confidence of this area is set to be small; if the Riemann distance between two overlapping candidate box areas is large, it means that although the two candidate box areas overlap, the features are not similar, that is, the two areas are likely to belong to different targets, and the confidence of this candidate box area is set to be large to prevent it from being set to zero. Therefore, by adding the parameters of the Riemann distance, the suppression of targets when overlapping is improved, the suppression effect on the candidate areas that should be set to zero is enhanced, and the overlapping road targets are avoided from being ignored.

[0114] The method provided in this embodiment maps each collected image in a sample image set to a Riemann space to obtain the Riemann means corresponding to each type of road target, uses the sample image set and each Riemann means as training data, inputs them into a preset convolutional neural network for training, and uses the trained convolutional neural network as a target detection model, wherein the target detection model includes a backbone network, a neck network, and a head network, wherein after obtaining the target weights of each convolution kernel in the weighted feature layer in the backbone network according to each Riemann means, a real-time image is obtained, and the real-time image is input into the backbone network, and target feature information of the real-time image is collected through the target weights of each convolution kernel, and based on the target feature information, an anchor box and a confidence are output through the head network, and the anchor box is used to indicate the type and position of each target in the real-time image, thereby improving the target feature extraction capability and detection capability through the target weights and anchor boxes obtained based on the Riemann space, while not increasing the detection time.

[0115] The embodiment of the present invention can divide the electronic device or the main control device into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0116] Figure 4 This is a schematic diagram of the structure of the target detection device provided in the embodiment of the present application. Figure 4 As shown, the device 40 includes:

[0117] An acquisition module 401 is used to acquire a sample image set, wherein the sample image set includes a plurality of collected images, wherein the collected images are used to indicate various types of road targets pre-collected by a vehicle sensor, and each of the collected images is mapped to a Riemann space to obtain a Riemann mean corresponding to each type of road target, and the sample image set and each of the Riemann means are used as training numbers;

[0118] The processing module 402 is used to construct a convolutional neural network, input the training data into the convolutional neural network for training, and use the trained convolutional neural network as a target detection model, wherein the target detection model includes a backbone network, a neck network, and a head network, wherein during the training process, the target weight of each convolution kernel of the weighted feature layer in the backbone network is obtained according to each Riemann mean;

[0119] The execution module 403 is used to obtain a real-time image, input the real-time image into the backbone network, and collect target feature information of the real-time image through the target weights of each convolution kernel, and output an anchor box and confidence through the head network based on the target feature information, wherein the anchor box is used to indicate the type and position of each target in the real-time image.

[0120] Furthermore, the acquisition module 401 is specifically used to extract the target area in each collected image, and the target areas of the same type are scaled to the same size, and the scaled size is determined according to the cluster center of the scale of each target area;

[0121] Acquire a covariance matrix of each of the scaled target regions, where the covariance matrix is ​​used to indicate feature information of each of the target regions;

[0122] Regularizing each of the covariance matrices to obtain a corresponding symmetric positive definite matrix, and mapping each of the symmetric positive definite matrices to the Riemann space to obtain a plurality of different types of road target sample point groups in the Riemann space;

[0123] The road target sample point group includes a plurality of road target Riemann mapping points of the same type, and the corresponding Riemann mean is obtained according to each of the road target sample point groups.

[0124] Furthermore, the acquisition module 401 is specifically used to obtain the Riemann mean by clustering each of the road target sample point groups, and indicate the sample center point corresponding to the road target sample point group through the Riemann mean;

[0125] In the process of training the convolutional neural network, images in the feature channels of the backbone network feature layers are obtained, and the target weights are calculated by calculating the Riemann distance between the target areas extracted by the feature channels in each feature layer of the backbone network and the Riemann means of the same type, and then the feature channels are weighted.

[0126] Among them, each of the target weights corresponds to the road target type indicated by each of the road target sample point groups, and the target weight is positively correlated with the ability of the feature channel in each feature layer of the backbone network to extract the target area.

[0127] Further, the processing module 402 is specifically used to obtain the target area in the feature map extracted by the feature channel of the feature layer of the backbone network through the scale invariance of the convolutional network during the training of the convolutional neural network, calculate the target Riemann distance between the extracted target area and the Riemann mean of the same type, and weight the feature channel, wherein the target Riemann distance is used to indicate the similarity between the extracted target area and the Riemann mean of the same type;

[0128] According to the target Riemann distance, the weight of each feature channel in the feature layer of the backbone network is calculated, and the weight is applied to each feature channel to obtain the weighted adjusted feature channel parameters.

[0129] Furthermore, the processing module 402 is specifically used to normalize each of the target Riemann distances to obtain a plurality of normalized Riemann distances, obtain corresponding weights according to each of the normalized Riemann distances, and use them as weights of corresponding feature channels in the feature layer of the backbone network.

[0130] Further, the execution module 403 is specifically used to fuse the feature information of each target through the neck network to obtain feature fusion information. According to the feature fusion information, the head network generates a candidate box set for each road target and gives a confidence, and each candidate box set refers to a road target in the image and gives multiple candidate boxes;

[0131] Through the soft non-maximum suppression algorithm, the candidate boxes belonging to the same target are screened in each candidate box set to obtain the anchor box of each road target.

[0132] Further, the execution module 403 is specifically used to obtain the confidence of each candidate box in the candidate box set, and obtain the target candidate box and the candidate box to be processed according to each confidence, wherein the target candidate box is used to indicate the candidate box with the highest confidence, and the candidate box to be processed is used to indicate the candidate box whose intersection-over-union ratio with the target candidate box is greater than a threshold;

[0133] Mapping the target candidate box and the candidate box to be processed to the Riemann space to obtain a candidate Riemann distance, where the candidate Riemann distance is used to indicate the distance between the target candidate box and the candidate box to be processed and each corresponding mapping point in the Riemann space, and obtaining an adjusted confidence according to the candidate Riemann distance, and using the adjusted confidence as a new confidence of the candidate box to be processed, so as to update the confidence of each candidate box in the candidate box set;

[0134] Based on the iteratively updated candidate box set, the candidate box corresponding to the highest confidence is used as the anchor box.

[0135] The target detection device provided in this embodiment can execute the target detection method of the above embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0136] In the specific implementation of the aforementioned target detection device, each module may be implemented as a processor, and the processor may execute computer-executable instructions stored in the memory, so that the processor executes the aforementioned target detection method.

[0137] Figure 5This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 50 includes: at least one processor 501 and a memory 502. The electronic device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0138] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the target detection method executed by the electronic device side as above.

[0139] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0140] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0141] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage.

[0142] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0143] The above-mentioned functions implemented by the electronic device and the main control device introduce the scheme provided by the embodiment of the present invention. It can be understood that in order to implement the above-mentioned functions, the electronic device or the main control device includes a hardware structure and / or software module corresponding to the execution of each function. In combination with the units and algorithm steps of each example described in the embodiment disclosed in the embodiment of the present invention, the embodiment of the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present invention.

[0144] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above target detection method is implemented.

[0145] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0146] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in an electronic device or a main control device.

[0147] The present application also provides a computer program product, which includes: a computer program, which is stored in a readable storage medium. At least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0148] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target detection method, characterized in that: The method comprises: Acquire a sample image set, the sample image set comprising a plurality of collected images, the collected images being used to indicate various types of road targets pre-collected by a vehicle sensor, and map each of the collected images to a Riemann space to obtain a Riemann mean corresponding to each type of road target, and use the sample image set and each of the Riemann means as training data; Constructing a convolutional neural network, inputting the training data into the convolutional neural network for training, and using the trained convolutional neural network as a target detection model, wherein the target detection model includes a backbone network, a neck network, and a head network, wherein the target weight of each convolution kernel of the weighted feature layer in the backbone network is obtained according to each of the Riemann means; A real-time image is acquired and input into the backbone network, and target feature information of the real-time image is collected through the target weights of each convolution kernel. Based on the target feature information, an anchor box and a confidence level are output through the head network, wherein the anchor box is used to indicate the type and position of each target in the real-time image.

2. The method according to claim 1, characterized in that Mapping the target area in each of the acquired images to the Riemann space includes: Extracting target areas in each collected image, scaling the target areas of the same type to the same size, and determining the scaled size according to the cluster center of the scale of each target area; Acquire a covariance matrix of each of the scaled target regions, where the covariance matrix is ​​used to indicate feature information of each of the target regions; Regularizing each of the covariance matrices to obtain a corresponding symmetric positive definite matrix, and mapping each of the symmetric positive definite matrices to the Riemann space to obtain a plurality of different types of road target sample point groups in the Riemann space; The road target sample point group includes a plurality of road target Riemann mapping points of the same type, and the corresponding Riemann mean is obtained according to each of the road target sample point groups.

3. The method according to claim 2, characterized in that The obtaining the corresponding Riemann mean according to each of the road target sample point groups includes: By clustering each of the road target sample point groups to obtain a Riemann mean, the Riemann mean indicates a sample center point corresponding to the road target sample point group; In the process of training the convolutional neural network, an image in a feature channel of a feature layer of a backbone network is obtained, and a target weight is calculated by calculating the Riemann distance between a target region extracted by a feature channel in each feature layer of the backbone network and a Riemann mean of the same type, and the feature channel is weighted; Among them, each of the target weights corresponds to the road target type indicated by each of the road target sample point groups, and the target weight is positively correlated with the ability of the feature channel in each feature layer of the backbone network to extract the target area.

4. The method according to claim 3, characterized in that The obtaining of the target weight value comprises: In the process of training the convolutional neural network, a target area in a feature map extracted by a feature channel of a feature layer of a backbone network is obtained through the scale invariance of the convolutional network, a target Riemann distance between the extracted target area and a Riemann mean of the same type is calculated, and the feature channel is weighted, wherein the target Riemann distance is used to indicate the similarity between the extracted target area and the Riemann mean of the same type; According to the target Riemann distance, the weight of each feature channel in the feature layer of the backbone network is calculated, and the weight is applied to each feature channel to obtain the weighted adjusted feature channel parameters.

5. The method according to claim 4, characterized in that The step of calculating the weights of feature channels in each feature layer of the backbone network and acting on the feature channels includes: Each of the target Riemann distances is normalized to obtain a plurality of normalized Riemann distances, and corresponding weights are obtained according to each of the normalized Riemann distances and used as weights of corresponding feature channels in the feature layer of the backbone network.

6. The method according to claim 1, characterized in that The outputting the anchor box and the confidence level through the head network based on the target feature information includes: The feature information of each target is fused through the neck network to obtain feature fusion information. According to the feature fusion information, the head network generates a candidate frame set for each road target and gives the confidence level. Each candidate frame set refers to a road target in the image and gives multiple candidate frames. Through the soft non-maximum suppression algorithm, candidate boxes belonging to the same target are screened in each candidate box set to obtain the anchor box of each road target.

7. The method according to claim 6, characterized in that The method of obtaining the anchor frame of each road target in each candidate frame set by using a soft non-maximum suppression algorithm includes: Obtaining the confidence of each candidate frame in the candidate frame set, and obtaining a target candidate frame and a candidate frame to be processed according to each confidence, wherein the target candidate frame is used to indicate a candidate frame with the highest confidence, and the candidate frame to be processed is used to indicate a candidate frame whose intersection-over-union ratio with the target candidate frame is greater than a threshold; Mapping the target candidate box and the candidate box to be processed to the Riemann space to obtain a candidate Riemann distance, where the candidate Riemann distance is used to indicate the distance between the target candidate box and the candidate box to be processed and each corresponding mapping point in the Riemann space, and obtaining an adjusted confidence according to the candidate Riemann distance, and using the adjusted confidence as a new confidence of the candidate box to be processed, so as to update the confidence of each candidate box in the candidate box set; Based on the iteratively updated candidate box set, the candidate box corresponding to the highest confidence is used as the anchor box.

8. A target detection device, characterized in that: include: An acquisition module is used to acquire a sample image set, wherein the sample image set includes a plurality of collected images, wherein the collected images are used to indicate various types of road targets pre-collected by a vehicle sensor, and each of the collected images is mapped to a Riemann space to obtain a Riemann mean corresponding to each type of road target, and the sample image set and each of the Riemann means are used as training data; A processing module, used to construct a convolutional neural network, input the training data into the convolutional neural network for training, and use the trained convolutional neural network as a target detection model, wherein the target detection model includes a backbone network, a neck network, and a head network, wherein the target weight of each convolution kernel of the weighted feature layer in the backbone network is obtained according to each of the Riemann means; An execution module is used to acquire a real-time image, input the real-time image into the backbone network, and collect target feature information of the real-time image through the target weights of each convolution kernel, and output an anchor box and confidence through the head network based on the target feature information, wherein the anchor box is used to indicate the type and position of each target in the real-time image.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images