Target detection model training method, target detection method, device and equipment

By performing feature fusion and foreground area annotation in multimodal fusion object detection, a multimodal training foreground heat map is generated, which solves the problem of foreground background confusion and improves the target detection accuracy.

CN119068296AActive Publication Date: 2024-12-03INSPUR (BEIJING) ELECTRONICS INFORMATION IND CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202310651391.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-02
Publication Date
2024-12-03
Estimated Expiration
2043-06-02

AI Technical Summary

Technical Problem

In multimodal fusion target detection, the foreground background confusion problem leads to a reduction in the target detection accuracy, which is difficult to effectively solve in the prior art.

Method used

By obtaining the training data features of multiple modals in the same view, performing feature fusion, and annotating the foreground area of ​​the samples in the view, using the annotation results to train the first foreground perception model, generate a multimodal training foreground heat map, and enhance Training fusion features and then training the object detection model.

Benefits of technology

It effectively improves the perception ability of the target detection model to the foreground area, reduces the false detection and missed detection rates, and improves the accuracy of target detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068296A_ABST
    Figure CN119068296A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of target detection, and discloses a target detection model training method, device and equipment, and a target detection method, device and equipment, and the training method comprises the steps: carrying out the fusion of training data features of all modes under the same view, and obtaining a training fusion feature; performing foreground region labeling on samples in the training set under the view, and training a first foreground perception model by using the training fusion features and a foreground region labeling result; using the first foreground perception model and the training fusion features to obtain a multi-modal training foreground heat map; obtaining foreground-enhanced training fusion features by using the multi-modal training foreground heat map and the training fusion features; and training by using the foreground-enhanced training fusion features to obtain a target detection model. According to the technical scheme disclosed by the invention, the first foreground perception model is utilized to perceive the foreground region to obtain the foreground heat map, and the foreground heat map is utilized to enhance the foreground in the training fusion feature, so that the foreground and background distinguishing capability is improved, and the target detection precision is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and more specifically, to a target detection model training method, a target detection method, a device and equipment. Background Art

[0002] Object detection uses the data collected by sensors as input to predict the information of the object of interest in space. Different sensors collect different modalities and have complementary advantages. Therefore, multimodal fusion solutions have become an important trend in object detection.

[0003] When multimodal fusion is used for target detection, there is a foreground-background confusion problem in the data features after the fusion of data features from multiple types of sensors. The foreground-background confusion problem can lead to false detection or missed detection of targets, thereby reducing the accuracy of target detection.

[0004] In summary, how to improve target detection accuracy is a technical problem that needs to be urgently solved by those skilled in the art. Summary of the invention

[0005] In view of this, an object of the present invention is to provide a target detection model training method, a target detection method, a target detection model training device, a target detection device, an electronic device and a readable storage medium, so as to improve the target detection accuracy.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A target detection model training method, comprising:

[0008] Obtain training data features of multiple modalities in the same view;

[0009] Fusing the training data features of each of the modalities under the view to obtain training fusion features;

[0010] Under the view, the samples in the training set are labeled with foreground regions, and a first foreground perception model is trained using the training fusion features and the foreground region labeling results; wherein in the foreground region labeling results, the foreground labeling value of the foreground region is greater than the foreground labeling value of the background region;

[0011] Obtaining a multimodal training foreground heat map using the first foreground perception model and the training fusion features;

[0012] Using the multimodal training foreground heat map and the training fusion feature, obtaining the training fusion feature with foreground enhancement;

[0013] The target detection model is trained using the training fusion features with foreground enhancement.

[0014] Preferably, after acquiring the training data features of multiple modalities in the same view, the method further includes:

[0015] Using the training data features of each of the modalities and the foreground area labeling results, training a second foreground perception model corresponding to each of the modalities;

[0016] Using the second foreground perception model corresponding to each of the modalities and the training data features, a single-modal training foreground heat map corresponding to each of the modalities is obtained;

[0017] Using the single-modality training foreground heat map corresponding to each of the modalities and the training data features, the training data features of the intra-modality foreground enhancement corresponding to each of the modalities are obtained;

[0018] The training data features of each of the modalities under the view are fused, including:

[0019] The training data features of the intra-modal foreground enhancement corresponding to each of the modalities are fused.

[0020] Preferably, when the training data features of each modality and the foreground area labeling result are used to train the second foreground perception model corresponding to each modality, it also includes:

[0021] The foreground-aware loss function corresponding to each of the second foreground-aware models is used to supervise the training of the corresponding second foreground-aware model.

[0022] Preferably, before using the foreground-aware loss function corresponding to each of the second foreground-aware models to supervise the training of the corresponding second foreground-aware model, the method further includes:

[0023] Annotating the confusion area in the samples in the training set; wherein the confusion area is a confusion area of ​​the foreground and background, and in the confusion area annotation result, the value in the confusion area is greater than 1, and the value in the non-confusion area is equal to 1;

[0024] weighting the foreground perception loss function corresponding to each of the second foreground perception models by using the confusion region labeling result to obtain the foreground perception loss function after the confusion region corresponding to each of the second foreground perception models is weighted;

[0025] Using the foreground-aware loss function corresponding to each of the second foreground-aware models to supervise the training of the corresponding second foreground-aware models includes:

[0026] The foreground-aware loss function weighted by the confusion region corresponding to each second foreground-aware model is used to supervise the training of the corresponding second foreground-aware model.

[0027] Preferably, before labeling the confusing areas in the samples in the training set, the method further includes:

[0028] Determine the outer boundary of the confusion region and the inner boundary of the confusion region according to the foreground region marked in the sample;

[0029] An area between the outer boundary of the confusion area and the inner boundary of the confusion area is determined as the confusion area.

[0030] Preferably, after obtaining the single-modality training foreground heat map corresponding to each of the modalities by using the second foreground perception model corresponding to each of the modalities and the training data features, the method further includes:

[0031] Using the training data features corresponding to each of the modalities and the single-modal training foreground heat maps corresponding to other modalities, the training data features corresponding to each of the modalities for inter-modal foreground enhancement are obtained;

[0032] The training data features of the intra-modality foreground enhancement corresponding to each of the modalities are fused, including:

[0033] The training data features of the intra-modality foreground enhancement corresponding to each of the modalities and the training data features of the inter-modality foreground enhancement corresponding to each of the modalities are fused.

[0034] Preferably, the training data features of the intra-modality foreground enhancement corresponding to each of the modalities and the training data features of the inter-modality foreground enhancement corresponding to each of the modalities are fused, including:

[0035] Combining the training data features of the intra-modality foreground enhancement corresponding to each of the modalities and the training data features of the inter-modality foreground enhancement corresponding to each of the modalities according to the channel dimension to obtain combined training data features;

[0036] The combined training data features are fused.

[0037] Preferably, the training data features of the intra-modality foreground enhancement corresponding to each of the modalities and the training data features of the inter-modality foreground enhancement corresponding to each of the modalities are combined according to the channel dimension, including:

[0038] Obtaining weight values ​​corresponding to the training data features of intra-modality foreground enhancement corresponding to each of the modalities and the training data features of inter-modality foreground enhancement corresponding to each of the modalities;

[0039] Using each of the weight values, weighted processing is performed on the corresponding training data features of intra-modality foreground enhancement and the training data features of inter-modality foreground enhancement;

[0040] The weighted training data features of intra-modality foreground enhancement and the weighted training data features of inter-modality foreground enhancement are combined.

[0041] Preferably, the first foreground perception model and the second foreground perception model corresponding to each of the modalities both include an encoding module and a decoding module.

[0042] Preferably, the encoding module includes multiple encoding units, the encoding unit includes a mean pooling unit and a first residual unit, and the decoding module includes multiple decoding units, the decoding unit includes an upsampling unit and a second residual unit.

[0043] Preferably, the first residual unit and the second residual unit both include a batch normalization layer, a non-linear activation layer and a 1×1 convolution layer.

[0044] Preferably, marking the foreground area of ​​the samples in the training set under the view includes:

[0045] Acquire two-dimensional information of each target bounding box in the sample; the two-dimensional information includes a center point, a width, a length, and an orientation angle of the target bounding box;

[0046] Marking the foreground area of ​​the sample according to the two-dimensional information of each target bounding box;

[0047] When the sample includes a plurality of the target bounding boxes and there is an overlap between at least two of the target bounding boxes, the foreground annotation values ​​of the overlapped parts are averaged.

[0048] Preferably, labeling the foreground area of ​​the sample according to the two-dimensional information of each target bounding box includes:

[0049] If the orientation angle of the current target bounding box in the sample is equal to 0, determining a first inscribed ellipse of the current target bounding box according to the center point, width and length of the current target bounding box, marking a foreground area for the first inscribed ellipse according to a two-dimensional Gaussian distribution, and marking an area outside the first inscribed ellipse as 0, to obtain a first marking result;

[0050] If the orientation angle of the current target bounding box in the sample is not equal to 0, determining a standard bounding box corresponding to the current target bounding box according to the orientation angle of the current target bounding box, determining a second inscribed ellipse of the standard bounding box according to the center point, width and length of the current target bounding box, annotating the foreground area of ​​the second inscribed ellipse according to a two-dimensional Gaussian distribution, and marking the area outside the second inscribed ellipse as 0 to obtain a second foreground annotation result, and converting the second foreground annotation result into a third foreground annotation result according to a coordinate mapping relationship between the current target bounding box and the standard bounding box; the orientation angle of the standard bounding box is equal to 0;

[0051] The next target bounding box in the sample is used as the current target bounding box, and the steps corresponding to if the orientation angle of the current target bounding box in the sample is equal to 0 or if the orientation angle of the current target bounding box in the sample is not equal to 0 are returned to be executed, until the foreground labeling results corresponding to all the target bounding boxes in the sample are obtained, and if there is an overlapping part between at least two of the target bounding boxes, the foreground labeling values ​​of the overlapping parts are averaged to obtain the global foreground labeling of the sample;

[0052] The global foreground label of the sample is added to the first auxiliary matrix to obtain a foreground area labeling result; each element in the first auxiliary matrix is ​​the same and non-zero.

[0053] Preferably, when the first foreground perception model is trained by using the training fusion features and the foreground area labeling results, the method further includes:

[0054] The foreground-aware loss function corresponding to the first foreground-aware model is used to supervise the training of the first foreground-aware model.

[0055] Preferably, before using the foreground-aware loss function corresponding to the first foreground-aware model to supervise the training of the first foreground-aware model, the method further includes:

[0056] Annotating the confusion area in the samples in the training set; wherein the confusion area is a confusion area of ​​the foreground and background, and in the confusion area annotation result, the value in the confusion area is greater than 1, and the value in the non-confusion area is equal to 1;

[0057] weighting the foreground perception loss function corresponding to the first foreground perception model by using the confusion region labeling result to obtain the foreground perception loss function weighted by the confusion region corresponding to the first foreground perception model;

[0058] Supervising the training of the first foreground-aware model by using a foreground-aware loss function corresponding to the first foreground-aware model includes:

[0059] The foreground-aware loss function weighted by the confusion region corresponding to the first foreground-aware model is used to supervise the training of the first foreground-aware model.

[0060] Preferably, when the modality includes a camera image and a point cloud, and the view is a bird's-eye view, obtaining training data features of multiple modalities in the same view includes:

[0061] Acquire training image data of a training image under a camera view, and extract training image features from the training image data;

[0062] Using a two-dimensional to three-dimensional conversion method, the training image features are converted into training image features under the bird's-eye view;

[0063] Training point cloud data is acquired, and training point cloud features under the bird's-eye view are extracted from the training point cloud data.

[0064] Preferably, after extracting the training image features from the training image data and before converting the training image features into the training image features under the bird's-eye view using a two-dimensional to three-dimensional conversion method, the method further includes:

[0065] Annotating the foreground area of ​​the training image;

[0066] Training a third foreground perception model using the training image features and the training image with the foreground area annotated;

[0067] Obtaining a camera view training foreground heat map using the third foreground perception model and the training image features;

[0068] The foreground heat map and the training image features are trained using the camera view to obtain the foreground-enhanced training image features.

[0069] Preferably, marking the foreground area of ​​the training image includes:

[0070] Acquire two-dimensional information of each target bounding box in the training image; the two-dimensional information includes a center point, a width, and a length of the target bounding box;

[0071] Annotating the foreground area of ​​the training image according to the two-dimensional information of each target bounding box;

[0072] Wherein, when the training image contains a plurality of the target bounding boxes and there is an overlapping portion between at least two of the target bounding boxes, the foreground annotation values ​​of the overlapping portions are averaged.

[0073] Preferably, labeling the foreground area of ​​the training image according to the two-dimensional information of each target bounding box includes:

[0074] Determine a third inscribed ellipse of each target bounding box according to the center point, width and length of each target bounding box, mark the foreground area of ​​the third inscribed ellipse according to a two-dimensional Gaussian distribution, and mark the area outside the third inscribed ellipse as 0, to obtain a third marking result;

[0075] If there is an overlapping portion between at least two of the target bounding boxes, averaging the foreground annotation values ​​of the overlapping portion to obtain a global foreground annotation of the training image;

[0076] The global foreground label of the training image is added to the second auxiliary matrix to obtain the training image with the foreground area labeled; each element in the second auxiliary matrix is ​​the same and non-zero.

[0077] Preferably, the target detection model is obtained by training the training fusion features with foreground enhancement, including:

[0078] Training the target detection model using the foreground-enhanced training fusion features;

[0079] The classification loss function and the regression loss function are used to supervise the training of the detection model until the classification loss function and the regression loss function meet the preset training conditions.

[0080] A target detection method, comprising:

[0081] Obtain the features of the data to be detected in multiple modalities in the same view;

[0082] Fusing the data features to be detected of each of the modalities under the view to obtain fused features to be detected;

[0083] Using the fusion features to be detected and the first foreground perception model, a multimodal foreground heat map to be detected is obtained; the first foreground perception model is trained using the target detection model training method described in any one of the above items;

[0084] Using the multimodal foreground heat map to be detected and the fused features to be detected, obtaining the fused features to be detected with foreground enhancement;

[0085] The foreground-enhanced fused features to be detected are input into a target detection model to obtain a target detection result; the target detection model is trained using any of the target detection model training methods described above.

[0086] Preferably, after acquiring the features of the data to be detected in the same view of multiple modalities, the method further includes:

[0087] Using the second foreground perception model corresponding to each of the modalities and the features of the data to be detected, a single-modal foreground heat map to be detected corresponding to each of the modalities is obtained; the second foreground perception model corresponding to each of the modalities is trained by the target detection model training method described in any one of the above items;

[0088] Using the single-modality foreground heat map to be detected corresponding to each of the modalities and the features of the data to be detected, the features of the data to be detected with enhanced intra-modal foreground corresponding to each of the modalities are obtained;

[0089] The features of the data to be detected of each of the modalities in the view are fused to obtain fused features to be detected, including:

[0090] The features of the data to be detected with foreground enhancement in the modalities corresponding to the modalities are fused.

[0091] Preferably, after obtaining the features of the data to be detected with enhanced intra-modal foregrounds corresponding to each of the modalities by using the single-modal foreground heat map to be detected corresponding to each of the modalities and the features of the data to be detected, the method further includes:

[0092] Using the features of the data to be detected corresponding to each of the modalities and the single-modal foreground heat maps to be detected corresponding to other modalities, the features of the data to be detected with inter-modal foreground enhancement corresponding to each of the modalities are obtained;

[0093] The features of the data to be detected with enhanced foreground in the modalities corresponding to the modalities are fused, including:

[0094] The features of the data to be detected with the intra-modality foreground enhancement corresponding to each of the modalities and the features of the data to be detected with the inter-modality foreground enhancement corresponding to each of the modalities are fused.

[0095] Preferably, the features of the data to be detected with the intra-modality foreground enhancement corresponding to each of the modalities and the features of the data to be detected with the inter-modality foreground enhancement corresponding to each of the modalities are fused, including:

[0096] Combining the data features to be detected with intra-modality foreground enhancement corresponding to each of the modalities and the data features to be detected with inter-modality foreground enhancement corresponding to each of the modalities according to the channel dimension to obtain combined data features to be detected;

[0097] The combined features of the data to be detected are fused.

[0098] Preferably, the features of the data to be detected with intra-modality foreground enhancement corresponding to each of the modalities and the features of the data to be detected with inter-modality foreground enhancement corresponding to each of the modalities are combined according to the channel dimension, including:

[0099] Obtaining weight values ​​corresponding to the features of the data to be detected for intra-modality foreground enhancement corresponding to each of the modalities and the features of the data to be detected for inter-modality foreground enhancement corresponding to each of the modalities;

[0100] Using each of the weight values, weighted processing is performed on the corresponding intra-modality foreground enhancement data features to be detected and the inter-modality foreground enhancement data features to be detected;

[0101] The features of the data to be detected with the intra-modality foreground enhancement after weighted processing and the features of the data to be detected with the inter-modality foreground enhancement after weighted processing are combined.

[0102] Preferably, when the modality includes a camera image and a point cloud, and the view is a bird's-eye view, acquiring features of data to be detected of multiple modalities in the same view includes:

[0103] Acquire image data to be detected of an image to be detected by a camera, and extract features of the image to be detected from the image data to be detected;

[0104] Using a two-dimensional to three-dimensional conversion method, the image features to be detected are converted into the image features to be detected under the bird's-eye view;

[0105] Obtaining the point cloud data to be detected, and extracting the point cloud features to be detected in the bird's-eye view from the point cloud data to be detected.

[0106] Preferably, after extracting the image features to be detected from the image data to be detected, and before converting the image features to be detected into the image features to be detected in the bird's-eye view by using a two-dimensional to three-dimensional conversion method, the method further includes:

[0107] Using the third foreground perception model and the features of the image to be detected, a heat map of the foreground to be detected in the camera view is obtained; the third foreground perception model is trained using the target detection model training method as described above;

[0108] The foreground-enhanced image features to be detected are obtained by utilizing the camera view foreground heat map to be detected and the image features to be detected.

[0109] A target detection model training device, comprising:

[0110] A first acquisition module is used to acquire training data features of multiple modalities in the same view;

[0111] A first fusion module, used for fusing the training data features of each of the modalities under the view to obtain a training fusion feature;

[0112] A first training module is used to label the foreground area of ​​the samples in the training set under the view, and train a first foreground perception model using the training fusion features and the foreground area labeling results; wherein the foreground labeling value of the foreground area in the foreground area labeling results is greater than the foreground labeling value of the background area;

[0113] A first heat map obtaining module, configured to obtain a multimodal training foreground heat map using the first foreground perception model and the training fusion features;

[0114] First, a foreground enhancement module is obtained, which is used to obtain the training fusion feature for foreground enhancement by using the multimodal training foreground heat map and the training fusion feature;

[0115] A target detection model module is obtained, which is used to train a target detection model using the training fusion features of foreground enhancement.

[0116] A target detection device, comprising:

[0117] The second acquisition module is used to acquire the features of the data to be detected in multiple modalities in the same view;

[0118] A second fusion module is used to fuse the data features to be detected of each of the modalities under the view to obtain a fused feature to be detected;

[0119] A fourth heat map obtaining module is used to obtain a multimodal foreground heat map to be detected by using the fusion features to be detected and the first foreground perception model; the first foreground perception model is trained by using the target detection model training method as described in any one of the above items;

[0120] Fifth, a foreground enhancement module is obtained, which is used to obtain the foreground enhanced fusion feature to be detected by using the multimodal foreground heat map to be detected and the fusion feature to be detected;

[0121] The detection result obtaining module is used to input the foreground-enhanced fusion feature to be detected into a target detection model to obtain a detection result; the target detection model is trained using the target detection model training method described in any one of the above items.

[0122] An electronic device, comprising:

[0123] Memory for storing computer programs;

[0124] A processor is used to implement the steps of the target detection model training method as described in any one of the above items when executing the computer program, and / or the steps of the target detection method as described in any one of the above items.

[0125] A readable storage medium having a computer program stored therein, wherein the computer program, when executed by a processor, implements the steps of the target detection model training method as described in any one of the above items, and / or the steps of the target detection method as described in any one of the above items.

[0126] The present invention provides a target detection model training method, a target detection method, an apparatus and a device, wherein the training method comprises: obtaining training data features of multiple modalities in the same view; fusing the training data features of each modality in the view to obtain training fusion features; marking the foreground area of ​​samples in a training set in the view, and training a first foreground perception model using the training fusion features and the foreground area marking results; wherein the foreground marking value in the foreground area marking result is greater than the foreground marking value of the background area; obtaining a multimodal training foreground heat map using the first foreground perception model and the training fusion features; obtaining a foreground enhanced training fusion feature using the multimodal training foreground heat map and the training fusion features; and obtaining a target detection model by training using the foreground enhanced training fusion features.

[0127] The above technical solution disclosed by the present invention fuses the training data features of multiple modalities in the same view to obtain the training fusion feature. Under the above view, the samples in the training set are labeled with the foreground area, wherein the foreground labeling value of the foreground area in the foreground area labeling result is greater than the foreground labeling value of the background area, so as to improve the perception ability of the target detection model to the foreground area. Afterwards, the first foreground perception model is trained using the training fusion feature and the foreground area labeling result, and the multimodal training foreground heat map is predicted using the first foreground perception model and the training fusion feature. Since the foreground labeling value of the foreground area in the foreground area labeling result is greater than the foreground labeling value of the background area, the value of the foreground area in the multimodal training foreground heat map predicted by the first foreground perception model is greater than the value of the background area, so as to realize the perception of the foreground area using the first foreground perception model. Then, the foreground in the training fusion feature is enhanced using the multimodal training foreground heat map to obtain the foreground enhanced training fusion feature, so as to facilitate the enhancement of the target detection model's ability to distinguish the foreground and improve the ability to distinguish the foreground from the background. Afterwards, the target detection model is trained using the foreground enhanced training fusion feature. The above method can solve the foreground and background confusion problem of fusion features, improve the target detection model's ability to distinguish and recognize the foreground in the fusion features, effectively reduce the probability of false detection and missed detection, and improve the target detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0128] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0129] Figure 1 A flowchart of a method for training a target detection model provided by an embodiment of the present invention;

[0130] Figure 2 A schematic diagram of another target detection model training method provided by an embodiment of the present invention;

[0131] Figure 3 A schematic diagram of implementing intra-modal and inter-modal foreground enhancement provided by an embodiment of the present invention;

[0132] Figure 4 A schematic diagram of multimodal feature fusion provided by an embodiment of the present invention;

[0133] Figure 5 A schematic diagram of the structure of a first foreground perception model and a second foreground perception model provided by an embodiment of the present invention;

[0134] Figure 6 A schematic diagram of the structure of a first residual unit and a second residual unit provided in an embodiment of the present invention;

[0135] Figure 7 It is a schematic diagram of a three-dimensional space coordinate system;

[0136] Figure 8 A schematic diagram of the distribution of the target bounding box provided by an embodiment of the present invention in the plane formed by the x-axis and the y-axis;

[0137] Fig. 9 A schematic diagram of foreground labeling of a target bounding box with a facing angle equal to 0 provided by an embodiment of the present invention;

[0138] Fig.10 A schematic diagram of the distribution of the outer boundary of the confusion area and the inner boundary of the confusion area provided in an embodiment of the present invention;

[0139] Fig.11 A schematic diagram of another target detection model training method provided by an embodiment of the present invention;

[0140] Fig.12 A flow chart of a target detection method provided by an embodiment of the present invention;

[0141] Fig.13 A schematic diagram of the structure of a target detection model training device provided by an embodiment of the present invention;

[0142] Fig.14 A schematic diagram of the structure of a target detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0143] Different types of environmental perception sensors collect data in different modalities, and their advantages can complement each other. Therefore, multimodal fusion has become an important trend in target detection.

[0144] In multimodal fusion, due to the different modalities, the foreground (i.e., the target of interest) and background (i.e., the target of non-interest) features of different modalities at the same spatial position may interfere with each other, resulting in foreground-background confusion in the fused features. Taking the two modalities of image and point cloud as an example, the significant background area in the image feature will suppress the foreground area of ​​the point cloud feature, resulting in foreground-background confusion in the fused features. Foreground-background confusion can easily lead to false detection or missed detection, reducing the accuracy of target detection.

[0145] To this end, the present invention provides a target detection model training method, a target detection method, a target detection model training device, a target detection device, an electronic device and a readable storage medium, which are used to solve the foreground and background confusion problem to improve the target detection accuracy.

[0146] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0147] See also Figure 1 , which shows a flow chart of a target detection model training method provided by an embodiment of the present invention. The target detection model training method provided by an embodiment of the present invention may include:

[0148] S11: Obtain training data features of multiple modalities in the same view.

[0149] In the present invention, training data collected by multiple sensors can be obtained, and one type of sensor corresponds to one modality. Since different types of sensors have different viewing angles and different modalities of the collected data, in order to improve the fusion effect of data features of multiple modalities, the training data features of the training data collected by multiple sensors can be converted to the same view to obtain the training data features of multiple modalities in the same view, thereby facilitating the fusion of training data features of multiple modalities in the same view and improving the multi-modal feature fusion effect.

[0150] The same view mentioned above can be a BEV (Bird's-Eye View) view, or other three-dimensional views, or a two-dimensional view, as long as the training data features of multiple modalities can be uniformly converted to the view. It should be noted that in order to achieve three-dimensional target detection and improve target detection performance, the above view can specifically be a three-dimensional view.

[0151] S12: Fuse the training data features of each modality under the view to obtain training fusion features.

[0152] On the basis of step S11, the training data features of each modality in the same view may be fused to obtain the training fusion features in the above view (i.e., the view in the same view mentioned above). Specifically, the training data features of each modality in the same view may be combined, such as concatenating or adding along the channel dimension, to obtain the combined training data features, and then the combined training data features are fused to output the training fusion features. Specifically, a plurality of residual units may be used to perform the fusion operation on the combined training features.

[0153] By fusing the training data features of each modality in the same view, the advantages of the training data features of each modality can be complemented to improve the accuracy of target detection.

[0154] S13: labeling the foreground area of ​​the samples in the training set under the view, and training the first foreground perception model using the training fusion features and the foreground area labeling results; wherein, in the foreground area labeling results, the foreground labeling value of the foreground area is greater than the foreground labeling value of the background area.

[0155] In the present invention, in order to improve the foreground perception ability of the target detection model, the samples (or data) in the training set can be annotated with foreground areas under the view to obtain foreground area annotation results. Among them, in the foreground area annotation results, the foreground annotation value of the foreground area is greater than the foreground annotation value of the background area. That is, the foreground area and background area corresponding to each target in each sample in the training set under the above view can be obtained, and a foreground annotation value is generated in the foreground area corresponding to each target and a foreground annotation value is generated in the background area. If there is an overlap between the foreground areas corresponding to the targets, the foreground annotation values ​​of the overlapping parts are averaged. Of course, other processing methods can also be used to process the foreground annotation values ​​of the overlapping parts.

[0156] After the samples are labeled with the foreground area under the view, the foreground labeling value of the foreground area is greater than the foreground labeling value of the background area, so that the samples after the foreground area labeling can highlight the foreground area, thereby facilitating the prediction of the foreground area in the feature and improving the perception of the foreground area. It should be noted that the foreground labeling of the samples in the training set under the view can be performed before step S11, that is, it can be generated before the first foreground perception model starts training, and the foreground area labeling results can be stored offline for reading and use in the model training stage, and the foreground area labeling of the samples in the training set can also be performed in real time under the view during the model training.

[0157] After obtaining the training fusion features and the foreground area annotation results, the first foreground perception model can be trained using the training fusion features and the foreground area annotation results. The training fusion features are the input of the first foreground perception model, the foreground area annotation results are the output of the first foreground perception model, and the first foreground perception model is trained using the input and output.

[0158] S14: Utilize the first foreground perception model and the training fusion features to obtain a multimodal training foreground heat map.

[0159] After completing the training of the first foreground perception model, or after each preset number of times (e.g., once) of training the first foreground perception model, the training fusion features can be input into the first foreground perception model to predict a multimodal training foreground heat map using the first foreground perception model. That is, the training fusion features under the above view can be used as input of the first foreground perception model to predict a multimodal training foreground heat map under the above view.

[0160] Since the first foreground perception model is trained using the foreground region annotation results, the value of the foreground region in the multimodal training foreground heat map predicted by the first foreground perception model is greater than the value of the background region. Therefore, the multimodal training foreground heat map can be regarded as a prediction score map of the foreground region and the background region, or as a prediction probability map of the foreground region and the background region.

[0161] From the above, it can be seen that the first foreground perception model can realize the prediction of the foreground area in the fusion feature, thereby improving the utilization of the foreground feature by the target detection model, so as to improve the foreground background discrimination ability of the target detection model and improve the target detection performance. That is, the present invention can use the first foreground perception model to realize the perception of the foreground area in the training fusion feature, so as to improve the discrimination ability of the target detection model for the foreground target, achieve the effect of combating the foreground background confusion and better perceiving the surrounding environment.

[0162] S15: Utilize the multimodal training foreground heat map and the training fusion features to obtain the foreground enhanced training fusion features.

[0163] After obtaining the multimodal training foreground heat map, the multimodal training foreground heat map can be used to perform weighted operations on the training fusion features to obtain foreground-enhanced training fusion features, so as to utilize the multimodal training foreground map to enhance the foreground features in the training fusion features, thereby continuously improving the foreground perception ability of the target detection model and further improving the target detection performance.

[0164] From the above, it can be seen that the present invention does not eliminate background features, but retains background features while enhancing foreground features, so as to comprehensively utilize the semantic information contained in the foreground background to achieve target detection, thereby facilitating the improvement of target detection accuracy. Moreover, the foreground enhancement strategy proposed by the present invention is to improve the foreground perception ability of the feature at the feature level, so as to improve the target detection performance.

[0165] S16: Utilize the foreground enhanced training fusion feature training to obtain the target detection model.

[0166] After obtaining the foreground enhanced training fusion features, the foreground enhanced training fusion features can be used to train the target detection model until the preset training conditions are met (for example, the number of training times reaches the preset number of times or the value of the corresponding loss function is less than the preset threshold, etc.), and the detection model corresponding to the preset training conditions is determined as the target detection model, so as to use the target detection model to perform target detection, thereby improving the target detection model's ability to discriminate and recognize the foreground in the fusion features, effectively reducing the probability of false detection and missed detection, and improving the target detection accuracy.

[0167] It should be noted that when the above view is a three-dimensional view, the target detection model finally obtained by training is a three-dimensional target detection model, which is used for three-dimensional target detection. In addition, the training of the first foreground perception model and the detection model can be end-to-end training together, or it can be independent pre-training. That is, the first foreground perception model and the detection model can be trained as a whole model: a complete training includes the training of the first foreground perception model and the training of the detection model, that is, for the current training, after the first foreground perception model completes the current training, the multimodal training foreground heat map output by the first foreground perception model is used in the current training of the detection model, and after the current training of the target detection model is completed, the next training of the overall model is performed. Alternatively, after the training of the first foreground perception model is completed, the target detection model can be trained to obtain the target detection model.

[0168] Among them, the trained first foreground perception model and target detection model can be applied in the field of autonomous driving, that is, after the first foreground perception model and target detection model are trained, the first foreground perception model and target detection model can be deployed in the autonomous driving system to improve the accuracy of the autonomous driving system in detecting three-dimensional targets, thereby improving the safety of the autonomous driving system. Of course, it can also be applied to smart terminals such as smart robots, smart homes, mobile phones, and industrial production fields such as aerospace assembly or other fields that require target detection.

[0169] The above technical solution disclosed by the present invention fuses the training data features of multiple modalities in the same view to obtain the training fusion feature. Under the above view, the samples in the training set are labeled with the foreground area, wherein the foreground labeling value of the foreground area in the foreground area labeling result is greater than the foreground labeling value of the background area, so as to improve the perception ability of the target detection model to the foreground area. Afterwards, the first foreground perception model is trained using the training fusion feature and the foreground area labeling result, and the multimodal training foreground heat map is predicted using the first foreground perception model and the training fusion feature. Since the foreground labeling value of the foreground area in the foreground area labeling result is greater than the foreground labeling value of the background area, the value of the foreground area in the multimodal training foreground heat map predicted by the first foreground perception model is greater than the value of the background area, so as to realize the perception of the foreground area using the first foreground perception model. Then, the foreground in the training fusion feature is enhanced using the multimodal training foreground heat map to obtain the foreground enhanced training fusion feature, so as to facilitate the enhancement of the target detection model's ability to distinguish the foreground and improve the ability to distinguish the foreground from the background. Afterwards, the target detection model is trained using the foreground enhanced training fusion feature. The above method can solve the foreground and background confusion problem of fusion features, improve the target detection model's ability to distinguish and recognize the foreground in the fusion features, effectively reduce the probability of false detection and missed detection, and improve the target detection accuracy.

[0170] For details, please refer to Figure 2 , which shows a schematic diagram of another target detection model training method provided by an embodiment of the present invention. A target detection model training method provided by an embodiment of the present invention, after acquiring training data features of multiple modalities in the same view, may also include:

[0171] Using the training data features of each modality and the foreground area annotation results, train the second foreground perception model corresponding to each modality;

[0172] Using the second foreground perception model and training data features corresponding to each modality, a single-modality training foreground heat map corresponding to each modality is obtained;

[0173] Using the single-modality training foreground heat map and training data features corresponding to each modality, the training data features of the intra-modality foreground enhancement corresponding to each modality are obtained;

[0174] The training data features of each modality under the view are integrated, which may include:

[0175] The training data features of intra-modal foreground enhancement corresponding to each modality are fused.

[0176] Considering that foreground-background confusion occurs not only in the data features after multimodal fusion, but also may exist in the data features of each modality, for example, when the sensor collects data at a long distance, the foreground information obtained is relatively small, so it is easy to be confused with the background. In addition, when it is raining and the light is relatively poor, the sensor may obtain less foreground data, so it is also easy to be confused with the background. For this reason, the present invention also performs foreground perception processing on the training data features of each modality before fusing the training data features of each modality to enhance the recognition ability of the foreground, thereby facilitating the improvement of target detection accuracy.

[0177] Specifically, after obtaining the training data features of multiple modalities in the same view, the training data features of each modality and the foreground area annotation results can be used to train the second foreground perception model corresponding to each modality. Specifically, each modality corresponds to a second foreground perception model. For any modality, the training data features of the modality in the above view can be used as the input of the second foreground perception model corresponding to the modality, and the foreground annotation results in the above view can be used as the output of the second foreground perception model corresponding to the modality, and the second foreground perception model can be trained using the input and output.

[0178] After completing the training of the second foreground perception model corresponding to the modality, or after each training of the second foreground notification model corresponding to the preset modality, the training data features corresponding to the modality can be input into the second foreground perception model corresponding to the modality, so as to use the second foreground perception model corresponding to the modality to predict the single-modal training foreground heat map corresponding to the modality. That is, the training data features of the modality under the above view can be used as the input of the second foreground perception model corresponding to the modality to predict the single-modal training foreground heat map corresponding to the modality under the above view.

[0179] Similar to the multimodal training foreground heat map, in the unimodal training foreground heat map corresponding to the modality, the value of the foreground area is greater than the value of the background area. That is, the unimodal training foreground heat map can also be regarded as a prediction score map of the foreground area and the background area, or as a prediction probability map of the foreground area and the background area, except that the unimodal training foreground heat map refers to the training foreground heat map under a single modality, and the multimodal training foreground heat map refers to the training foreground heat map under multimodality. It can be seen that the second foreground perception model can realize the prediction of the foreground area in the corresponding modality, thereby improving the utilization of the foreground features by the target detection model, so as to improve the foreground background discrimination ability of the target detection model and improve the target detection performance.

[0180] After obtaining the single-modality training foreground heat map corresponding to each modality, for any modality, the single-modality training foreground heat map corresponding to the modality can be used to perform weighted operations on the training data features of the modality to obtain the training data features with foreground enhancement within the modality corresponding to the modality. That is, the single-modality training foreground heat map corresponding to the modality is used to enhance the foreground features in the training data features corresponding to the modality within the modality, thereby effectively solving the foreground-background confusion problem existing in each modality, and thus improving the detection performance of the target detection model for the target.

[0181] On the basis of the above, when the training data features of each modality under the view are fused, the training data features of the modality foreground enhancement corresponding to each modality are fused. It should be noted that Figure 2 Take the case of including the first mode and the second mode as an example. As for the case of including more modes, it is the same as Figure 2 Similar.

[0182] The target detection model training method provided by the embodiment of the present invention may further include:

[0183] The foreground-aware loss function corresponding to each second foreground-aware model is used to supervise the training of the corresponding second foreground-aware model.

[0184] In the present invention, before using the training data features of each modality and the foreground area annotation results to train the second foreground perception model corresponding to each modality, a foreground perception loss function corresponding to the second foreground perception model corresponding to each modality can also be constructed: Among them, L foreground,2 The foreground perception loss function corresponding to the second foreground perception model corresponding to the modality is represented by the subscript 2, which indicates the second foreground perception model. jrepresents the value at the jth position in the foreground area annotation result under the above view, where the m value in the foreground area is greater than the m value in the background area, and p j,2 It represents the value of the unimodal training foreground heat map predicted by the second foreground perception model corresponding to the modality at the j-th position, and n represents the total number of positions in the foreground area result (or the unimodal training foreground heat map).

[0185] When the training data features and foreground area annotation results of each modality are used to train the second foreground perception model corresponding to each modality, the foreground perception loss function corresponding to each second foreground perception model can be used to supervise the training of the corresponding second foreground perception model. Specifically, the foreground perception loss function corresponding to each second foreground perception model can be used to determine whether the corresponding second foreground notification model has completed training. If the value of the foreground perception loss function is not less than the first preset threshold, it indicates that the gap between the single-modal training foreground heat map predicted by the second foreground perception model and the foreground annotation result is relatively large, that is, it indicates that the second foreground perception model has not completed training. If the value of the corresponding foreground perception loss function is less than the first preset threshold, it indicates that the single-modal training foreground heat map predicted by the second foreground perception model is very close to the foreground area annotation result, that is, it indicates that the single-modal training foreground heat map predicted by the second foreground perception model and the foreground area annotation result have good consistency. At this time, it can be considered that the second foreground perception model has completed training. That is, in the present invention, the numerical value in the foreground area annotation result is used as the true value and the corresponding foreground perception loss function is used to supervise the training of the corresponding second foreground notification model. Among them, the first preset threshold can be set according to demand.

[0186] Among them, since there are multiple loss functions, in order to improve the model training performance, a balance factor can be set for all or part of all loss functions to balance the supervision effect of each loss function. Specifically, all or part of the loss functions can be used for model training.

[0187] The target detection model training method provided by the embodiment of the present invention may further include, before using the foreground perception loss function corresponding to each second foreground perception model to supervise the training of the corresponding second foreground perception model:

[0188] The confusion area in the samples in the training set is labeled; the confusion area is the confusion area of ​​the foreground and background. In the confusion area labeling result, the value in the confusion area is greater than 1, and the value in the non-confusion area is equal to 1;

[0189] The foreground perception loss function corresponding to each second foreground perception model is weighted by using the confusion region labeling result to obtain the weighted foreground perception loss function of the confusion region corresponding to each second foreground perception model;

[0190] Using the foreground-aware loss function corresponding to each second foreground-aware model to supervise the training of the corresponding second foreground-aware model may include:

[0191] The foreground-aware loss function weighted by the confusion region corresponding to each second foreground-aware model is used to supervise the training of the corresponding second foreground-aware model.

[0192] In the present invention, in order to further improve the second foreground perception model's ability to distinguish the confused areas of the foreground and background, so as to reduce the interference of the background area and thus reduce false detection and missed detection, the confused areas of the samples in the training set can be labeled under the above-mentioned view, and the confused area labeling results can be used to participate in the second foreground perception loss function to obtain a confused area weighted foreground perception loss function, thereby enhancing the second foreground perception model's ability to distinguish the confused areas of the foreground and background.

[0193] Specifically, after obtaining the training set under the above view, the confusion area in the sample in the training set can be determined, wherein the confusion area is the confusion area of ​​the foreground and background, that is, the area where the foreground and background are confused. Then, the confusion area in the sample can be marked, wherein the value in the confusion area in the sample is marked as greater than 1, and the value in the non-confusion area is marked as equal to 1. In order to ensure that the second foreground perception model has a better unimodal foreground perception heat map prediction effect, the value of the confusion area in the confusion area annotation result can be between (1, 2], wherein, for the confusion area, the values ​​at each position inside it can be the same, or can increase from the inside to the outside, etc. It should be noted that in each sample, if there is an overlap between the confusion areas, the values ​​of the annotations of the overlapping parts can be averaged to serve as the values ​​of the annotations of the corresponding positions. Of course, other processing methods can also be used to process the values ​​of the annotations of the overlapping parts. That is, a confusion area mask can be generated, and the confusion area mask can be represented by A. The value rule of A is: the value in the confusion area corresponding to each target is set to ρ, and the value range of ρ is (1, 2], and the value in the non-confusion area is 1. It should be noted that the annotation of the confusion area in the samples in the training set can be generated before the second foreground perception model starts training, and the confusion area annotation results can be stored offline for reading and use in the model training stage, or the confusion area annotation of the samples in the training set can be performed in real time during the model training.

[0194] After the confusion regions in the images in the training set are labeled, the confusion region labeling results can be used to weight the foreground perception loss functions corresponding to the second foreground perception models to obtain the weighted foreground perception loss functions of the confusion regions corresponding to the second foreground perception models: Among them, a jRepresents the value at the jth position in the confusion region annotation result, where the a value in the confusion region is greater than the a value in the non-confusion region, and the a value in the confusion region is between (1,2], and the a value in the non-confusion region is 1.

[0195] Afterwards, when using the foreground perception loss function corresponding to each second foreground perception model to supervise the training of the corresponding second foreground perception model, the foreground perception loss function weighted by the confusion area corresponding to each second foreground perception model can be used to supervise the training of the corresponding second foreground perception model, so as to improve the discrimination ability of the corresponding second foreground perception model for the confusion area by using the foreground perception loss function weighted by the confusion area, thereby reducing the interference of the background area and improving the target detection performance.

[0196] Of course, other loss functions that can constrain the second foreground perception model to predict the consistency between the unimodal training foreground heat map and the foreground annotation can also be used to supervise the training of the second foreground perception model.

[0197] The target detection model training method provided by the embodiment of the present invention may further include, before labeling the confusing areas in the samples in the training set:

[0198] According to the foreground area marked in the sample, the outer boundary of the confusion area and the inner boundary of the confusion area are determined;

[0199] The area between the outer boundary of the confusion area and the inner boundary of the confusion area is determined as the confusion area.

[0200] In the present invention, when annotating the confusion region in the image in the training set, first, based on the foreground region annotated in the sample (the region can be determined when the foreground region is annotated for the sample), the foreground region can be expanded using a preset expansion coefficient α to obtain the outer boundary of the confusion region, and the foreground region can be shrunk using a preset contraction coefficient β to obtain the inner boundary of the confusion region. After that, the region between the outer boundary of the confusion region and the inner boundary of the confusion region can be determined as the confusion region. After determining the confusion region, the confusion region in the image can be annotated.

[0201] Among them, the expansion coefficient is greater than 1, and the contraction coefficient β is less than 1. The expansion coefficient α and the contraction coefficient β are used to control the size of the confusion area. The specific values ​​can be set according to the actual situation. For example, the value range of the expansion coefficient α can be (1, 1.5], and the value range of the contraction coefficient β can be [0.5, 1]. Taking the foreground area as a rectangular box as an example, the rectangular box can be expanded by the expansion coefficient α, that is, the rectangular box is expanded by α times to obtain an expanded rectangular box (i.e., the outer boundary of the confusion area), and the rectangular box can be contracted by the contraction coefficient β, that is, the rectangular box is contracted by β times to obtain a contracted rectangular box (i.e., the inner boundary of the confusion area). The area between the expanded rectangular box and the contracted rectangular box is the confusion area.

[0202] The above method can improve the rationality and reliability of the determination of the confusing region, so as to improve the discrimination capability of the confusing region of the second foreground perception model.

[0203] For details, please refer to Figure 3 , which shows a schematic diagram of implementing intra-modal and inter-modal foreground enhancement, where: Figure 3 Taking the two modalities of the first modality and the second modality as an example for explanation. An object detection model training method provided by an embodiment of the present invention, after obtaining a single-modal training foreground heat map corresponding to each modality by using the second foreground perception model and training data features corresponding to each modality, may also include:

[0204] Using the training data features corresponding to each modality and the single-modality training foreground heat map corresponding to other modalities, the training data features of inter-modality foreground enhancement corresponding to each modality are obtained;

[0205] The training data features of the intra-modal foreground enhancement corresponding to each modality are fused, which may include:

[0206] The training data features of intra-modal foreground enhancement corresponding to each modality and the training data features of inter-modal foreground enhancement corresponding to each modality are fused.

[0207] In the present invention, after obtaining the unimodal training foreground heat map corresponding to each modality by using the second foreground perception model and training data features corresponding to each modality, for the current modality (i.e., any modality), the unimodal training foreground heat map corresponding to other modalities can be used to perform weighted operations on the training data features corresponding to the current modality to obtain the inter-modal foreground enhanced training data features corresponding to the current modality. Wherein, when the number of modalities is more than two, for the current modality, the unimodal training foreground heat map corresponding to each of the remaining modalities can be used to perform weighted operations on the training data features corresponding to the current modality to obtain the multiple inter-modal foreground enhanced training data features corresponding to the current modality, and the multiple inter-modal foreground enhanced training data features corresponding to the current modality are averaged to obtain the inter-modal foreground enhanced training data features corresponding to the current modality.

[0208] On the basis of the above, when fusing the training data features of the intra-modal foreground enhancement corresponding to each modality, the training data features of the intra-modal foreground enhancement corresponding to each modality and the training data features of the inter-modal foreground enhancement corresponding to each modality can be specifically fused.

[0209] Through the above method, full utilization of the single-modal training foreground heat map corresponding to each modality can be achieved, so as to achieve the effect of complementary advantages of foreground perception results between modalities, so as to further enhance the foreground features in the training fusion features, thereby further improving the first foreground perception model and the target detection model's ability to distinguish the foreground, reducing false detections and missed detections, and improving target detection performance.

[0210] An object detection model training method provided by an embodiment of the present invention fuses the training data features of intra-modality foreground enhancement corresponding to each modality and the training data features of inter-modality foreground enhancement corresponding to each modality, which may include:

[0211] The training data features of the intra-modality foreground enhancement corresponding to each modality and the training data features of the inter-modality foreground enhancement corresponding to each modality are combined according to the channel dimension to obtain the combined training data features;

[0212] The combined training data features are fused.

[0213] In the present invention, when fusing the training data features of intra-modality foreground enhancement corresponding to each modality and the training data features of inter-modality foreground enhancement corresponding to each modality, first, the training data features of intra-modality foreground enhancement corresponding to all modalities and the training data features of inter-modality foreground enhancement corresponding to all modalities can be combined according to the channel dimension to obtain the combined training data features. Afterwards, the combined training data features can be fused (specifically, a fusion operation can be performed using several residual units) to improve the feature fusion effect.

[0214] For details, please refer to Figure 4 , which shows a schematic diagram of multimodal feature fusion provided by an embodiment of the present invention, wherein: Figure 4 Take the case where the first modality and the second modality are included, and the weight value corresponding to the training data feature of inter-modal foreground enhancement corresponding to the first modality is b1, the weight value corresponding to the training data feature of intra-modal foreground enhancement corresponding to the first modality is b2, the weight value corresponding to the training data feature of intra-modal foreground enhancement corresponding to the second modality is b3, and the weight value corresponding to the training data feature of inter-modal foreground enhancement corresponding to the second modality is b4 as an example. As for the case where more modalities are included, the weight value is the same as Figure 4 An object detection model training method provided by an embodiment of the present invention combines the intra-modality foreground enhancement training data features corresponding to each modality and the inter-modality foreground enhancement training data features corresponding to each modality according to the channel dimension, which may include:

[0215] Obtaining weight values ​​corresponding to the training data features of intra-modality foreground enhancement corresponding to each modality and the training data features of inter-modality foreground enhancement corresponding to each modality;

[0216] Using each weight value, weighted processing is performed on the corresponding intra-modality foreground enhancement training data features and inter-modality foreground enhancement training data features;

[0217] The weighted intra-modality foreground enhancement training data features and the weighted inter-modality foreground enhancement training data features are combined.

[0218] In the present invention, when the training data features of the intra-modal foreground enhancement corresponding to each modality and the training data features of the inter-modal foreground enhancement corresponding to each modality are combined according to the channel dimension, the weight values ​​corresponding to the training data features of the intra-modal foreground enhancement corresponding to each modality and the training data features of the inter-modal foreground enhancement corresponding to each modality can be obtained first. Among them, the weight value can be set by artificial experience or obtained by autonomous learning of the model. Considering that the effect of foreground enhancement between different modalities may not be as good as the effect of foreground enhancement within the modality, the weight value corresponding to the training data features of the intra-modal foreground enhancement can be set to be greater than the weight value of the training data features of the inter-modal foreground enhancement. Afterwards, the training data features of the intra-modal foreground enhancement corresponding to each modality and the training data features of the inter-modal foreground enhancement corresponding to each modality can be weighted with the corresponding weight values ​​respectively. Then, the weighted intra-modal foreground enhancement training data features corresponding to each modality and the weighted inter-modal foreground enhancement training data features corresponding to each modality can be combined.

[0219] The above method can better integrate the training data features of intra-modal foreground enhancement corresponding to each modality and the training data features of inter-modal foreground enhancement corresponding to each modality, so as to improve the foreground discrimination ability of the target detection model and improve the target detection accuracy.

[0220] An embodiment of the present invention provides a target detection model training method, wherein the first foreground perception model and the second foreground perception model corresponding to each modality may include an encoding module and a decoding module.

[0221] In the present invention, the first foreground perception model and the second foreground perception model corresponding to each modality may include an encoding module and a decoding module connected to the encoding module, the encoding module is used to encode the input training features to obtain encoded features, and the decoding module is used to decode the encoded features and process the encoded features into a foreground area heat map having the same spatial size as the input training features, so as to enhance the foreground features in the training features.

[0222] For details, please refer to Figure 5 , which shows a schematic diagram of the structure of the first foreground perception model and the second foreground perception model provided by an embodiment of the present invention. In an object detection model training method provided by an embodiment of the present invention, the encoding module may include multiple encoding units, the encoding unit may include a mean pooling unit and a first residual unit, the decoding module may include multiple decoding units, the decoding unit may include an upsampling unit and a second residual unit.

[0223] In the present invention, the encoding modules included in the first foreground perception and the second foreground perception models each include a plurality of connected encoding units, each encoding unit includes a mean pooling unit and a first residual unit connected to the mean pooling unit in the encoding unit, and for the last encoding unit, the first residual unit inside it is connected to the decoding module. The decoding units included in the first foreground perception and the second foreground perception models each include a plurality of connected decoding units, wherein each decoding unit includes an upsampling unit and a second residual unit connected to the upsampling unit in the decoding unit, and the first decoding unit is connected to the last encoding unit, specifically, the upsampling unit in the first decoding unit is connected to the first residual unit in the last encoding unit.

[0224] After the training features are input into the corresponding foreground perception models (first foreground perception model, second foreground perception model), the input training features are first processed by mean pooling and the first residual unit to obtain the encoded features. Subsequently, the encoded features are processed by upsampling and the second residual unit operation to process the encoded features into a training foreground heat map with the same spatial size as the input training features.

[0225] For details, please refer to Figure 6, which shows a schematic diagram of the structure of the first residual unit and the second residual unit provided by an embodiment of the present invention. In an object detection model training method provided by an embodiment of the present invention, the first residual unit and the second residual unit may both include a batch normalization layer, a nonlinear activation layer and a 1×1 convolution layer.

[0226] In the present invention, the structures of the first residual unit in the encoding unit and the second residual unit in the decoding unit may be the same, and both may include multiple sub-residual units, each of which includes a batch normalization layer, a nonlinear activation layer connected to the batch normalization layer, and a 1×1 convolution layer connected to the nonlinear activation layer.

[0227] An object detection model training method provided by an embodiment of the present invention labels a foreground area of ​​a sample in a training set under a view, which may include:

[0228] Obtaining two-dimensional information of each target bounding box in the sample; the two-dimensional information may include a center point, width, length, and orientation angle of the target bounding box;

[0229] According to the two-dimensional information of each target bounding box, the foreground area of ​​the sample is marked;

[0230] When a sample contains multiple target bounding boxes and there is an overlap between at least two target bounding boxes, the foreground annotation values ​​of the overlapped parts are averaged.

[0231] In the present invention, when the foreground area of ​​the samples in the training set is annotated under the view, if the view is a three-dimensional view, the three-dimensional information of the target bounding box in the sample can be obtained, and the height axis can be compressed to obtain the two-dimensional information of the target bounding box. If the view is a two-dimensional view, the two-dimensional information of the target bounding box in the sample can be directly obtained. The target bounding box in the sample can be identified and annotated in advance by manual methods or machine learning methods.

[0232] Taking the view as a three-dimensional view as an example, the three-dimensional information of the target bounding box can be described by (u, v, d, w, l, h, θ), where (u, v, d) represents the coordinates of the center point of the target bounding box, u is the coordinate on the x-axis, v is the coordinate on the y-axis, d is the coordinate on the z-axis, (w, l, h) describes the three-dimensional size of the target bounding box, w is the width of the target bounding box, l is the length of the target bounding box, h is the height of the target bounding box, and θ is the orientation angle of the target bounding box. For the orientation angle of the target bounding box, please refer to Figure 7 and Figure 8 ,in, Figure 7A schematic diagram of a three-dimensional space coordinate system is shown. In the defined three-dimensional space coordinate system, the z-axis is used as the gravity axis. On a plane perpendicular to the gravity axis z-axis, the x-axis represents the forward direction. The x-axis is also the reference direction of the target orientation, and the y-axis is perpendicular to the x-axis to the left. Figure 8 The diagram shows the distribution of the target bounding box in the plane formed by the x-axis and the y-axis. In the three-dimensional view, the gravity axis z-axis points to the human eye (i.e., the top view of the three-dimensional space coordinate diagram). The angle between the target bounding box and the x-axis is the target bounding box orientation angle θ, and the orientation angle θ increases in the counterclockwise direction. The orientation of the target bounding box is annotated when the target bounding box is annotated for the samples in the training set. For example, if the target in the target bounding box is a vehicle, the direction in which the front of the vehicle points is the orientation of the target bounding box. After obtaining the three-dimensional information (u, v, d, w, l, h, θ) of the target bounding box, the height axis can be compressed and the two-dimensional information of the target bounding box can be represented by (u, v, w, l, θ).

[0233] After obtaining the two-dimensional information of the target bounding box in the sample, the foreground area of ​​the sample can be annotated according to the two-dimensional information of the target bounding box. For example, the position of the target bounding box and the size of the range included can be determined according to the two-dimensional information of the target bounding box. Then, the area inside the target bounding box can be used as the foreground area, and the area outside the target bounding box can be used as the background area. The sample is annotated with the foreground area, and finally the foreground annotation value inside the target bounding box is greater than the foreground annotation value outside the target bounding box.

[0234] Among them, when the sample contains multiple target bounding boxes and at least two bounding boxes have overlapping parts, after marking the area within each target bounding box, for the overlapping part, the foreground annotation value of the overlapping part can be processed by averaging, and the obtained average foreground annotation value is used as the foreground annotation value of the overlapping part.

[0235] The above method can well implement the labeling of the foreground area of ​​the sample, so as to improve the model's ability to distinguish the foreground area, thereby improving the target detection performance.

[0236] An object detection model training method provided by an embodiment of the present invention labels a foreground area of ​​a sample according to two-dimensional information of each object bounding box, which may include:

[0237] If the orientation angle of the current target bounding box in the sample is equal to 0, then determine the first inscribed ellipse of the current target bounding box according to the center point, width and length of the current target bounding box, mark the foreground area of ​​the first inscribed ellipse according to the two-dimensional Gaussian distribution, and mark the area outside the first inscribed ellipse as 0 to obtain a first marking result;

[0238] If the orientation angle of the current target bounding box in the sample is not equal to 0, the standard bounding box corresponding to the current target bounding box is determined according to the orientation angle of the current target bounding box, the second inscribed ellipse of the standard bounding box is determined according to the center point, width and length of the current target bounding box, the foreground area is annotated for the second inscribed ellipse according to the two-dimensional Gaussian distribution, and the area outside the second inscribed ellipse is annotated as 0 to obtain a second foreground annotation result, and the second foreground annotation result is converted into a third foreground annotation result according to the coordinate mapping relationship between the current target bounding box and the standard bounding box; the orientation angle of the standard bounding box is equal to 0;

[0239] The next target bounding box in the sample is used as the current target bounding box, and the steps corresponding to if the orientation angle of the current target bounding box in the sample is equal to 0 or if the orientation angle of the current target bounding box in the sample is not equal to 0 are returned until all target bounding boxes in the sample are processed. If there is an overlap between at least two target bounding boxes, the foreground label values ​​of the overlapped parts are averaged to obtain the global foreground label of the sample.

[0240] The global foreground label of the sample is added to the first auxiliary matrix to obtain the foreground area labeling result; each element in the first auxiliary matrix is ​​the same and non-zero.

[0241] In the present invention, considering that the area within the target bounding box is not a strict foreground area, but also includes a part of the background area, simply using the area within the target bounding box for foreground annotation will inevitably introduce background information, which does not help the model's ability to distinguish foreground and background confusion. Intuitively, the foreground information is concentrated at the center of the target bounding box, and the importance of background information gradually decreases from the center point to the outside. Therefore, a two-dimensional Gaussian distribution that meets this characteristic can be used to perform foreground annotation in the target bounding box.

[0242] Specifically, when annotating the foreground area of ​​a sample according to the two-dimensional information of the target bounding box, the first target bounding box in the sample can be used as the current target bounding box. If the orientation angle of the current target bounding box is equal to 0, the first inscribed ellipse of the current target bounding box can be determined according to the center point (u, v), width w and length l of the current target bounding box:

[0243]

[0244] Among them, σ u , σ v They represent the semi-axis lengths in the width direction (i.e., the x-axis direction) and the length direction (i.e., the y-axis direction), i.e., σ u =w / 2,σ v =l / 2, (x,y) represents the coordinates of the inner point of the first inscribed ellipse.

[0245] The probability density function of the two-dimensional Gaussian distribution can be expressed as:

[0246]

[0247] Where X represents the coordinates (x, y) on the view plane corresponding to the above view, μ represents the mean vector, and Σ represents the covariance matrix. In order to use the two-dimensional Gaussian distribution to model the foreground annotation, let (X-μ) T Σ -1 (X-μ)=1, then the first inscribed ellipse of the two-dimensional current target bounding box is the contour of the two-dimensional Gaussian distribution probability density, so the two-dimensional target bounding box (u, v, w, l, θ) is modeled as a two-dimensional Gaussian distribution with a mean vector μ and a covariance matrix Σ satisfying the following formula:

[0248]

[0249] Afterwards, the foreground area of ​​the first inscribed ellipse within the current target bounding box can be labeled according to the two-dimensional Gaussian distribution, and the area outside the first inscribed ellipse can be labeled as 0 to obtain the first foreground labeling result (i.e., the foreground labeling result corresponding to the current target bounding box). For details, please refer to Fig. 9 , which shows a schematic diagram of foreground labeling of a target bounding box with an orientation angle equal to 0 provided by an embodiment of the present invention, wherein the dotted line represents the target bounding box with an orientation angle equal to 0, the inner ellipse is the first inscribed ellipse, and the foreground labeling value decreases from the center point of the first inscribed ellipse outward, which conforms to the two-dimensional Gaussian distribution.

[0250] If the orientation angle of the current target bounding box is not equal to 0, the standard bounding box corresponding to the current target bounding box can be determined according to the orientation angle of the current target bounding box, wherein the orientation angle of the standard bounding box is equal to 0, that is, the current target bounding box with an orientation angle of not 0 is rotated clockwise around the center point (u, v) until the orientation angle is equal to 0 to obtain the standard bounding box. Then, the second inscribed ellipse of the standard bounding box is determined according to the center point, width and length of the current target bounding box, and the coordinates of the inner point of the second inscribed ellipse are represented by (x', y'), then the second inscribed ellipse can be expressed as:

[0251]

[0252] After that, the foreground area of ​​the second inscribed ellipse in the standard bounding box can be annotated according to the two-dimensional Gaussian distribution, and the area outside the second inscribed ellipse can be annotated as 0, thereby obtaining the second foreground annotation result (i.e., the foreground annotation result corresponding to the standard bounding box corresponding to the current target bounding box). Then, according to the coordinate mapping relationship between the current target bounding box and the standard bounding box, the second foreground annotation result can be converted into the third foreground annotation result (i.e., the foreground annotation result corresponding to the current target bounding box), that is, the foreground annotation value at (x', y') is used as the foreground annotation value at (x, y). Among them, the coordinate mapping relationship between the current target bounding box and the standard bounding box is:

[0253]

[0254] After obtaining the foreground annotation result corresponding to the current target bounding box in the sample, the next target bounding box in the sample can be used as the current target bounding box, and return to execute the above steps corresponding to if the orientation angle of the current target bounding box in the sample is equal to 0 or if the orientation angle of the current target bounding box in the sample is not equal to 0. That is, if the orientation angle of the current target bounding box is equal to 0, the first inscribed ellipse of the current target bounding box is determined according to the center point, width and length of the current target bounding box, the foreground area is marked on the first inscribed ellipse according to the two-dimensional Gaussian distribution, and the area outside the first inscribed ellipse is marked as 0 to obtain the first marking result; if the orientation angle of the current target bounding box is not equal to 0, the standard bounding box corresponding to the current target bounding box is determined according to the orientation angle of the current target bounding box, the second inscribed ellipse of the standard bounding box is determined according to the center point, width and length of the current target bounding box, the foreground area is marked on the second inscribed ellipse according to the two-dimensional Gaussian distribution, and the area outside the second inscribed ellipse is marked as 0 to obtain the second foreground marking result, and the second foreground marking result is converted into the third foreground marking result according to the coordinate mapping relationship between the current target bounding box and the standard bounding box, thereby determining the foreground marking result corresponding to the current target bounding box. Repeat the above process until the foreground marking results corresponding to all target bounding boxes in the sample are obtained. Then, if there are at least two overlapping bounding boxes in the sample, the foreground annotation value of the overlapping part is processed, and the average foreground annotation value is used as the foreground annotation value of the overlapping part, thereby obtaining the global foreground annotation M of the sample. O .

[0255] In addition, considering the global foreground annotation M of the sample O In the example, the foreground label value of the background area is 0. In order to avoid the background features in the subsequent enhanced features becoming 0 and affecting the training and detection of the target detection model, the global foreground label M of the sample is O With the first auxiliary matrix M IAdd together to get the foreground area annotation result M S , that is, M S =M O +M I , where the first auxiliary matrix M I The size of the sample is related to the global foreground annotation M O The size of the first auxiliary matrix M I All elements in are the same and not equal to 0. Specifically, the first auxiliary matrix M I Each element in can be equal to 1. Of course, you can also get the global foreground mark M of the sample O Afterwards, the foreground area labeling result is directly obtained. At this time, when using the training features and the foreground labeling result to train the foreground perception model, the foreground labeling value at each position of the foreground area labeling result can be added with a non-zero value, and then the foreground perception model can be trained using the samples after the foreground area is labeled and added with a non-zero value.

[0256] Through the inscribed ellipse and two-dimensional Gaussian distribution, a more realistic and accurate foreground annotation can be obtained, thereby improving the effect of foreground enhancement, enhancing the model's ability to distinguish foreground-background confusion, and improving target detection performance.

[0257] In addition, if the orientation angle of the current target bounding box is equal to 0, it can be expanded and contracted on the basis of the first inscribed ellipse to obtain the outer boundary of the confusion area and the inner boundary of the confusion area respectively, so as to determine the confusion area corresponding to the current target bounding box according to the outer boundary of the confusion area and the inner boundary of the confusion area, so as to label the confusion area and participate in the supervision of the corresponding foreground perception model training, etc. The details are as follows:

[0258] The outer boundary of the confusion area is: The inner boundary of the confusion region is:

[0259] For details, please refer to Fig.10 , which shows a schematic diagram of the distribution of the outer boundary and the inner boundary of the confusion area, x where the inner dotted ellipse represents the inner boundary of the confusion area, and the outer dotted ellipse represents the outer boundary of the confusion area. α and β respectively represent the expansion coefficient and contraction coefficient of the confusion area relative to the foreground annotation ellipse (here, the first inscribed ellipse), which are used to control the size of the confusion area.

[0260] If the orientation angle of the current target bounding box is not equal to 0, after determining the standard bounding box corresponding to the current target bounding box and determining the second inscribed ellipse of the standard bounding box, expansion and contraction can be performed on the basis of the second inscribed ellipse to obtain the outer boundary of the confusion area under the standard bounding box and the inner boundary of the confusion area under the standard bounding box, respectively, as follows:

[0261] The outer boundary of the confusion area is: The inner boundary of the confusion region is:

[0262] Afterwards, according to the coordinate mapping relationship between the current target bounding box and the standard bounding box, the outer boundary of the confusion area under the standard bounding box can be converted to the outer boundary of the confusion area under the current target bounding box, and the inner boundary of the confusion area under the standard bounding box can be converted to the inner boundary of the confusion area under the current target bounding box, so as to determine the confusion area corresponding to the current target bounding box according to the outer boundary of the confusion area under the current target bounding box and the inner boundary of the confusion area under the current target bounding box, so as to label the confusion area and participate in the supervision of the corresponding foreground perception model training.

[0263] The target detection model training method provided by the embodiment of the present invention may further include:

[0264] The foreground-aware loss function corresponding to the first foreground-aware model is used to supervise the training of the first foreground-aware model.

[0265] In the present invention, before training the first foreground perception model using the training fusion features and the foreground area labeling results, a foreground perception loss function corresponding to the first foreground perception model can be constructed: Among them, L foreground,1 represents the foreground perception loss function corresponding to the first foreground perception model. The 1 in the subscript represents the first foreground perception model, and m j represents the value at the jth position in the foreground area result under the above view, p j,1 represents the value of the multimodal training foreground heat map predicted by the first foreground perception model at the jth position, and n represents the total number of positions in the foreground area annotation result.

[0266] When the first foreground perception model is trained by using the training fusion features and the foreground area annotation results, the foreground perception loss function corresponding to the first foreground perception model can be used to supervise the training of the first foreground perception model. The training supervision of the first foreground perception model is similar to the training supervision of the second foreground perception model. For a detailed description of this part, please refer to the relevant description of the second foreground perception model training supervision part above, which will not be repeated here. The difference is that the foreground perception loss function corresponding to the first foreground perception model can be compared with the second preset threshold, and the second preset threshold can be the same as the first preset threshold.

[0267] The target detection model training method provided by the embodiment of the present invention may further include, before using the foreground perception loss function corresponding to the first foreground perception model to supervise the training of the first foreground perception model:

[0268] Annotate the confusion area in the sample image in the training set; the confusion area is the confusion area of ​​the foreground and background, and in the confusion area annotation result, the value in the confusion area is greater than 1, and the value in the non-confusion area is equal to 1;

[0269] The foreground perception loss function corresponding to the first foreground perception model is weighted by using the confusion region labeling result to obtain a weighted foreground perception loss function of the confusion region corresponding to the first foreground perception model;

[0270] Using the foreground-aware loss function corresponding to the first foreground-aware model to supervise the training of the first foreground-aware model may include:

[0271] The training of the first foreground-aware model is supervised by using the foreground-aware loss function weighted by the confusion region corresponding to the first foreground-aware model.

[0272] In the present invention, similar to the processing of the foreground perception loss function corresponding to the second foreground perception model, the samples in the training set can also be annotated with confusion regions in the above view, and the confusion region annotation results can be used to participate in the foreground perception loss function corresponding to the first foreground perception model to obtain the corresponding confusion region weighted foreground perception loss function, thereby enhancing the first foreground perception model's ability to distinguish the confusion region of the foreground background. In particular, similar to the second foreground perception model and the first foreground perception model using the same foreground region annotation results for training, the second foreground perception model and the first foreground perception model can also use the same confusion region annotation results to process the corresponding foreground perception loss function.

[0273] Specifically, after obtaining the training set under the above view, the confusion area in the sample in the training set can be determined and the confusion area in the sample can be labeled. For the determination and labeling of the confusion area, please refer to the detailed description of the corresponding part above, which will not be repeated here. Afterwards, the foreground perception loss function corresponding to the first foreground perception model can be weighted using the confusion area labeling result to obtain the weighted foreground perception loss function of the confusion area corresponding to the first foreground perception model: Then, the foreground perception loss function weighted by the confusion area corresponding to the first foreground perception model can be used to supervise the training of the first foreground perception model, so as to improve the discrimination ability of the first foreground perception model for the confusion area by using the foreground perception loss function weighted by the confusion area, thereby reducing the interference of the background area and improving the target detection performance.

[0274] See also Fig.11 , which shows a schematic diagram of another target detection model training method provided by an embodiment of the present invention. A target detection model training method provided by an embodiment of the present invention, when the modality may include camera images and point clouds, and the view is a bird's-eye view, obtaining training data features of multiple modalities in the same view may include:

[0275] Acquire training image data of a training image under a camera view, and extract training image features from the training image data;

[0276] Using a two-dimensional to three-dimensional conversion method, the training image features are converted into training image features under a bird's-eye view;

[0277] The training point cloud data is acquired, and the training point cloud features under the bird's-eye view are extracted from the training point cloud data.

[0278] In the present invention, the modality may specifically include a camera image modality and a point cloud modality, wherein the camera image corresponds to a camera, and the point cloud corresponds to a laser radar or other sensors capable of acquiring point clouds. Furthermore, the above-mentioned view may specifically be a bird's-eye view.

[0279] On the basis of the above, when acquiring the training data features of multiple modalities in the same view, the training image data of the training image in the camera view (i.e., the training image data in the camera view) can be specifically acquired, and the training image features can be extracted from the training image data using the image backbone network. Since the camera image is a two-dimensional image, and the bird's-eye view is a three-dimensional view, the extracted training image features can be converted into training image features in the bird's-eye view using a two-dimensional to three-dimensional conversion method. Among them, when using the two-dimensional to three-dimensional conversion method, the depth prediction model can be used to predict the depth distribution first, and then the LSS (Lift, Splat, Shoot) method can be used to process the training image features to obtain the training image features in the bird's-eye view. Among them, the depth prediction model is also obtained by model training, and during model training, the corresponding depth distribution loss function can be used to supervise its training effect.

[0280] For the point cloud modality, training point cloud data can be obtained, and a point cloud backbone network can be used to extract training point cloud features under a bird's-eye view from the training point cloud data.

[0281] From the above, it can be seen that when the modalities include camera images and point clouds, a three-dimensional target detection dataset containing camera data and point cloud data can be used to train the model to ultimately obtain a target detection model.

[0282] Mapping the training image features and training point cloud features into a unified bird's-eye view space not only retains the rich semantic information in the image features, but also retains the three-dimensional spatial information contained in the point cloud features. It can better realize the complementarity of multimodal features and thus improve the target detection performance.

[0283] The target detection model training method provided by the embodiment of the present invention may further include, after extracting training image features from training image data and before converting the training image features into training image features under a bird's-eye view using a two-dimensional to three-dimensional conversion method:

[0284] Label the foreground area of ​​the training image;

[0285] Using the training image features and the training image annotated with the foreground region, training a third foreground perception model;

[0286] Using the third foreground perception model and training image features, a camera view training foreground heat map is obtained;

[0287] The foreground heat map and training image features are trained using the camera view to obtain the foreground enhanced training image features.

[0288] In the present invention, after extracting training image features from training image data, before using a two-dimensional to three-dimensional conversion method to convert the training image features into training image features under a bird's-eye view, the foreground area of ​​the training image can be annotated. When the training images corresponding to multiple cameras are included, the foreground area of ​​the training images corresponding to each camera image can be annotated respectively, that is, the foreground area of ​​the corresponding training images under the camera view is annotated, wherein the foreground area annotation under the i-th camera view can be recorded as Moreover, in the training image after the foreground area is labeled, the labeling value of the foreground area is greater than the labeling value of the background area, so that the training image after the foreground area is labeled can highlight the foreground area, thereby facilitating the prediction of the foreground area in the image and improving the perception ability of the foreground area.

[0289] Then, the third foreground perception model can be trained using the training image features and the training image after the foreground area is annotated. Among them, the training image features are the input of the third foreground perception model, and the training image after the foreground area is annotated is the output of the third foreground perception model. The third foreground perception model is trained using the input and output. After completing the training of the third foreground perception model, or after each preset number (such as once) of training the third foreground perception model, the training image features can be input into the third foreground perception model to predict the camera view training foreground heat map using the third foreground perception model. In the camera view training foreground heat map, the value of the foreground area is greater than the value of the background area. It can be seen that the camera view training foreground heat map can be regarded as a prediction score map of the foreground area and the background area, or as a prediction probability map of the foreground area and the background area. As can be seen from the above, the third foreground perception model can realize the prediction of the foreground area in the training image features, thereby improving the utilization of the foreground features by the target detection model, so as to improve the foreground background discrimination ability of the target detection model and improve the target detection performance.

[0290] After obtaining the camera view training foreground heat map, the camera view training foreground heat map can be used to perform weighted operations on the training image features to obtain foreground enhanced training image features, so as to utilize the camera view training foreground heat map to enhance the foreground features in the training data features, thereby facilitating the improvement of the target detection model's foreground perception capability and improving the target detection performance.

[0291] Among them, the corresponding foreground perception loss function can also be used for supervision during the training of the third foreground perception model, and the foreground perception loss function corresponding to the third foreground perception model is in the same form as the foreground perception loss function corresponding to the first foreground perception model and the second foreground perception model, respectively. The difference is that the foreground perception loss function corresponding to the third foreground perception model is under the camera view. In addition, the foreground perception loss function corresponding to the third foreground perception model can also be used to mark the confusion area in the training image (the confusion area marking method is the same as the confusion area marking method in the sample mentioned above, which will not be repeated here), and the foreground perception loss function corresponding to the third foreground perception model is weighted by the training image after the confusion area is marked to obtain the weighted foreground perception loss function of the confusion area corresponding to the third foreground perception model, and the weighted foreground perception loss function of the confusion area corresponding to the third foreground perception model is used to supervise the training of the third foreground perception model, and the weighted foreground perception loss function of the confusion area corresponding to the third foreground perception model is in the same form as the weighted foreground perception loss function of the confusion area corresponding to the first foreground perception model and the second foreground perception model. In addition, the structure of the third foreground perception model is the same as that of the first foreground perception model and the second foreground perception model.

[0292] An object detection model training method provided by an embodiment of the present invention may include:

[0293] Obtaining two-dimensional information of each target bounding box in the training image; the two-dimensional information includes the center point, width and length of the target bounding box;

[0294] Based on the two-dimensional information of each target bounding box, the foreground area of ​​the training image is labeled;

[0295] When a training image contains multiple target bounding boxes and there is an overlap between at least two target bounding boxes, the foreground annotation values ​​of the overlapped parts are averaged.

[0296] An object detection model training method provided by an embodiment of the present invention may include: labeling a foreground area of ​​a training image according to two-dimensional information of each object bounding box;

[0297] According to the center point, width and length of each target bounding box, the third inscribed ellipse of each target bounding box is correspondingly determined, the foreground area of ​​the third inscribed ellipse is marked according to the two-dimensional Gaussian distribution, and the area outside the third inscribed ellipse is marked as 0, so as to obtain the third marking result;

[0298] If there is an overlap between at least two target bounding boxes, the foreground annotation values ​​of the overlapped parts are averaged to obtain a global foreground annotation of the training image;

[0299] The global foreground annotation of the training image is added to the second auxiliary matrix to obtain the training image with the foreground region annotated; each element in the second auxiliary matrix is ​​the same and non-zero.

[0300] It should be noted that the method of labeling the foreground area of ​​the training image under the camera view is similar to the method of labeling the foreground area of ​​the samples in the training set under the above view. The difference is that there is no orientation angle in the two-dimensional information of the target bounding box in the training image. For details, please refer to the specific implementation process of the corresponding part in labeling the foreground area of ​​the samples in the training set under the above view, which will not be repeated here. In addition, each element in the second auxiliary matrix can also be equal to 1.

[0301] An object detection model training method provided by an embodiment of the present invention uses foreground enhancement training fusion feature training to obtain an object detection model, which may include:

[0302] Train the object detection model using the foreground-enhanced training fusion features;

[0303] The classification loss function and the regression loss function are used to supervise the training of the detection model until the classification loss function and the regression loss function meet the preset training conditions.

[0304] In the present invention, when the target detection model is obtained by training the foreground enhanced training fusion feature, the target detection model can be trained by using the foreground enhanced training fusion feature, and the classification loss function and the regression loss function can be used to supervise the training of the detection model until the classification loss function and the regression loss function meet the preset training conditions (the preset training conditions can specifically be that the classification loss function is less than or equal to the third preset threshold, the value of the regression loss function is less than or equal to the fourth preset threshold, etc.), so as to train the target detection model. In the above manner, a high-performance target detection model can be obtained to improve the target detection performance.

[0305] The present invention also provides a method for detecting a target. Fig.12 , which shows a flow chart of a target detection method provided by an embodiment of the present invention, which may include:

[0306] S121: Acquire features of data to be detected in multiple modalities in the same view.

[0307] S122: Fusing the data features to be detected in each modality under the view to obtain fused features to be detected.

[0308] S123: Obtain a multimodal foreground heat map of the foreground to be detected by using the fusion features to be detected and the first foreground perception model.

[0309] Among them, the first foreground perception model is trained using any of the target detection model training methods mentioned above.

[0310] S124: Utilizing the multimodal foreground heat map to be detected and the fused features to be detected, a foreground-enhanced fused feature to be detected is obtained.

[0311] S125: Input the foreground-enhanced fused features to be detected into the target detection model to obtain the target detection result.

[0312] The target detection model is trained using any target detection model training method.

[0313] An object detection method provided by an embodiment of the present invention may further include, after acquiring features of data to be detected in multiple modalities in the same view:

[0314] Using the second foreground perception model corresponding to each modality and the features of the data to be detected, a single-modality foreground heat map to be detected corresponding to each modality is obtained; the second foreground perception model corresponding to each modality is trained by any one of the target detection model training methods from the second target detection model training method to the eleventh target detection model training method described above;

[0315] Using the single-modality foreground heat map to be detected and the features of the data to be detected corresponding to each modality, the features of the data to be detected with enhanced intra-modality foreground corresponding to each modality are obtained;

[0316] The features of the data to be detected in each modality under the view are fused to obtain the fused features to be detected, which may include:

[0317] The features of the data to be detected with foreground enhancement in the modalities corresponding to each modality are fused.

[0318] An object detection method provided by an embodiment of the present invention may further include: after obtaining the intra-modal foreground enhanced data features to be detected corresponding to each modality by using the single-modal foreground heat map to be detected and the features of the data to be detected corresponding to each modality, the following steps may be performed:

[0319] Using the features of the data to be detected corresponding to each modality and the single-modality foreground heat map to be detected corresponding to other modalities, the features of the data to be detected with inter-modal foreground enhancement corresponding to each modality are obtained;

[0320] The features of the data to be detected with foreground enhancement in the modalities corresponding to each modality are fused, which may include:

[0321] The features of the data to be detected with the intra-modality foreground enhancement corresponding to each modality and the features of the data to be detected with the inter-modality foreground enhancement corresponding to each modality are fused.

[0322] An object detection method provided by an embodiment of the present invention fuses the features of the data to be detected with the intra-modality foreground enhancement corresponding to each modality and the features of the data to be detected with the inter-modality foreground enhancement corresponding to each modality, which may include:

[0323] The intra-modality foreground enhanced data features to be detected corresponding to each modality and the inter-modality foreground enhanced data features to be detected corresponding to each modality are combined according to the channel dimension to obtain the combined data features to be detected;

[0324] The combined features of the data to be detected are fused.

[0325] An object detection method provided by an embodiment of the present invention combines the features of the data to be detected with the intra-modality foreground enhancement corresponding to each modality and the features of the data to be detected with the inter-modality foreground enhancement corresponding to each modality according to the channel dimension, which may include:

[0326] Obtaining weight values ​​corresponding to the intra-modality foreground enhancement to-be-detected data features corresponding to each modality and the inter-modality foreground enhancement to-be-detected data features corresponding to each modality;

[0327] Using each weight value, weighted processing is performed on the corresponding intra-modality foreground enhanced data features to be detected and the inter-modality foreground enhanced data features to be detected;

[0328] The weighted intra-modality foreground enhanced data features to be detected and the weighted inter-modality foreground enhanced data features to be detected are combined.

[0329] An object detection method provided by an embodiment of the present invention, when the modality may include a camera image and a point cloud, and the view is a bird's-eye view, obtaining features of data to be detected of multiple modalities in the same view may include:

[0330] Acquire the image data to be detected of the camera's image to be detected, and extract the image features to be detected from the image data to be detected;

[0331] Using a two-dimensional to three-dimensional conversion method, the features of the image to be detected are converted into features of the image to be detected under a bird's-eye view;

[0332] Obtain the point cloud data to be detected, and extract the point cloud features to be detected under the bird's-eye view from the point cloud data to be detected.

[0333] An object detection method provided by an embodiment of the present invention may further include, after extracting the image features to be detected from the image data to be detected and before converting the image features to be detected under a bird's-eye view by using a two-dimensional to three-dimensional conversion method:

[0334] Using the third foreground perception model and the features of the image to be detected, a heat map of the foreground to be detected in the camera view is obtained; the third foreground perception model is obtained by using any one of the target detection model training methods from the seventeenth target detection model training method to the nineteenth target detection model training method;

[0335] The foreground heat map to be detected in the camera view and the features of the image to be detected are used to obtain the foreground enhanced features of the image to be detected.

[0336] Since the training process of target detection is similar to that of the target detection model, the detailed description of the relevant parts of the target detection method mentioned above in the present invention can refer to the description of the corresponding parts in the target detection model provided by the present invention. The difference is that target detection is achieved by processing the data to be detected using the trained foreground perception model and target detection model.

[0337] The present invention also provides a target detection model training device, see Fig.13 , which shows a schematic diagram of the structure of a target detection model training device provided by an embodiment of the present invention, which may include:

[0338] A first acquisition module 131 is used to acquire training data features of multiple modalities in the same view;

[0339] A first fusion module 132 is used to fuse the training data features of each modality under the view to obtain a training fusion feature;

[0340] The first training module 133 is used to mark the foreground area of ​​the samples in the training set under the view, and train the first foreground perception model by using the training fusion feature and the foreground area marking result; wherein the foreground marking value of the foreground area in the foreground area marking result is greater than the foreground marking value of the background area;

[0341] A first heat map obtaining module 134 is used to obtain a multimodal training foreground heat map using the first foreground perception model and the training fusion feature;

[0342] First, a foreground enhancement module 135 is obtained, which is used to obtain a foreground enhanced training fusion feature by using a multimodal training foreground heat map and a training fusion feature;

[0343] The target detection model obtaining module 136 is used to obtain the target detection model by training with the foreground enhanced training fusion features.

[0344] An object detection model training device provided by an embodiment of the present invention may further include:

[0345] A second training module is used to train a second foreground perception model corresponding to each modality by using the training data features of each modality and the foreground area annotation results after acquiring the training data features of multiple modalities in the same view;

[0346] The second heat map obtaining module is used to obtain a single-modality training foreground heat map corresponding to each modality by using the second foreground perception model and training data features corresponding to each modality;

[0347] Second, a foreground enhancement module is obtained, which is used to obtain the training data features of intra-modal foreground enhancement corresponding to each modality by using the single-modal training foreground heat map and training data features corresponding to each modality;

[0348] The first fusion module may include:

[0349] The first fusion unit is used to fuse the intra-modality foreground enhancement training data features corresponding to each modality.

[0350] An object detection model training device provided by an embodiment of the present invention may further include:

[0351] The first supervision module is used to supervise the training of the corresponding second foreground perception model by using the foreground perception loss function corresponding to each second foreground perception model when training the second foreground perception model corresponding to each modality by using the training data features of each modality and the image after the foreground area is annotated.

[0352] An object detection model training device provided by an embodiment of the present invention may further include:

[0353] A first labeling module is used to label the confusion area in the samples in the training set before supervising the training of the corresponding second foreground perception model using the foreground perception loss function corresponding to each second foreground perception model; wherein the confusion area is a confusion area of ​​the foreground and background, and in the confusion area labeling result, the value in the confusion area is greater than 1, and the value in the non-confusion area is equal to 1;

[0354] A first weighting module is used to weight the foreground perception loss function corresponding to each second foreground perception model by using the confusion region labeling result to obtain the weighted foreground perception loss function of the confusion region corresponding to each second foreground perception model;

[0355] The first supervision module may include:

[0356] The first supervision unit is used to supervise the training of the corresponding second foreground perception model by using the foreground perception loss function weighted by the confusion area corresponding to each second foreground perception model.

[0357] An object detection model training device provided by an embodiment of the present invention may further include:

[0358] A first determination module is used to determine the outer boundary of the confusion region and the inner boundary of the confusion region according to the foreground region marked in the sample before marking the confusion region in the training set;

[0359] The second determining module is used to determine the area between the outer boundary of the confusion area and the inner boundary of the confusion area as the confusion area.

[0360] An object detection model training device provided by an embodiment of the present invention may further include:

[0361] Third, a foreground enhancement module is used to obtain the training data features of inter-modal foreground enhancement corresponding to each modality by using the training data features corresponding to each modality and the single-modal training foreground heat maps corresponding to other modalities after obtaining the single-modal training foreground heat map corresponding to each modality by using the second foreground perception model corresponding to each modality and the training data features;

[0362] The first fusion unit may include:

[0363] The first fusion subunit is used to fuse the training data features of the intra-modality foreground enhancement corresponding to each modality and the training data features of the inter-modality foreground enhancement corresponding to each modality.

[0364] In an object detection model training device provided by an embodiment of the present invention, the first fusion subunit may include:

[0365] A first combining subunit is used to combine the training data features of the intra-modality foreground enhancement corresponding to each modality and the training data features of the inter-modality foreground enhancement corresponding to each modality according to the channel dimension to obtain combined training data features;

[0366] The second fusion subunit is used to fuse the combined training data features.

[0367] In an object detection model training device provided by an embodiment of the present invention, the first combination subunit may include:

[0368] A first acquisition subunit is used to acquire weight values ​​corresponding to the training data features of intra-modality foreground enhancement corresponding to each modality and the training data features of inter-modality foreground enhancement corresponding to each modality;

[0369] A first weighted processing subunit, configured to perform weighted processing on corresponding intra-modality foreground enhancement training data features and inter-modality foreground enhancement training data features using respective weight values;

[0370] The second combining subunit is used to combine the weighted intra-modality foreground enhancement training data features and the weighted inter-modality foreground enhancement training data features.

[0371] An embodiment of the present invention provides a target detection model training device, wherein the first foreground perception model and the second foreground perception model corresponding to each modality may include an encoding module and a decoding module.

[0372] An embodiment of the present invention provides a target detection model training device, wherein the encoding module may include multiple encoding units, the encoding unit may include a mean pooling unit and a first residual unit, the decoding module may include multiple decoding units, the decoding unit may include an upsampling unit and a second residual unit.

[0373] An embodiment of the present invention provides a target detection model training device, wherein the first residual unit and the second residual unit may include a batch normalization layer, a nonlinear activation layer and a 1×1 convolution layer.

[0374] In an object detection model training device provided by an embodiment of the present invention, the first training module 133 may include:

[0375] A first acquisition unit is used to acquire two-dimensional information of each target bounding box in the sample; the two-dimensional information may include a center point, width, length and orientation angle of the target bounding box;

[0376] A first labeling unit, used to label the foreground area of ​​the sample according to the two-dimensional information of each target bounding box;

[0377] When a sample contains multiple target bounding boxes and there is an overlap between at least two target bounding boxes, the foreground annotation values ​​of the overlapped parts are averaged.

[0378] In an object detection model training device provided by an embodiment of the present invention, the first labeling unit may include:

[0379] A first labeling subunit is used for determining a first inscribed ellipse of the current target bounding box according to the center point, width and length of the current target bounding box if the orientation angle of the current target bounding box in the sample is equal to 0, labeling the foreground area of ​​the first inscribed ellipse according to a two-dimensional Gaussian distribution, and labeling the area outside the first inscribed ellipse as 0 to obtain a first labeling result;

[0380] The second labeling subunit is used for determining a standard bounding box corresponding to the current target bounding box according to the orientation angle of the current target bounding box if the orientation angle of the current target bounding box in the sample is not equal to 0, determining a second inscribed ellipse of the standard bounding box according to the center point, width and length of the current target bounding box, labeling the foreground area of ​​the second inscribed ellipse according to a two-dimensional Gaussian distribution, and labeling the area outside the second inscribed ellipse as 0 to obtain a second foreground labeling result, and converting the second foreground labeling result into a third foreground labeling result according to a coordinate mapping relationship between the current target bounding box and the standard bounding box; the orientation angle of the standard bounding box is equal to 0;

[0381] A return subunit is used to take the next target bounding box in the sample as the current target bounding box, and return to execute the steps corresponding to if the orientation angle of the current target bounding box in the sample is equal to 0 or if the orientation angle of the current target bounding box in the sample is not equal to 0, until all target bounding boxes in the sample are processed, and if there is an overlap between at least two target bounding boxes, the foreground label values ​​of the overlapped parts are averaged to obtain the global foreground label of the sample;

[0382] The first adding subunit is used to add the global foreground label of the sample to the first auxiliary matrix to obtain an image with the foreground area labeled; each element in the first auxiliary matrix is ​​the same and non-zero.

[0383] An object detection model training device provided by an embodiment of the present invention may further include:

[0384] The second supervision module is used to supervise the training of the first foreground perception model by using the foreground perception loss function corresponding to the first foreground perception model when training the first foreground perception model by using the training fusion features and the image annotated with the foreground area.

[0385] An object detection model training device provided by an embodiment of the present invention may further include:

[0386] A second labeling module is used to label the confusion area in the samples in the training set before supervising the training of the first foreground perception model using the foreground perception loss function corresponding to the first foreground perception model; wherein the confusion area is a confusion area of ​​the foreground background, and in the confusion area labeling result, the value in the confusion area is greater than 1, and the value in the non-confusion area is equal to 1;

[0387] A second weighting module is used to weight the foreground perception loss function corresponding to the first foreground perception model by using the confusion region labeling result to obtain the weighted foreground perception loss function of the confusion region corresponding to the first foreground perception model;

[0388] The second supervision module may include:

[0389] The second supervision unit is used to supervise the training of the first foreground perception model by using the foreground perception loss function weighted by the confusion area corresponding to the first foreground perception model.

[0390] In an object detection model training device provided by an embodiment of the present invention, when the modality may include a camera image and a point cloud, and the view is a bird's-eye view, the first acquisition module 131 may include:

[0391] A second acquisition unit is used to acquire training image data of the training image under the camera view, and extract training image features from the training image data;

[0392] A first conversion unit, configured to convert the training image features into training image features under a bird's-eye view by using a two-dimensional to three-dimensional conversion method;

[0393] The third acquisition unit is used to acquire training point cloud data and extract training point cloud features under a bird's-eye view from the training point cloud data.

[0394] In the target detection model training device provided by the embodiment of the present invention, the first acquisition module 131 may further include:

[0395] A second labeling unit is used to label the foreground area of ​​the training image after extracting the training image features from the training image data and before converting the training image features into training image features under a bird's-eye view using a two-dimensional to three-dimensional conversion method;

[0396] A first training unit is used to train a third foreground perception model using training image features and training images annotated with foreground regions;

[0397] A first heat map obtaining unit is used to obtain a camera view training foreground heat map using a third foreground perception model and training image features;

[0398] First, a foreground enhancement unit is obtained, which is used to train the foreground heat map and the training image features using the camera view to obtain the foreground enhanced training image features.

[0399] In an object detection model training device provided by an embodiment of the present invention, the second labeling unit may include:

[0400] A second acquisition subunit is used to acquire two-dimensional information of each target bounding box in the training image; the two-dimensional information includes a center point, a width and a length of the target bounding box;

[0401] A third annotation subunit is used to annotate the foreground area of ​​the training image according to the two-dimensional information of each target bounding box;

[0402] When a training image contains multiple target bounding boxes and there is an overlap between at least two target bounding boxes, the foreground annotation values ​​of the overlapped parts are averaged.

[0403] In an object detection model training device provided by an embodiment of the present invention, the third labeling subunit may include:

[0404] The fourth labeling subunit is used to determine the third inscribed ellipse of each target bounding box according to the center point, width and length of each target bounding box, label the foreground area of ​​the third inscribed ellipse according to the two-dimensional Gaussian distribution, and label the area outside the third inscribed ellipse as 0 to obtain a third labeling result;

[0405] A global annotation subunit is obtained, which is used for averaging the foreground annotation values ​​of the overlapping parts if there is an overlapping part between at least two target bounding boxes, so as to obtain a global foreground annotation of the training image;

[0406] The second adding subunit is used to add the global foreground annotation of the training image to the second auxiliary matrix to obtain the training image after the foreground area is annotated; each element in the second auxiliary matrix is ​​the same and non-zero. An object detection model training device provided by an embodiment of the present invention can obtain the object detection model module 136, which may include:

[0407] The second training unit is used to train the target detection model using the foreground enhanced training fusion features;

[0408] A target detection model unit is obtained, which is used to supervise the training of the detection model using a classification loss function and / or a regression loss function until the classification loss function and the regression loss function meet preset training conditions.

[0409] The present invention also provides a target detection device, see Fig.14 , which shows a schematic diagram of the structure of a target detection device provided by an embodiment of the present invention, which may include:

[0410] The second acquisition module 141 is used to acquire features of data to be detected in multiple modalities in the same view;

[0411] The second fusion module 142 is used to fuse the data features to be detected in each modality under the view to obtain the fused features to be detected;

[0412] The fourth heat map obtaining module 143 is used to obtain a multimodal foreground heat map to be detected by using the fusion features to be detected and the first foreground perception model; the first foreground perception model is trained by using any of the target detection model training methods described above;

[0413] Fifth, a foreground enhancement module 144 is used to obtain foreground enhanced fusion features to be detected by using the multimodal foreground heat map to be detected and the fusion features to be detected;

[0414] The detection result obtaining module 145 is used to input the foreground enhanced fused features to be detected into the target detection model to obtain the detection result; the target detection model is trained using any of the target detection model training methods mentioned above.

[0415] An object detection model training device provided by an embodiment of the present invention may further include:

[0416] The fifth heat map obtaining module is used to obtain the features of the data to be detected in multiple modalities in the same view, and then obtain the single-modal foreground heat map to be detected corresponding to each modality by using the second foreground perception model corresponding to each modality and the features of the data to be detected; the second foreground perception model corresponding to each modality is obtained by training with any one of the target detection model training methods from the second target detection model training method to the eleventh target detection model training method mentioned above;

[0417] Sixth, a foreground enhancement module is used to obtain the foreground enhancement data features to be detected in the modality corresponding to each modality by using the single-modality foreground heat map to be detected and the data features to be detected corresponding to each modality;

[0418] The second fusion module may include:

[0419] The second fusion unit is used to fuse the features of the data to be detected with foreground enhancement in the modalities corresponding to each modality.

[0420] An object detection model training device provided by an embodiment of the present invention may further include:

[0421] Seventh, a foreground enhancement module is used to obtain the data features to be detected with the foreground enhancement within each modality corresponding to each modality by using the foreground heat map to be detected of the unimodality corresponding to each modality and the data features to be detected, and then obtain the data features to be detected with the foreground enhancement between each modality corresponding to each modality by using the data features to be detected corresponding to each modality and the unimodal foreground heat map to be detected corresponding to other modalities;

[0422] The second fusion unit may include:

[0423] The third fusion subunit is used to fuse the features of the to-be-detected data with intra-modality foreground enhancement corresponding to each modality and the features of the to-be-detected data with inter-modality foreground enhancement corresponding to each modality.

[0424] In an object detection model training device provided by an embodiment of the present invention, the third fusion subunit may include:

[0425] The third combining subunit is used to combine the data features to be detected with the intra-modality foreground enhancement corresponding to each modality and the data features to be detected with the inter-modality foreground enhancement corresponding to each modality according to the channel dimension to obtain the combined data features to be detected;

[0426] The fourth fusion subunit is used to fuse the combined features of the data to be detected.

[0427] In an object detection model training device provided by an embodiment of the present invention, the third fusion unit may include:

[0428] The third acquisition subunit is used to obtain weight values ​​corresponding to the data features to be detected for intra-modality foreground enhancement corresponding to each modality and the data features to be detected for inter-modality foreground enhancement corresponding to each modality;

[0429] A second weighted processing subunit is used to perform weighted processing on the corresponding intra-modality foreground enhancement to-be-detected data features and inter-modality foreground enhancement to-be-detected data features using each weight value;

[0430] The fourth combining subunit is used to combine the weighted intra-modality foreground enhanced data features to be detected and the weighted inter-modality foreground enhanced data features to be detected.

[0431] In an object detection model training device provided by an embodiment of the present invention, when the modality may include a camera image and a point cloud, and the view is a bird's-eye view, the second acquisition module 141 may include:

[0432] A fourth acquisition unit is used to acquire the image data to be detected of the image to be detected by the camera, and extract the image features to be detected from the image data to be detected;

[0433] A second conversion unit is used to convert the image features to be detected into the image features to be detected in a bird's-eye view by using a two-dimensional to three-dimensional conversion method;

[0434] The fifth acquisition unit is used to acquire the to-be-detected point cloud data, and extract the to-be-detected point cloud features in the bird's-eye view from the to-be-detected point cloud data.

[0435] In the target detection model training device provided by the embodiment of the present invention, the second acquisition module 141 may further include:

[0436] The second heat map obtaining unit is used to obtain a camera view foreground heat map of the image to be detected by using a third foreground perception model and the image features to be detected after extracting the image features to be detected from the image data to be detected and before converting the image features to be detected under a bird's-eye view by using a two-dimensional to three-dimensional conversion method; the third foreground perception model is obtained by using any one of the target detection model training methods from the seventeenth target detection model training method to the nineteenth target detection model training method mentioned above;

[0437] The second is to obtain a foreground enhancement unit, which is used to obtain foreground enhanced image features to be detected by using the foreground heat map to be detected in the camera view and the image features to be detected.

[0438] An embodiment of the present invention further provides an electronic device, which may include:

[0439] Memory for storing computer programs;

[0440] A processor is used to implement the steps of any of the above-mentioned target detection model training methods when executing a computer program, and / or the steps of any of the above-mentioned target detection methods.

[0441] An embodiment of the present invention also provides a readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of any of the above-mentioned target detection model training methods and / or the steps of any of the above-mentioned target detection methods.

[0442] The readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0443] For the description of the relevant parts of a target detection method, a target detection model training device, a target detection device, an electronic device and a readable storage medium provided by the present invention, please refer to the detailed description of the corresponding parts of a target detection model training method provided in an embodiment of the present invention, and will not be repeated here.

[0444] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device that includes a series of elements are inherent to the elements. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device that includes the elements. In addition, the part of the above-mentioned technical solution provided by the embodiment of the present invention that is consistent with the corresponding technical solution in the prior art in principle is not described in detail, so as to avoid excessive elaboration.

[0445] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target detection model training method, characterized in that: include: Obtain training data features of multiple modalities in the same view; Fusing the training data features of each of the modalities under the view to obtain training fusion features; Under the view, the samples in the training set are labeled with foreground regions, and a first foreground perception model is trained using the training fusion features and the foreground region labeling results; wherein in the foreground region labeling results, the foreground labeling value of the foreground region is greater than the foreground labeling value of the background region; Obtaining a multimodal training foreground heat map using the first foreground perception model and the training fusion features; Using the multimodal training foreground heat map and the training fusion feature, obtaining the training fusion feature with foreground enhancement; The target detection model is trained using the training fusion features with foreground enhancement.

2. The target detection model training method according to claim 1, characterized in that: After obtaining the training data features of multiple modalities in the same view, it also includes: Using the training data features of each of the modalities and the foreground area labeling results, training a second foreground perception model corresponding to each of the modalities; Using the second foreground perception model corresponding to each of the modalities and the training data features, a single-modal training foreground heat map corresponding to each of the modalities is obtained; Using the single-modality training foreground heat map corresponding to each of the modalities and the training data features, the training data features of the intra-modality foreground enhancement corresponding to each of the modalities are obtained; The training data features of each of the modalities under the view are fused, including: The training data features of the intra-modal foreground enhancement corresponding to each of the modalities are fused.

3. The target detection model training method according to claim 2, characterized in that: When the training data features of each of the modalities and the foreground area labeling results are used to train the second foreground perception model corresponding to each of the modalities, the method further includes: The foreground-aware loss function corresponding to each of the second foreground-aware models is used to supervise the training of the corresponding second foreground-aware model.

4. The target detection model training method according to claim 3, characterized in that: Before using the foreground-aware loss function corresponding to each of the second foreground-aware models to supervise the training of the corresponding second foreground-aware model, the method further includes: Annotating the confusion area in the samples in the training set; wherein the confusion area is a confusion area of ​​the foreground and background, and in the confusion area annotation result, the value in the confusion area in the image is greater than 1, and the value in the non-confusion area is equal to 1; weighting the foreground perception loss function corresponding to each of the second foreground perception models by using the confusion region labeling result to obtain the foreground perception loss function after the confusion region corresponding to each of the second foreground perception models is weighted; Using the foreground-aware loss function corresponding to each of the second foreground-aware models to supervise the training of the corresponding second foreground-aware models includes: The foreground-aware loss function weighted by the confusion region corresponding to each second foreground-aware model is used to supervise the training of the corresponding second foreground-aware model.

5. The target detection model training method according to claim 4, characterized in that: Before labeling the confusing areas in the samples in the training set, the method further includes: Determine the outer boundary of the confusion region and the inner boundary of the confusion region according to the foreground region marked in the sample; An area between the outer boundary of the confusion area and the inner boundary of the confusion area is determined as the confusion area.

6. The target detection model training method according to claim 2, characterized in that: After obtaining the single-modality training foreground heat map corresponding to each of the modalities by using the second foreground perception model corresponding to each of the modalities and the training data features, the method further includes: Using the training data features corresponding to each of the modalities and the single-modal training foreground heat maps corresponding to other modalities, the training data features corresponding to each of the modalities for inter-modal foreground enhancement are obtained; The training data features of the intra-modality foreground enhancement corresponding to each of the modalities are fused, including: The training data features of the intra-modality foreground enhancement corresponding to each of the modalities and the training data features of the inter-modality foreground enhancement corresponding to each of the modalities are fused.

7. The target detection model training method according to claim 6, characterized in that: The training data features of the intra-modality foreground enhancement corresponding to each of the modalities and the training data features of the inter-modality foreground enhancement corresponding to each of the modalities are fused, including: Combining the training data features of the intra-modality foreground enhancement corresponding to each of the modalities and the training data features of the inter-modality foreground enhancement corresponding to each of the modalities according to the channel dimension to obtain combined training data features; The combined training data features are fused.

8. The target detection model training method according to claim 7, characterized in that: The training data features of the intra-modality foreground enhancement corresponding to each of the modalities and the training data features of the inter-modality foreground enhancement corresponding to each of the modalities are combined according to the channel dimension, including: Obtaining weight values ​​corresponding to the training data features of intra-modality foreground enhancement corresponding to each of the modalities and the training data features of inter-modality foreground enhancement corresponding to each of the modalities; Using each of the weight values, weighted processing is performed on the corresponding training data features of intra-modality foreground enhancement and the training data features of inter-modality foreground enhancement; The weighted training data features of intra-modality foreground enhancement and the weighted training data features of inter-modality foreground enhancement are combined.

9. The target detection model training method according to claim 2, characterized in that: The first foreground perception model and the second foreground perception model corresponding to each of the modalities both include an encoding module and a decoding module.

10. The target detection model training method according to claim 9, characterized in that: The encoding module includes multiple encoding units, each encoding unit includes a mean pooling unit and a first residual unit. The decoding module includes multiple decoding units, each decoding unit includes an upsampling unit and a second residual unit.

11. The target detection model training method according to claim 10, characterized in that: The first residual unit and the second residual unit each include a batch normalization layer, a non-linear activation layer, and a 1×1 convolution layer.

12. The target detection model training method according to claim 1, characterized in that: The foreground area of ​​the samples in the training set is marked under the view, including: Acquire two-dimensional information of each target bounding box in the sample; the two-dimensional information includes a center point, a width, a length, and an orientation angle of the target bounding box; Marking the foreground area of ​​the sample according to the two-dimensional information of each target bounding box; When the sample includes a plurality of the target bounding boxes and there is an overlap between at least two of the target bounding boxes, the foreground annotation values ​​of the overlapped parts are averaged.

13. The target detection model training method according to claim 12, characterized in that: According to the two-dimensional information of each target bounding box, the foreground area of ​​the sample is marked, including: If the orientation angle of the current target bounding box in the sample is equal to 0, determining a first inscribed ellipse of the current target bounding box according to the center point, width and length of the current target bounding box, marking a foreground area for the first inscribed ellipse according to a two-dimensional Gaussian distribution, and marking an area outside the first inscribed ellipse as 0, to obtain a first marking result; If the orientation angle of the current target bounding box in the sample is not equal to 0, determining a standard bounding box corresponding to the current target bounding box according to the orientation angle of the current target bounding box, determining a second inscribed ellipse of the standard bounding box according to the center point, width and length of the current target bounding box, annotating the foreground area of ​​the second inscribed ellipse according to a two-dimensional Gaussian distribution, and marking the area outside the second inscribed ellipse as 0 to obtain a second foreground annotation result, and converting the second foreground annotation result into a third foreground annotation result according to a coordinate mapping relationship between the current target bounding box and the standard bounding box; the orientation angle of the standard bounding box is equal to 0; The next target bounding box in the sample is used as the current target bounding box, and the steps corresponding to if the orientation angle of the current target bounding box in the sample is equal to 0 or if the orientation angle of the current target bounding box in the sample is not equal to 0 are returned to be executed, until the foreground labeling results corresponding to all the target bounding boxes in the sample are obtained, and if there is an overlapping part between at least two of the target bounding boxes, the foreground labeling values ​​of the overlapping parts are averaged to obtain the global foreground labeling of the sample; The global foreground label of the sample is added to the first auxiliary matrix to obtain a foreground area labeling result; each element in the first auxiliary matrix is ​​the same and non-zero.

14. The target detection model training method according to claim 1, characterized in that: When the first foreground perception model is trained by using the training fusion features and the foreground area labeling results, the method further includes: The foreground-aware loss function corresponding to the first foreground-aware model is used to supervise the training of the first foreground-aware model.

15. The target detection model training method according to claim 14, characterized in that: Before using the foreground-aware loss function corresponding to the first foreground-aware model to supervise the training of the first foreground-aware model, the method further includes: Annotating the confusion area in the samples in the training set; wherein the confusion area is a confusion area of ​​the foreground and background, and in the confusion area annotation result, the value in the confusion area is greater than 1, and the value in the non-confusion area is equal to 1; weighting the foreground perception loss function corresponding to the first foreground perception model by using the confusion region labeling result to obtain the foreground perception loss function weighted by the confusion region corresponding to the first foreground perception model; Supervising the training of the first foreground-aware model by using a foreground-aware loss function corresponding to the first foreground-aware model includes: The foreground-aware loss function weighted by the confusion region corresponding to the first foreground-aware model is used to supervise the training of the first foreground-aware model.

16. The target detection model training method according to claim 1, characterized in that: When the modality includes a camera image and a point cloud, and the view is a bird's-eye view, obtaining training data features of multiple modalities in the same view includes: Acquire training image data of a training image under a camera view, and extract training image features from the training image data; Using a two-dimensional to three-dimensional conversion method, the training image features are converted into training image features under the bird's-eye view; Training point cloud data is acquired, and training point cloud features under the bird's-eye view are extracted from the training point cloud data.

17. The target detection model training method according to claim 16, characterized in that: After extracting the training image features from the training image data, and before converting the training image features into the training image features under the bird's-eye view using a two-dimensional to three-dimensional conversion method, the method further includes: Annotating the foreground area of ​​the training image; Training a third foreground perception model using the training image features and the training image with the foreground area annotated; Obtaining a camera view training foreground heat map using the third foreground perception model and the training image features; The foreground heat map and the training image features are trained using the camera view to obtain the foreground enhanced training image features.

18. The target detection model training method according to claim 17, characterized in that: The foreground area of ​​the training image is marked, including: Acquire two-dimensional information of each target bounding box in the training image; the two-dimensional information includes a center point, a width, and a length of the target bounding box; Annotating the foreground area of ​​the training image according to the two-dimensional information of each target bounding box; Wherein, when the training image contains a plurality of the target bounding boxes and there is an overlapping portion between at least two of the target bounding boxes, the foreground annotation values ​​of the overlapping portions are averaged.

19. The target detection model training method according to claim 18, characterized in that: According to the two-dimensional information of each target bounding box, the foreground area of ​​the training image is marked, including: Determine a third inscribed ellipse of each target bounding box according to the center point, width and length of each target bounding box, mark the foreground area of ​​the third inscribed ellipse according to a two-dimensional Gaussian distribution, and mark the area outside the third inscribed ellipse as 0, to obtain a third marking result; If there is an overlapping portion between at least two of the target bounding boxes, averaging the foreground annotation values ​​of the overlapping portion to obtain a global foreground annotation of the training image; The global foreground labeling of the training image is added to the second auxiliary matrix to obtain the training image with the foreground region labeled; each element in the second auxiliary matrix is ​​the same and non-zero.

20. The target detection model training method according to claim 1, characterized in that: The target detection model is obtained by training the training fusion features with foreground enhancement, including: Training the target detection model using the foreground-enhanced training fusion features; The classification loss function and the regression loss function are used to supervise the training of the detection model until the classification loss function and the regression loss function meet the preset training conditions.

21. A target detection method, characterized in that: include: Obtain the features of the data to be detected in multiple modalities in the same view; Fusing the data features to be detected of each of the modalities under the view to obtain fused features to be detected; Using the fusion features to be detected and the first foreground perception model, a multimodal foreground heat map to be detected is obtained; the first foreground perception model is trained using the target detection model training method according to any one of claims 1 to 20; Using the multimodal foreground heat map to be detected and the fused features to be detected, obtaining the fused features to be detected with foreground enhancement; The foreground-enhanced fused features to be detected are input into a target detection model to obtain a target detection result; the target detection model is trained using the target detection model training method as described in any one of claims 1 to 20.

22. The target detection method according to claim 21, characterized in that: After obtaining the data features to be detected in multiple modalities in the same view, it also includes: Using the second foreground perception model corresponding to each of the modalities and the features of the data to be detected, a single-modal foreground heat map to be detected corresponding to each of the modalities is obtained; the second foreground perception model corresponding to each of the modalities is trained by the target detection model training method according to any one of claims 2 to 11; Using the single-modality foreground heat map to be detected corresponding to each of the modalities and the features of the data to be detected, the features of the data to be detected with enhanced intra-modal foreground corresponding to each of the modalities are obtained; The features of the data to be detected of each of the modalities in the view are fused to obtain fused features to be detected, including: The features of the data to be detected with foreground enhancement in the modalities corresponding to the modalities are fused.

23. The target detection method according to claim 22, characterized in that: After obtaining the features of the data to be detected with enhanced intra-modal foregrounds corresponding to each of the modalities by using the single-modal foreground heat maps to be detected corresponding to each of the modalities and the features of the data to be detected, the method further includes: Using the features of the data to be detected corresponding to each of the modalities and the single-modal foreground heat maps to be detected corresponding to other modalities, the features of the data to be detected with inter-modal foreground enhancement corresponding to each of the modalities are obtained; The features of the data to be detected with enhanced foreground in the modalities corresponding to the modalities are fused, including: The features of the data to be detected with the intra-modality foreground enhancement corresponding to each of the modalities and the features of the data to be detected with the inter-modality foreground enhancement corresponding to each of the modalities are fused.

24. The target detection method according to claim 23, characterized in that: The features of the data to be detected with the intra-modality foreground enhancement corresponding to each of the modalities and the features of the data to be detected with the inter-modality foreground enhancement corresponding to each of the modalities are fused, including: Combining the data features to be detected with intra-modality foreground enhancement corresponding to each of the modalities and the data features to be detected with inter-modality foreground enhancement corresponding to each of the modalities according to the channel dimension to obtain combined data features to be detected; The combined features of the data to be detected are fused.

25. The target detection method according to claim 24, characterized in that: The features of the data to be detected for the intra-modality foreground enhancement corresponding to each of the modalities and the features of the data to be detected for the inter-modality foreground enhancement corresponding to each of the modalities are combined according to the channel dimension, including: Obtaining weight values ​​corresponding to the features of the data to be detected for intra-modality foreground enhancement corresponding to each of the modalities and the features of the data to be detected for inter-modality foreground enhancement corresponding to each of the modalities; Using each of the weight values, weighted processing is performed on the corresponding intra-modality foreground enhancement data features to be detected and the inter-modality foreground enhancement data features to be detected; The features of the data to be detected with the intra-modality foreground enhancement after weighted processing and the features of the data to be detected with the inter-modality foreground enhancement after weighted processing are combined.

26. The target detection method according to claim 21, characterized in that: When the modality includes a camera image and a point cloud, and the view is a bird's-eye view, acquiring features of data to be detected of multiple modalities in the same view includes: Acquire image data to be detected of an image to be detected by a camera, and extract features of the image to be detected from the image data to be detected; Using a two-dimensional to three-dimensional conversion method, the image features to be detected are converted into the image features to be detected under the bird's-eye view; Obtaining the point cloud data to be detected, and extracting the point cloud features to be detected in the bird's-eye view from the point cloud data to be detected.

27. The target detection method according to claim 26, characterized in that: After extracting the image features to be detected from the image data to be detected, and before converting the image features to be detected into the image features to be detected in the bird's-eye view by using a two-dimensional to three-dimensional conversion method, the method further includes: Using the third foreground perception model and the features of the image to be detected, a heat map of the foreground to be detected in the camera view is obtained; the third foreground perception model is trained using the target detection model training method according to any one of claims 17 to 19; The foreground-enhanced image features to be detected are obtained by utilizing the camera view foreground heat map to be detected and the image features to be detected.

28. A target detection model training device, characterized in that: include: A first acquisition module is used to acquire training data features of multiple modalities in the same view; A first fusion module, used for fusing the training data features of each of the modalities under the view to obtain a training fusion feature; A first training module is used to label the foreground area of ​​the samples in the training set under the view, and train a first foreground perception model using the training fusion features and the foreground area labeling results; wherein the foreground labeling value of the foreground area in the foreground area labeling results is greater than the foreground labeling value of the background area; A first heat map obtaining module, configured to obtain a multimodal training foreground heat map using the first foreground perception model and the training fusion features; First, a foreground enhancement module is obtained, which is used to obtain the training fusion feature for foreground enhancement by using the multimodal training foreground heat map and the training fusion feature; A target detection model module is obtained, which is used to train a target detection model using the training fusion features of foreground enhancement.

29. A target detection device, characterized in that: include: The second acquisition module is used to acquire the features of the data to be detected in multiple modalities in the same view; A second fusion module is used to fuse the data features to be detected of each of the modalities under the view to obtain a fused feature to be detected; A fourth heat map obtaining module is used to obtain a multimodal foreground heat map to be detected by using the fusion features to be detected and the first foreground perception model; the first foreground perception model is trained by the target detection model training method according to any one of claims 1 to 20; Fifth, a foreground enhancement module is obtained, which is used to obtain the foreground enhanced fusion feature to be detected by using the multimodal foreground heat map to be detected and the fusion feature to be detected; A detection result obtaining module is used to input the foreground-enhanced fusion feature to be detected into a target detection model to obtain a detection result; the target detection model is trained using the target detection model training method as described in any one of claims 1 to 20.

30. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the target detection model training method as described in any one of claims 1 to 20 and / or the steps of the target detection method as described in any one of claims 21 to 27 when executing the computer program.

31. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the target detection model training method as described in any one of claims 1 to 20, and / or the steps of the target detection method as described in any one of claims 21 to 27.

Citation Information

Patent Citations

  • Automatic difference calibration method, system and device in cross-modal target detection

    CN110334708A

  • Radar distance view and image-based multi-sensor fusion detection method, model and model training method

    CN114359664A

  • Image small target detection method based on improved Center Net

    CN114494812A

  • Target detection model training method and target detection method

    CN114663846A

  • High-angle shot target detection method and device based on multi-modal image fusion, and medium

    CN114998821A