Target detection method and device based on multi-stage knowledge distillation

Abstract

The invention provides a target detection method and device based on multi-stage knowledge distillation, and the method comprises the steps: inputting the multi-source perception data of a to-be-detected region collected by a plurality of different target detection devices in a first environment into a feature extraction module of a target detection model, and obtaining a multi-source feature map; inputting the multi-source feature map into a feature fusion module of a target detection model to obtain a multi-modal fusion feature; inputting the multi-modal fusion features into a detection module of a target detection model to obtain a target detection result of the to-be-detected area; wherein the target detection model is obtained by performing multi-stage knowledge distillation training on the basis of first multi-source sample sensing data, collected in a first environment, of a sample detection area, second multi-source sample sensing data, collected in a second environment, of the sample detection area and a target detection label of the sample detection area. According to the invention, a stable and high-precision target detection result can be effectively obtained in a complex environment.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a target detection method and device based on multi-stage knowledge distillation. Background Art

[0002] 3D object detection is an important part of realizing intelligent driving and intelligent manufacturing. This technology combines computer vision and deep learning methods to enable machines to accurately detect and recognize objects in the surrounding environment. Therefore, 3D object detection has become an indispensable key technology in application scenarios such as self-driving cars sensing road conditions and industrial robots detecting workpieces.

[0003] In existing 3D object detection algorithms, the accuracy of detection often depends on the quality of data collection. However, in actual scenarios, severe weather conditions such as rain, snow, fog and haze often occur, which reduces the quality of data collection and makes it difficult to ensure the accuracy of object detection.

[0004] Therefore, there is an urgent need to provide a target detection method and device based on multi-stage knowledge distillation to solve the above technical problems. Summary of the invention

[0005] The present invention provides a target detection method and device based on multi-stage knowledge distillation, which are used to solve the defects in the prior art that the quality of data collection under severe weather conditions is low, resulting in difficulty in ensuring the accuracy of target detection, and to effectively obtain stable and high-precision target detection results in complex environments.

[0006] The present invention provides a target detection method based on multi-stage knowledge distillation, comprising: Acquire multi-source perception data of the area to be detected collected by multiple different target detection devices in the first environment; Inputting the multi-source perception data into a feature extraction module of a target detection model to obtain a multi-source feature map; Inputting the multi-source feature map into the feature fusion module of the target detection model to obtain a multimodal fusion feature; Inputting the multimodal fusion features into the detection module of the target detection model to obtain the target detection result of the area to be detected; Among them, the target detection model is obtained by multi-stage knowledge distillation training based on the first multi-source sample perception data, the second multi-source sample perception data and the target detection label of the sample detection area; the first multi-source sample perception data is obtained by multiple different sample detection devices collecting data on the sample detection area under the first environment, and the second multi-source sample perception data is obtained by multiple different sample detection devices collecting data on the sample detection area under the second environment; the environmental quality of the second environment is higher than the environmental quality of the first environment.

[0007] According to a target detection method based on multi-stage knowledge distillation provided by the present invention, the target detection model is trained based on the following steps: Training a teacher model according to the second multi-source sample perception data and the target detection label, and constructing a student model corresponding to the teacher model; Obtaining a first loss function according to a change value between a global entropy value of the first multi-source sample perception data and a global entropy value of the second multi-source sample perception data; Obtaining a second loss function according to the self-affinity matrix of the first multi-source sample perception data and the self-affinity matrix of the second multi-source sample perception data; the self-affinity matrix of the first multi-source sample perception data is calculated by the student model, and the self-affinity matrix of the second multi-source sample perception data is calculated by the teacher model; Obtaining a third loss function according to a first target detection result and a second target detection result of the sample detection area; the first target detection result is obtained by the student model performing target detection according to the first multi-source sample perception data, and the second target detection result is obtained by the teacher model performing target detection according to the second multi-source sample perception data; Obtaining a fourth loss function according to noise data of each predicted bounding box in the first target detection result; Performing multi-stage knowledge distillation training on the student model according to the first loss function, the second loss function, the third loss function and the fourth loss function; The target detection model is obtained according to the student model after multiple iterations of training.

[0008] According to a target detection method based on multi-stage knowledge distillation provided by the present invention, the step of obtaining a change value between a global entropy value of the first multi-source sample perception data and a global entropy value of the second multi-source sample perception data comprises: For each multi-source sample perception data in the first multi-source sample perception data and the second multi-source sample perception data, dividing the multi-source sample perception data into a plurality of data blocks of the same size; Calculating the probability of occurrence of each pixel value in each of the data blocks; Calculating a local entropy value of each of the data blocks according to the probability; Summing the local entropy values ​​of all data blocks in the multi-source sample perception data to obtain a global entropy value of the multi-source sample perception data; Traversing the first multi-source sample perception data and the second multi-source sample perception data to obtain a global entropy value of the first multi-source sample perception data and a global entropy value of the second multi-source sample perception data; The absolute value of the difference between the global entropy value of the first multi-source sample perception data and the global entropy value of the second multi-source sample perception data is calculated to obtain the change value.

[0009] According to a target detection method based on multi-stage knowledge distillation provided by the present invention, obtaining a first loss function according to a change value between a global entropy value of the first multi-source sample perception data and a global entropy value of the second multi-source sample perception data includes: According to the change value, the multimodal fusion features of each position point in the first multi-source sample perception data output by the student model, the multimodal fusion features of each position point in the second multi-source sample perception data output by the teacher model, the Gaussian distribution mask value of each position point, and the feature scale information corresponding to each position point, the difference between the foreground feature learning performance of the student model and the foreground feature learning performance of the teacher model is calculated to obtain the first loss function.

[0010] According to a target detection method based on multi-stage knowledge distillation provided by the present invention, obtaining a third loss function according to the first target detection result and the second target detection result of the sample detection area includes: According to the distance between each detected target and the preset target, obtaining the distance weight corresponding to each detected target; According to the distance weight, weighted fusion is performed on the errors between the classification results of each of the detection targets in the first target detection result and the classification results of each of the detection targets in the second target detection result to obtain a classification loss function; According to the distance weight, weighted fusion is performed on the errors between the regression results of each of the detection targets in the first target detection result and the regression results of each of the detection targets in the second target detection result to obtain a regression loss function; The classification loss function and the regression loss function are fused and calculated to obtain the third loss function.

[0011] According to a target detection method based on multi-stage knowledge distillation provided by the present invention, obtaining a fourth loss function according to noise data of each predicted bounding box in the first target detection result includes: Calculating a noise ratio according to the noise data of each predicted bounding box to obtain a noise ratio value of each predicted bounding box; Calculate the noise weight according to the noise ratio value to obtain the noise weight of each predicted bounding box; According to the noise weight, a weighted average calculation is performed on the confidence scores of the predicted bounding boxes in the first target detection result to obtain the fourth loss function.

[0012] According to a target detection method based on multi-stage knowledge distillation provided by the present invention, the feature extraction module includes a data alignment unit and a plurality of feature extraction units corresponding to different data types; The step of inputting the multi-source perception data into a feature extraction module of a target detection model to obtain a multi-source feature map includes: The multi-source perception data is input into the data alignment unit to obtain multi-source mapping data; the multi-source mapping data is obtained by the data alignment unit mapping the multi-source perception data to the data space and time space corresponding to any target detection device according to the external parameter matrix and the internal parameter matrix of any target detection device; Inputting each of the mapping data into a feature extraction unit corresponding to the data type of each of the perception data to obtain a feature map of each of the perception data; The feature maps of the perception data from multiple sources are integrated to obtain the multi-source feature map.

[0013] The present invention also provides a target detection device based on multi-stage knowledge distillation, comprising: A data acquisition unit, used to acquire multi-source perception data of the area to be detected collected by multiple different target detection devices in the first environment; A first data processing unit, configured to input the multi-source perception data into a feature extraction module of a target detection model to obtain a multi-source feature map; A second data processing unit is used to input the multi-source feature map into the feature fusion module of the target detection model to obtain a multimodal fusion feature; A target detection unit, used for inputting the multimodal fusion feature into the detection module of the target detection model to obtain the target detection result of the area to be detected; Among them, the target detection model is obtained by multi-stage knowledge distillation training based on the first multi-source sample perception data, the second multi-source sample perception data and the target detection label of the sample detection area; the first multi-source sample perception data is obtained by multiple different sample detection devices collecting data on the sample detection area under the first environment, and the second multi-source sample perception data is obtained by multiple different sample detection devices collecting data on the sample detection area under the second environment; the environmental quality of the second environment is higher than the environmental quality of the first environment.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements a target detection method based on multi-stage knowledge distillation as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a target detection method based on multi-stage knowledge distillation as described in any one of the above.

[0016] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned target detection methods based on multi-stage knowledge distillation.

[0017] The target detection method and device based on multi-stage knowledge distillation provided by the present invention integrate multiple sample detection devices to detect multi-source sample perception data of the same sample detection area under different environments, and use the multi-source sample perception data of the same sample detection area under different environments to perform multi-stage knowledge distillation, so as to improve data quality through multi-source perception data integration, and through multi-stage knowledge distillation training, the target detection model trained in this way can show a detection effect equivalent to or even better than that in normal weather conditions under severe weather conditions. Therefore, in practical applications, the target detection model based on this training is used to perform feature extraction, feature fusion and target detection on multi-source perception data collected by multiple different target detection devices under severe weather conditions, so that the target detection results obtained by detection can have good adaptability and robustness in severe weather environments, so as to effectively overcome the adverse effects of severe weather on target detection accuracy, thereby achieving effective acquisition of stable and high-precision target detection results in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of the target detection method based on multi-stage knowledge distillation provided by the present invention.

[0020] Figure 2 It is a flowchart of the training steps of the target detection model provided by the present invention.

[0021] Figure 3 It is a schematic diagram of the data alignment effect provided by the present invention.

[0022] Figure 4 It is a schematic diagram of simulation comparison results of different target detection methods provided by the present invention.

[0023] Figure 5 It is a structural schematic diagram of a target detection device based on multi-stage knowledge distillation provided by the present invention.

[0024] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Three-dimensional object detection is an important part of realizing intelligent driving and intelligent manufacturing. This technology combines computer vision and deep learning methods to enable machines to accurately detect and recognize objects in the surrounding environment. The uniqueness of three-dimensional object detection is that it can not only identify the category of the object, but also determine the precise spatial position and orientation angle of the object in the real world, as well as the spatial relationship between the object and other surrounding objects. This comprehensive and detailed environmental perception capability is difficult to achieve with other technologies. For this reason, three-dimensional object detection has become an indispensable key technology in application scenarios such as self-driving cars sensing road conditions and industrial robots detecting workpieces.

[0027] In existing 3D object detection technologies, the accuracy of detection often depends on the quality of data collection. However, adverse weather conditions (or bad weather conditions), such as rain, snow, fog and haze, seriously affect the data collection quality of detection equipment, thereby reducing the performance of detection technology. Therefore, adverse weather conditions pose severe challenges to 3D object detection technology. Specifically, adverse weather conditions pose severe challenges to 3D object detection in the following aspects: First, rainy and snowy weather will generate a lot of noise and blur in the camera image, making the outline of the target in the image unclear and the key features submerged. This greatly increases the difficulty of accurately extracting target information from the image. At the same time, rainy and snowy weather will also form a layer of dynamic occlusion on the image, partially or completely covering the target, causing the detection algorithm to be unable to capture the complete target.

[0028] Secondly, haze weather can seriously weaken the penetration of light, making distant targets blurry or even disappearing in the image. This reduction in visibility makes it difficult for detection algorithms to obtain reliable information about distant targets. In addition, haze can also change the color and contrast of the target, making it difficult to distinguish it from the background, making it more difficult to accurately segment the target.

[0029] Furthermore, adverse weather conditions can also interfere with LiDAR detection. Raindrops, snowflakes, etc. will generate a large amount of noise, which will mix into the real target point cloud and disrupt the structure and distribution of the point cloud. These scattered noise points will mislead the detection algorithm, generate false targets or block the real targets. At the same time, lasers will scatter and attenuate when passing through haze, causing the echo signal to weaken, the point cloud density to decrease, and the distant targets to disappear completely.

[0030] In summary, in actual scenarios, severe weather conditions will lead to reduced data collection quality, making it difficult to ensure target detection accuracy.

[0031] In this regard, an embodiment of the present application provides a target detection method based on multi-stage knowledge distillation to solve the above technical problems.

[0032] Figure 1 This is a flow chart of the target detection method based on multi-stage knowledge distillation provided by the present invention; this method can be applied to various scenarios where target detection is required, such as autonomous driving scenarios, industrial robot scenarios, etc., and this embodiment does not specifically limit this. The following takes the autonomous driving scenario as an example to describe the target detection method provided by this embodiment.

[0033] like Figure 1 As shown, the method includes step 110 , step 120 , step 130 and step 140 .

[0034] Step 110: Acquire multi-source perception data of the area to be detected collected by multiple different target detection devices in the first environment.

[0035] The multiple different target detection devices here are multiple different devices that can collect perception data, including but not limited to cameras, radar sensors, and satellite remote sensors, which are not specifically limited in this embodiment.

[0036] The following describes the method provided in this embodiment by taking a camera and a radar sensor as target detection devices as an example. The target detection method can also be implemented by referring to this for other detection devices.

[0037] Optionally, when performing target detection, multiple different target detection devices are first called to collect perception data of the area to be detected in the first environment to obtain the perception data of the area to be detected collected by different target detection devices, and the perception data from multiple different sources are integrated to obtain multi-source perception data of the area to be detected, which includes but is not limited to image data collected by the camera and point cloud data collected by the radar sensor. The first environment here can be an environment under adverse weather conditions, such as cloudy days, rainy days, foggy days, etc.; the area to be detected here is the area where target detection is required, such as the surrounding area of ​​a vehicle, etc., and this embodiment does not specifically limit this.

[0038] Step 120, input the multi-source perception data into the feature extraction module of the target detection model to obtain a multi-source feature map; wherein the target detection model is obtained by multi-stage knowledge distillation training based on the first multi-source sample perception data, the second multi-source sample perception data and the target detection label of the sample detection area; the first multi-source sample perception data is obtained by multiple different sample detection devices collecting data on the sample detection area under the first environment, and the second multi-source sample perception data is obtained by multiple different sample detection devices collecting data on the sample detection area under the second environment; the environmental quality of the second environment is higher than the environmental quality of the first environment.

[0039] Optionally, after acquiring the multi-source perception data, the multi-source perception data may be input into a feature extraction module of the target detection model, so that the feature extraction module performs feature extraction on each perception data to obtain a multi-source feature map.

[0040] It should be noted that the multi-source feature map here can be obtained by first performing data processing on the multi-source perception data by the feature extraction module, such as time data alignment and / or spatial data alignment, and then performing feature extraction, so as to effectively solve the errors caused by the sampling frequency of multi-source perception data (such as point cloud data and image data) in adverse weather conditions, thereby improving the target detection performance.

[0041] Exemplarily, in some embodiments, the feature extraction module includes a data alignment unit and a plurality of feature extraction units corresponding to different data types; Step 120 specifically includes: The multi-source perception data is input into the data alignment unit to obtain multi-source mapping data; the multi-source mapping data is obtained by the data alignment unit mapping the multi-source perception data to the data space and time space corresponding to any target detection device according to the external parameter matrix and the internal parameter matrix of any target detection device; Inputting each of the mapping data into a feature extraction unit corresponding to the data type of each of the perception data to obtain a feature map of each of the perception data; The feature maps of the perception data from multiple sources are integrated to obtain the multi-source feature map.

[0042] It should be noted that adverse weather conditions can also affect the temporal and spatial consistency of data collected by different detection devices. For example, raindrops appear differently in image data collected by cameras and point cloud data collected by lidar sensors, making it difficult to accurately align the two types of data. This inconsistency is further exacerbated when the vehicle moves at high speeds. The mismatch between different detection devices makes multimodal fusion difficult.

[0043] In this regard, in step 120 provided in the embodiment of the present application, a data alignment unit and feature extraction units corresponding to multiple different data types are configured in the feature extraction module, so that in the feature extraction process, the multi-source perception data collected by different detection devices can be first unified into the same data space and time space through the data alignment unit, so that the multi-source perception data collected by different detection devices can be accurately configured, and then, feature extraction is performed on the configured perception data of different data types through the feature extraction units corresponding to the different data types, so as to effectively solve the errors in the perception data of different data sources under adverse weather conditions caused by the sampling frequencies of the corresponding detection devices, thereby accurately obtaining the multi-source feature map and improving the performance of subsequent multimodal feature fusion.

[0044] Optionally, the specific implementation steps of step 120 are as follows: The multi-source perception data is input into a data alignment unit, which uses the external parameter matrix and the internal parameter matrix of any target detection device (such as a camera) to project the multi-source perception data into the data space corresponding to any target detection device. Then, the multi-source perception data after the data space alignment is optimized and adjusted in time and space in a step, so that the time and space of the optimized and adjusted multi-source perception data are synchronized with the time and space corresponding to any target detection device, thereby obtaining the final data space and time-space synchronized multi-source mapping data.

[0045] The following describes the data alignment steps provided in this embodiment in detail using a specific example.

[0046] Assume that the data collected by two detection devices, camera and lidar sensor, are used as the source of multi-source perception data; in the data alignment process, the data alignment module can use the external parameters of the camera to convert the point cloud data collected by the lidar sensor from the vehicle coordinate system to the camera coordinate system, and then use the internal parameters of the camera to project the point cloud data in the camera coordinate system onto the image plane, and establish the corresponding relationship between the point cloud data and the image data. The specific calculation formula is as follows: = ; in, are pixel coordinates, is the camera’s intrinsic parameter matrix, is the physical coordinate.

[0047] After the point cloud image mapping is performed, the obtained point cloud image mapping relationship can be optimized and adjusted by the time domain modeling method. Specifically, by mapping the radar sampling time to the image sampling time, the point cloud data is converted from the acquisition time t1 to the time t2 consistent with the image data, thereby achieving time synchronization between the point cloud and the image, which significantly reduces the error and noise caused by the lack of synchronization, thereby not only enhancing the geometric consistency of the data, but also improving the robustness and reliability of the system under severe weather conditions. Among them, the process of time-space alignment can be expressed by the following formula: P t2 = T t1 → t2 × P t1 ; Among them, P t1 represents the point cloud data collected at time t1, P t2 Represents the point cloud data after conversion to time t2, T t1 → t2 Represents the transformation matrix from time t1 to t2.

[0048] It should be noted that since the conversion coordinates of the two time periods cannot be directly converted, the reference time t0 is needed as an intermediate bridge. First, the transformation matrix T from time t0 to t1 is calculated by the following formula: t0 → t1 and the transformation matrix T from time t0 to t2 t0 → t2 : T t0 → t1 = M t1 × M t0 -1 ; T t0 → t2 = M t2 × M t0 -1 ; Among them, M t0 、M t1 and M t2 Represent the transformation matrices at time t0, t1 and t2, M t0 -1 Indicates M t0 The inverse matrix of .

[0049] Next, we can use T t0 → t2 and T t0 → t1 Multiply the inverse matrix of to get the direct transformation matrix T from time t1 to t2 t1 → t2 : T t1 → t2 = T t0 → t2 × T t0 → t1 -1 ; Among them, T t0 → t1 -1 Indicates T t0 → t1 The inverse matrix of .

[0050] Finally, traverse the point cloud data P t1 For each point in , multiply it by the transformation matrix T t1 → t2 , get the point cloud data P after time consistency correction t2 : p t2 = T t1 → t2 × p t1 , p t1 P t1 ; Among them, p t1 Represents point cloud data P t1 A point in t2 Represents a point after time-space conversion.

[0051] In summary, the above time domain modeling method can effectively optimize the point cloud image mapping relationship and improve the accuracy and robustness of multi-sensor data fusion.

[0052] After obtaining multi-source mapping data through the data alignment module, each mapping data can be input into the feature extraction unit corresponding to the data type of each perception data, so that the feature extraction unit corresponding to the data type of each perception data can extract features from each mapping data to obtain a feature map of each perception data, and the feature maps of each perception data can be integrated to obtain a multi-source feature map. Among them, the model structure of the feature extraction unit corresponding to different data types is different, such as the feature extraction unit corresponding to image data can be a feature pyramid network, the feature extraction unit corresponding to point cloud data can be a voxel-based feature extraction network, etc. This embodiment does not specifically limit this.

[0053] The following describes the feature extraction steps performed by the feature extraction unit using a specific example.

[0054] In the feature extraction process, the corresponding two-dimensional and three-dimensional feature extraction networks can be used to extract features of image data and point cloud data respectively; for example, voxel-based regional convolutional neural network (Voxel-RCNN for short) and feature pyramid network (FPN) are used as three-dimensional backbone networks to extract feature representation of point cloud data and feature representation of image data respectively.

[0055] Among them, for the point cloud data, it is input into the Voxel-RCNN network for feature extraction. The Voxel-RCNN network is a voxel-based 3D target detection network that can efficiently process sparse point cloud data. Specifically, the dimension of the point cloud data is (N, 4), where N is the number of point clouds, and 4 represents the attributes of the point, including x, y, z coordinate values ​​and reflection intensity. Before entering the network, the Voxel-RCNN network divides the point cloud into a series of voxels, each of which contains all the points located in the volume. The data dimension after voxelization becomes (X, Y, Z, M, F), where X, Y, Z represent the number of voxels divided along the x, y, and z axes, respectively, M represents the maximum number of points in each voxel, and F represents the feature dimension of each point. Next, the Voxel-RCNN network encodes the point cloud in each voxel and generates a voxel feature vector. At this time, the output dimension becomes (X, Y, Z, C), where C represents the feature vector dimension of each voxel. Then, the voxel feature vector is input into the 3D convolutional neural network to extract the spatial features of the point cloud. After a series of 3D convolution operations, the dimension of the point cloud feature becomes (X', Y', Z', C'), where X', Y', Z' represents the voxel size after convolution, and C' represents the number of feature channels after convolution. Finally, the Voxel-RCNN network projects the 3D voxel features onto a 2D plane and uses 2D convolution to further extract features to obtain the final feature representation of the point cloud data, with dimensions (Hp, Wp, Cp), where Hp and Wp represent the height and width of the feature map, respectively, and Cp represents the number of feature channels.

[0056] For image data, it is input into the FPN two-dimensional backbone network for feature extraction. FPN is a feature pyramid network for target detection, which can extract image features at different scales. Specifically, first, an image with a dimension of (H, W, 3) is input into a pre-trained FPN backbone network (that is, a backbone network of FPN pre-trained based on general image data), where H and W represent the height and width of the image, respectively, and 3 represents the channel. After the image is convolved and pooled, a series of feature maps are obtained at different feature layers. These feature maps have different spatial resolutions and semantic information. In order to make comprehensive use of these feature maps, FPN will upsample the feature maps from top to bottom and add them element by element with the feature maps of the corresponding layers to obtain the fused feature maps. This process will be repeated at multiple scales, and finally a series of feature maps with rich semantic information and high resolution are obtained, whose dimensions are (H', W', C'), where H' and W' represent the height and width of the feature map, and C' represents the number of feature channels.

[0057] As can be seen from the above, by combining the two backbone networks of Voxel-RCNN and FPN, it is possible to extract the three-dimensional features of point cloud data from two different perspectives, voxel and image, and make full use of the geometric and semantic information of the data. At the same time, the multi-scale feature fusion strategy further enhances the representation ability of the features, so that the target detection method of this embodiment can achieve high-precision target detection in complex three-dimensional scenes.

[0058] In summary, the method provided in this embodiment, in the feature extraction process, obtains multi-source mapping data by inputting multi-source perception data into a data alignment unit, and using the external and internal parameter matrices of any target detection device to accurately map it to a unified data and time space; then, for each data type of perception data, its mapping data is input into the corresponding feature extraction unit to extract a feature map; finally, the feature maps of these multi-source perception data are integrated to form a more comprehensive and accurate multi-source feature map, thereby significantly improving the accuracy and robustness of target detection, and effectively coping with complex and changeable detection environments.

[0059] The target detection model here is a model that can accurately detect targets in an unfavorable weather environment (i.e., the first environment); the training steps of the target detection model specifically include: based on the first multi-source sample perception data of the sample detection area collected by multiple different sample detection devices in the first environment, the second multi-source sample perception data of the sample detection area collected by multiple different sample detection devices in the second environment, and the target detection label of the sample detection area, the student model and the teacher model are trained by multi-stage knowledge distillation. Exemplarily, the initial teacher model is first iteratively trained based on the second multi-source sample perception data and the target detection label of the sample detection area to train a teacher model that can accurately detect targets in the second environment (i.e., normal weather environment, such as sunny days); then, based on the feature data and detection results output by the student model in the first multi-source sample perception data, and the feature data and detection results output by the teacher model in the second multi-source sample perception data, multiple loss functions are obtained, so as to iteratively train the student model based on the obtained multiple loss functions to obtain a target detection model that can accurately detect targets in unfavorable weather environments. The loss function here includes but is not limited to a loss function used to characterize the difference in environmental distribution corresponding to the first multi-source sample perception data and the second multi-source sample perception data, a loss function for the difference in feature distribution corresponding to the first multi-source sample perception data and the second multi-source sample perception data, a loss function for the difference in response distribution of target detection results corresponding to the first multi-source sample perception data and the second multi-source sample perception data, and a loss function for suppressing the noise of the target detection results corresponding to the first multi-source sample perception data. This embodiment does not specifically limit this.

[0060] Step 130: input the multi-source feature map into the feature fusion module of the target detection model to obtain a multimodal fusion feature.

[0061] Optionally, after obtaining the multi-source feature map, the multi-source feature map can be input into a feature fusion module, which first performs perspective conversion on the multi-source feature map to convert it to the same perspective, and fuses the multi-source feature map after perspective conversion to provide more comprehensive environmental perception information, which is more helpful for understanding the data distribution characteristics in complex scenarios such as adverse weather, thereby helping to improve target detection performance.

[0062] The feature fusion module here can be constructed based on various network models that can perform multimodal feature fusion, such as convolutional neural networks, etc., and this embodiment does not specifically limit this.

[0063] The following is a specific description of the multimodal feature fusion steps provided in this embodiment using a specific example.

[0064] The image features are converted into a bird's-eye view, and then the point cloud features are fused with the image cone features using a convolutional neural network to generate multimodal fusion features. The specific formula is as follows: = Conv(concat( )); Among them, Conv represents convolution operation, concat represents concatenation operation, They are the point cloud features and image features after perspective conversion respectively.

[0065] Step 140: input the multimodal fusion feature into the detection module of the target detection model to obtain the target detection result of the area to be detected.

[0066] Optionally, after obtaining the multimodal fusion features, the multimodal fusion features can be input into the detection module, so that the detection module applies the multimodal fusion features to perform target detection on the area to be detected to obtain the target detection result of the area to be detected; the target detection result here includes at least the category of the detected target and the coordinate information of the bounding box.

[0067] The method provided in this embodiment integrates multiple sample detection devices to detect multi-source sample perception data of the same sample detection area under different environments, and uses the multi-source sample perception data of the same sample detection area under different environments to perform multi-stage knowledge distillation, so as to improve data quality through multi-source perception data integration, and through multi-stage knowledge distillation training, the target detection model trained in this way can show detection effects under severe weather conditions that are equivalent to or even better than those under normal weather conditions. Therefore, in actual applications, the target detection model based on this training is used to perform feature extraction, feature fusion and target detection on multi-source perception data collected by multiple different target detection devices under severe weather conditions, so that the target detection results obtained by detection can have good adaptability and robustness in severe weather environments, so as to effectively overcome the adverse effects of severe weather on target detection accuracy, and thus achieve effective acquisition of stable and high-precision target detection results in complex environments.

[0068] In some embodiments, the target detection model is trained based on the following steps: Training a teacher model according to the second multi-source sample perception data and the target detection label, and constructing a student model corresponding to the teacher model; Obtaining a first loss function according to a change value between a global entropy value of the first multi-source sample perception data and a global entropy value of the second multi-source sample perception data; Obtaining a second loss function according to the self-affinity matrix of the first multi-source sample perception data and the self-affinity matrix of the second multi-source sample perception data; the self-affinity matrix of the first multi-source sample perception data is calculated by the student model, and the self-affinity matrix of the second multi-source sample perception data is calculated by the teacher model; Obtaining a third loss function according to a first target detection result and a second target detection result of the sample detection area; the first target detection result is obtained by the student model performing target detection according to the first multi-source sample perception data, and the second target detection result is obtained by the teacher model performing target detection according to the second multi-source sample perception data; Obtaining a fourth loss function according to noise data of each predicted bounding box in the first target detection result; Performing multi-stage knowledge distillation training on the student model according to the first loss function, the second loss function, the third loss function and the fourth loss function; The target detection model is obtained according to the student model after multiple iterations of training.

[0069] Optionally, the training steps of the target detection model specifically include: First, a teacher model is trained using the second multi-source sample perception data and corresponding target detection labels under normal weather conditions (i.e., high-quality environment, or second environment), so that the teacher model trained thereby can accurately detect targets under normal weather conditions. An initial student model is then constructed accordingly.

[0070] In addition, the global entropy changes of the first multi-source sample perception data under severe weather conditions (i.e., low-quality environment, or first environment) and the second multi-source sample perception data under normal weather conditions are calculated, and the first loss function is constructed based on this, aiming to measure the changes in the distribution of data features under different weather conditions.

[0071] In addition, the first multi-source sample perception data and the second multi-source sample perception data input into the student model and the teacher model are similarly evaluated to measure the overall impact of the data distribution differences in different scenarios, thereby obtaining the second loss function. The specific implementation steps are as follows: The self-affinity matrices of the teacher model and the student model are calculated separately to capture the intrinsic structure and similarity relationship of the model output. The affinity matrix includes structural similarity and spatial similarity, which are defined as follows: S1(i,j)= ; S2(i,j) = ; Among them, S1(i, j) and S2(i, j) represent the structural similarity and spatial similarity between the multimodal fusion sub-features corresponding to position indexes i and j, respectively. i and j represent position indexes to represent features at different positions. represents a constant whose value is less than the set value, and ||·||2 represents the L2 norm.

[0072] Then, the self-affinity matrix is ​​obtained by combining these two similarities: A = × ; in, and denote the normalized structural similarity and spatial similarity, respectively. is a hyperparameter used to balance the two similarity weights.

[0073] Next, a self-attention mechanism is applied to the self-affinity matrix of the student model to help the student model capture the global context and suppress noise in the output. The calculation formula is as follows: Attn. = softmax(Q s , K s T ) V s ; Among them, Q, K and V are obtained by convolution and projection of affinity matrix, and s represents the student model.

[0074] Next, by applying a cross-attention mechanism between the student model and the teacher model, the knowledge of the teacher model is used to guide and optimize the similarity relations in the student model: = softmax(Q s , K t T ) V t ; in, The final self-affinity matrix output by the student model; Kt and Vt are calculated in the same way as the Student model; Finally, the second loss function L is used sce The self-affinity matrix of the student model is encouraged to be similar to that of the teacher model. The specific calculation formula is as follows: L sce = ||A s - A t || 2 ; Among them, A s The final self-affinity matrix output by the student model (that is, the self-affinity matrix corrected by the attention mechanism) ), At =A is the original self-affinity matrix output by the teacher model (that is, the self-affinity matrix obtained by combining structural similarity and spatial similarity).

[0075] In addition, by comparing the first target detection result obtained by the student model based on the first multi-source sample perception data and the second target detection result obtained by the teacher model based on the second multi-source sample perception data, a third loss function is constructed to directly optimize the detection performance of the student model.

[0076] In addition, in order to improve the robustness of the student model to noisy data, a fourth loss function can be constructed according to the noise data of each predicted bounding box in the first target detection result.

[0077] Finally, these four loss functions are combined to perform multi-stage knowledge distillation training on the student model. Through multiple iterative training, the student model gradually converges, and ultimately the optimized target detection model can not only maintain high accuracy under normal weather conditions, but also show excellent detection performance under severe weather conditions, effectively overcoming the adverse effects of reduced data acquisition quality on target detection accuracy.

[0078] In some embodiments, the step of obtaining the change value between the global entropy value of the first multi-source sample perception data and the global entropy value of the second multi-source sample perception data includes: For each multi-source sample perception data in the first multi-source sample perception data and the second multi-source sample perception data, dividing the multi-source sample perception data into a plurality of data blocks of the same size; Calculating the probability of occurrence of each pixel value in each of the data blocks; Calculating a local entropy value of each of the data blocks according to the probability; Summing the local entropy values ​​of all data blocks in the multi-source sample perception data to obtain a global entropy value of the multi-source sample perception data; Traversing the first multi-source sample perception data and the second multi-source sample perception data to obtain a global entropy value of the first multi-source sample perception data and a global entropy value of the second multi-source sample perception data; The absolute value of the difference between the global entropy value of the first multi-source sample perception data and the global entropy value of the second multi-source sample perception data is calculated to obtain the change value.

[0079] It should be noted that when conducting model training, the adverse weather conditions can be evaluated to obtain the severity of the adverse weather, that is, the change value between the global entropy value of the first multi-source sample perception data and the global entropy value of the second multi-source sample perception data, so that the system can better cope with various environmental conditions in the subsequent distillation process.

[0080] Optionally, for each multi-source sample perception data in the first multi-source sample perception data and the second multi-source sample perception data, data entropy may be introduced to evaluate the data quality so as to convert the measurement result into a unified form and capture more uncertainty information. The specific steps include: The multi-source sample perception data collected by the detection device is divided into k data blocks of size M×N, and the local entropy value of each data block is calculated using the following formula: k = k，I log( k，I ); in, k，I It represents the probability that the pixel value in the kth block is I, and the calculation formula is: k，I = ; in, Indicated in The pixel value at position, (·) represents the Kronecker function; I represents the pixel value of the pixel, T represents the number of pixel values; M and N represent the data block sizes respectively.

[0081] Next, the local entropy values ​​of all data blocks in the multi-source sample perception data are summed to calculate the global entropy of the multi-source sample perception data. The greater the absolute value change of the entropy value between the perception data under adverse weather conditions and the perception data under normal weather conditions, the greater the impact of the environment on the detection equipment.

[0082] By traversing the first multi-source sample perception data and the second multi-source sample perception data through the above steps, the global entropy value of the first multi-source sample perception data and the global entropy value of the second multi-source sample perception data can be obtained.

[0083] After obtaining the global entropy value of the first multi-source sample perception data and the global entropy value of the second multi-source sample perception data, the absolute value of the difference between the global entropy value of the first multi-source sample perception data and the global entropy value of the second multi-source sample perception data can be calculated to obtain the change value of the perception data under different environments. The specific calculation formula is: = | s - t |; in, is the change value; s is the global entropy value of the first multi-source sample perception data; t is the global entropy value of the second multi-source sample perception data.

[0084] The method provided in this embodiment, through precise calculation and utilization of the global entropy value changes of multi-source sample perception data under different weather conditions, enables the training process of the target detection model to more effectively adapt to environmental changes, and significantly improves the detection accuracy and robustness of the model in severe weather conditions.

[0085] In some embodiments, obtaining a first loss function according to a change value between a global entropy value of the first multi-source sample perception data and a global entropy value of the second multi-source sample perception data includes: According to the change value, the multimodal fusion features of each position point in the first multi-source sample perception data output by the student model, the multimodal fusion features of each position point in the second multi-source sample perception data output by the teacher model, the Gaussian distribution mask value of each position point, and the feature scale information corresponding to each position point, the difference between the foreground feature learning performance of the student model and the foreground feature learning performance of the teacher model is calculated to obtain the first loss function.

[0086] It should be noted that during the model training process, feature distillation can be performed on the multimodal fusion features corresponding to the multi-source sample perception data under different environments, and the first loss function can be calculated according to the data quality corresponding to the change in the global entropy value of the multi-source sample perception data under different weather conditions, so as to adaptively adjust the proportion of the student model learning from the teacher model. During the model training process, when the data quality is poor, the student model is guided to learn more from the teacher model to prevent the student model from making too many errors; and when the data quality is good, the student model is guided to rely more on its own detection results to prevent the student model from relying on the teacher model.

[0087] The calculation steps of the first loss function include: The foreground distillation method is used to encourage the student model to accurately learn the foreground features in the teacher model. In the multimodal fusion features of multi-source sample perception data, each actual center (x i ,y i ) draws a Gaussian distribution and sets all masks x i ,y i Merge into a single mask, and use the data indicators corresponding to the global entropy value changes of multi-source sample perception data under different weather conditions to adaptively change the proportion of student model learning from teacher model to obtain the first loss function L fea , which can be expressed by the following formula: Lfea = ; in, = | s - t | represents the change value between the global entropy value of the first multi-source sample perception data and the global entropy value of the second multi-source sample perception data, H, W represents the height and width of the characteristic scale information corresponding to each position point, Represents the scaling factor that controls the data indicator. and Represent the multimodal fusion features of the position point (i, j) output by the teacher network and the student network respectively, It is the Gaussian distribution mask value of the Gaussian distribution mask G at the position point (i, j) calculated based on the center position of the foreground target in the feature map.

[0088] The method provided in this embodiment combines the global entropy change and the foreground distillation of multimodal fusion features so that the target detection model training can adaptively optimize the learning process of the student model, and can adjust the relevant learning ratio according to the quality of the data to improve its detection accuracy and generalization ability under different data qualities.

[0089] In some embodiments, acquiring a third loss function according to the first target detection result and the second target detection result of the sample detection area includes: According to the distance between each detected target and the preset target, obtaining the distance weight corresponding to each detected target; According to the distance weight, weighted fusion is performed on the errors between the classification results of each of the detection targets in the first target detection result and the classification results of each of the detection targets in the second target detection result to obtain a classification loss function; According to the distance weight, weighted fusion is performed on the errors between the regression results of each of the detection targets in the first target detection result and the regression results of each of the detection targets in the second target detection result to obtain a regression loss function; The classification loss function and the regression loss function are fused and calculated to obtain the third loss function.

[0090] It should be noted that in order to address the difficulty of detecting small targets at a long distance in adverse weather conditions, a distance-dependent response distillation loss function, that is, the third loss function, can be designed to enable the student model to learn the detection strategy and response method of the teacher model at different distances.

[0091] Optionally, a nonlinear function is first used to flexibly adjust the weight difference between near and far objects to ensure high-precision detection of near objects in adverse weather. For distant objects, it is generally believed that when the weather is severely adverse, it is crucial to set the weight of distant objects to be smaller to ensure that they do not affect the overall accuracy of the model. Therefore, in this embodiment, a fixed smaller weight μ is assigned to them; wherein the specific definition of the distance weight corresponding to each detection target is as follows: ; in, is the distance weight corresponding to the i-th detected target; Indicates the distance between the i-th detected target and the preset target (such as a car), A, B, C, It is a hyperparameter in the distance weight, which can be set according to actual needs. For example, the set values ​​are -0.0005, 0.02, 0.6, 0.2 and 50 respectively.

[0092] After obtaining the distance weight, the errors between the classification results of each detection target in the first target detection result and the classification results of each detection target in the second target detection result can be weighted and fused according to the distance weight to obtain the classification loss function L cls , and based on the distance weight, the error between the regression result of each detection target in the first target detection result and the regression result of each detection target in the second target detection result is weighted fused to obtain the regression loss function L reg , the specific calculation formula is as follows: L cls = ( , )× ; L reg = × ; Among them, S is the number of detected targets (or objects), is the normalized distance weight of the i-th detected target, that is, the weight coefficient, QFL is the Quality Focal (positioning quality) loss function, is the smooth L1 loss function. The output of the teacher model is denoted by t, the output of the student model is denoted by s, the regression task is denoted by r, and the probability vector of the classification task is denoted by p.

[0093] Finally, the classification loss function and the regression loss function are weighted and fused to obtain the third loss function (that is, the final response loss function) L res : L res =L cls +Lreg ; The method provided in this embodiment introduces a distance-related response distillation loss function, so that the target detection model training can flexibly adjust the weights for detection targets at different distances, and can use the distance to limit the weights of distant small targets in adverse weather conditions and focus on nearby targets, thereby improving the detection accuracy and overall performance of small targets under adverse weather conditions.

[0094] In some embodiments, obtaining a fourth loss function according to noise data of each predicted bounding box in the first target detection result includes: Calculating a noise ratio according to the noise data of each predicted bounding box to obtain a noise ratio value of each predicted bounding box; Calculate the noise weight according to the noise ratio value to obtain the noise weight of each predicted bounding box; According to the noise weight, a weighted average calculation is performed on the confidence scores of each of the predicted bounding boxes in the first target detection result to obtain the fourth loss function.

[0095] It should be noted that during the model training process, the detection results output by the student model can also be used to determine the loss function according to the set noise ratio threshold to suppress noise in adverse weather environments (such as thick fog, water splashes, or exhaust smoke in cold weather). The target detection optimization method can be used to effectively reduce false alarms caused by bad weather, thereby filtering out detection frames that may contain too much noise to improve the reliability of detection.

[0096] Optionally, for each detected target, the noise ratio is calculated and the bounding box is adjusted accordingly, thereby effectively alleviating the false alarm of bad weather, wherein the noise ratio value Ri and the noise weight Ωi of the predicted bounding box of each detected target can be expressed as: R i = ; Ω i = 1-tan(R i ); in, is a small constant that prevents division by zero, is the noisy data, It is non-noise data.

[0097] After obtaining the noise weight of each predicted bounding box, the confidence score of each predicted bounding box in the first target detection result can be weighted averaged according to the noise weight to obtain the fourth loss function L nos , specifically defined as follows: L nos = Ω i ) i ; in, represents the set of predicted bounding boxes, i is the confidence score of the i-th predicted bounding box.

[0098] The method provided in this embodiment calculates the noise ratio of the predicted bounding box and adjusts the noise weight accordingly, so that the target detection model training can adaptively filter the noise according to the noise ratio to optimize the detection results, reduce false alarms caused by bad weather, and improve the reliability and accuracy of detection.

[0099] The following describes the training steps of the target detection model provided in this embodiment with specific examples.

[0100] Figure 2 It is a flowchart of the training steps of the target detection model provided by the present invention; Figure 2 As shown in the figure, the training steps of the target detection model specifically include: Step 1: Use two detection devices, camera and radar sensor, as the perception data source to obtain multi-source sample perception data, and use the camera's external parameters to convert the point cloud data from the vehicle coordinate system to the camera coordinate system. Then use the camera's internal parameters to project the point cloud data in the camera coordinate system onto the image plane, establish the corresponding relationship between the point cloud data and the image data, and lay the foundation for subsequent multimodal information fusion. Step 2: Optimize and adjust the point cloud image mapping relationship obtained in step 1. By mapping the radar sensor sampling time to the image sampling time, the error and noise caused by asynchrony are significantly reduced. This method not only enhances the geometric consistency of the data, but also improves the robustness and reliability of the system under severe weather conditions; Step 3, using the corresponding two-dimensional and three-dimensional feature extraction networks to extract features from the image data and the point cloud data after the alignment processing in step 1 and step 2 respectively; Step 4: The image features obtained in step 3 are transformed into a perspective and fused with the point cloud features to form a multimodal fusion feature, namely, a bird's-eye view feature. The bird's-eye view feature can provide a more comprehensive environmental perception and is more helpful for understanding complex scenes such as adverse weather. Step 5, evaluate the adverse weather conditions to obtain the data quality evaluation index of the adverse weather, that is, the change value between the global entropy value of the first multi-source sample perception data and the global entropy value of the second multi-source sample perception data. The data quality evaluation index of the adverse weather will be used in the subsequent distillation process to enable the system to better cope with various environmental conditions; Step 6: Input the multimodal fusion features into the scene distillation part, and evaluate the data distribution difference and overall impact of adverse weather by measuring the structural similarity and spatial similarity of the outputs of the student model and the teacher model, and obtain the second loss function; Step 7: Perform feature distillation on the bird's-eye view features. According to the data quality evaluation index of adverse weather in step 5, determine the first loss function to adaptively adjust the ratio of the student model to the teacher model. When the data quality is poor, guide the student model to learn more from the teacher model to prevent the student model from making too many errors. When the data quality is good, guide the student model to rely more on its own detection results to prevent the student model from relying on the teacher model. Step 8: To address the difficulty of detecting small targets at long distances in bad weather, we design a distance-dependent response distillation loss function, also known as the third loss function, so that the student model can learn the detection strategy and response method of the teacher model at different distances. In step nine, the detection result generated in step eight is subjected to a fourth loss function according to a set noise ratio threshold to filter out detection frames that may contain too much noise and improve the reliability of detection.

[0101] In summary, the method provided in the embodiment of the present application uses time domain modeling to achieve precise alignment of multimodal data, and determines multiple loss functions through data quality assessment, distance setting, noise suppression, etc., so as to transfer the knowledge of the teacher model to the student model through a staged adaptive knowledge distillation method, and constructs a comprehensive three-dimensional target detection process, which enhances the robustness and perception accuracy of the detection framework, improves the robustness in the face of complex scenes, reduces the interference caused by noise, and shows excellent performance and strong adaptability in complex scenes such as adverse weather, and has achieved good performance.

[0102] The target detection method provided in this embodiment is simulated and verified by using a specific example below.

[0103] Figure 3 is a schematic diagram of the data alignment effect provided by the present invention; Figure 3 As shown, Figure 3 (a) and Figure 3 (b) is the data distribution diagram before alignment of different point cloud data. Figure 3 (c) and Figure 3 (d) is the data distribution diagram after different point cloud data are aligned; Figure 3 Comparative analysis shows that the point cloud data after data alignment is more conducive to improving the consistency between modalities.

[0104] Figure 4 is a schematic diagram of simulation comparison results of different target detection methods provided by the present invention; Figure 4As shown, by comparing the method provided in the embodiment of the present application with other traditional three-dimensional target detection methods, such as the fast encoder (PointPillars) for point cloud target detection, the focal sparse convolutional network (FC for short), the sparse embedded convolutional detection (SECOND for short), the point voxel region convolutional neural network (PV-RCNN for short), and the point cloud-based regional convolutional neural network three-dimensional target detection (PointRCNN for short), it can be seen that whether it is a sunny day or other complex weather, the method provided in the embodiment of the present application can obtain target detection results that are better than other traditional three-dimensional target detection methods.

[0105] Through the above experiments, the effectiveness of the method provided in the embodiment of the present application is effectively demonstrated.

[0106] The target detection device based on multi-stage knowledge distillation provided by the present invention is described below. The target detection device based on multi-stage knowledge distillation described below and the target detection method based on multi-stage knowledge distillation described above can be referenced to each other.

[0107] Figure 5 is a schematic diagram of the structure of the target detection device based on multi-stage knowledge distillation provided by the present invention; Figure 5 As shown, the device comprises: The data acquisition unit 510 is used to acquire multi-source perception data of the area to be detected collected by multiple different target detection devices in the first environment; The first data processing unit 520 is used to input the multi-source perception data into a feature extraction module of a target detection model to obtain a multi-source feature map; The second data processing unit 530 is used to input the multi-source feature map into the feature fusion module of the target detection model to obtain a multi-modal fusion feature; The target detection unit 540 is used to input the multimodal fusion feature into the detection module of the target detection model to obtain the target detection result of the area to be detected; Among them, the target detection model is obtained by multi-stage knowledge distillation training based on the first multi-source sample perception data, the second multi-source sample perception data and the target detection label of the sample detection area; the first multi-source sample perception data is obtained by multiple different sample detection devices collecting data on the sample detection area under the first environment, and the second multi-source sample perception data is obtained by multiple different sample detection devices collecting data on the sample detection area under the second environment; the environmental quality of the second environment is higher than the environmental quality of the first environment.

[0108] The device provided in this embodiment integrates multiple sample detection devices to detect multi-source sample perception data of the same sample detection area under different environments, and uses the multi-source sample perception data of the same sample detection area under different environments to perform multi-stage knowledge distillation, so as to improve data quality through multi-source perception data integration, and through multi-stage knowledge distillation training, the target detection model trained in this way can show a detection effect equivalent to or even better than that in normal weather conditions under severe weather conditions. Therefore, in actual applications, the target detection model based on this training is used to perform feature extraction, feature fusion and target detection on multi-source perception data collected by multiple different target detection devices under severe weather conditions, so that the target detection results obtained by detection can have good adaptability and robustness in severe weather environments, so as to effectively overcome the adverse effects of severe weather on target detection accuracy, and thus achieve effective acquisition of stable and high-precision target detection results in complex environments.

[0109] The device provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0110] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor (processor) 610 , a communication interface (Communications Interface) 620 , a memory (memory) 630 and a communication bus 640 , wherein the processor 610 , the communication interface 620 , and the memory 630 communicate with each other through the communication bus 640 . The processor 610 can call the logic instructions in the memory 630 to execute the target detection method based on multi-stage knowledge distillation, which includes: obtaining multi-source perception data of the area to be detected collected by multiple different target detection devices in a first environment; inputting the multi-source perception data into the feature extraction module of the target detection model to obtain a multi-source feature map; inputting the multi-source feature map into the feature fusion module of the target detection model to obtain a multimodal fusion feature; inputting the multimodal fusion feature into the detection module of the target detection model to obtain the target detection result of the area to be detected; wherein the target detection model is obtained by multi-stage knowledge distillation training based on the first multi-source sample perception data, the second multi-source sample perception data and the target detection label of the sample detection area; the first multi-source sample perception data is obtained by multiple different sample detection devices collecting data on the sample detection area in the first environment, and the second multi-source sample perception data is obtained by multiple different sample detection devices collecting data on the sample detection area in the second environment; the environmental quality of the second environment is higher than the environmental quality of the first environment.

[0111] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0112] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the target detection method based on multi-stage knowledge distillation provided by the above methods, the method comprising: obtaining multi-source perception data of a to-be-detected area collected by multiple different target detection devices in a first environment; inputting the multi-source perception data into a feature extraction module of a target detection model to obtain a multi-source feature map; inputting the multi-source feature map into a feature fusion module of the target detection model to obtain a multimodal fusion feature; inputting the multimodal fusion feature into a detection module of the target detection model to obtain a target detection result of the to-be-detected area; wherein the target detection model is obtained by multi-stage knowledge distillation training based on the first multi-source sample perception data, the second multi-source sample perception data and the target detection label of the sample detection area; the first multi-source sample perception data is obtained by multiple different sample detection devices collecting data on the sample detection area in the first environment, and the second multi-source sample perception data is obtained by multiple different sample detection devices collecting data on the sample detection area in the second environment; the environmental quality of the second environment is higher than the environmental quality of the first environment.

[0113] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the target detection method based on multi-stage knowledge distillation provided by the above methods, the method comprising: obtaining multi-source perception data of the area to be detected collected by multiple different target detection devices in a first environment; inputting the multi-source perception data into a feature extraction module of a target detection model to obtain a multi-source feature map; inputting the multi-source feature map into a feature fusion module of the target detection model to obtain a multimodal fusion feature; inputting the multimodal fusion feature into a detection module of the target detection model to obtain a target detection result of the area to be detected; wherein the target detection model is obtained by multi-stage knowledge distillation training based on the first multi-source sample perception data, the second multi-source sample perception data and the target detection label of the sample detection area; the first multi-source sample perception data is obtained by multiple different sample detection devices collecting data on the sample detection area in the first environment, and the second multi-source sample perception data is obtained by multiple different sample detection devices collecting data on the sample detection area in the second environment; the environmental quality of the second environment is higher than the environmental quality of the first environment.

[0114] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0115] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target detection method based on multi-stage knowledge distillation, characterized in that: include: Acquire multi-source perception data of the area to be detected collected by multiple different target detection devices in the first environment; Inputting the multi-source perception data into a feature extraction module of a target detection model to obtain a multi-source feature map; Inputting the multi-source feature map into the feature fusion module of the target detection model to obtain a multimodal fusion feature; Inputting the multimodal fusion features into the detection module of the target detection model to obtain the target detection result of the area to be detected; The target detection model is obtained by multi-stage knowledge distillation training based on the first multi-source sample perception data, the second multi-source sample perception data and the target detection label of the sample detection area; The first multi-source sample perception data is obtained by multiple different sample detection devices collecting data on the sample detection area under the first environment, and the second multi-source sample perception data is obtained by multiple different sample detection devices collecting data on the sample detection area under the second environment; the environmental quality of the second environment is higher than the environmental quality of the first environment.

2. The target detection method based on multi-stage knowledge distillation according to claim 1, characterized in that: The target detection model is trained based on the following steps: Training a teacher model according to the second multi-source sample perception data and the target detection label, and constructing a student model corresponding to the teacher model; Obtaining a first loss function according to a change value between a global entropy value of the first multi-source sample perception data and a global entropy value of the second multi-source sample perception data; Obtaining a second loss function according to the self-affinity matrix of the first multi-source sample perception data and the self-affinity matrix of the second multi-source sample perception data; the self-affinity matrix of the first multi-source sample perception data is calculated by the student model, and the self-affinity matrix of the second multi-source sample perception data is calculated by the teacher model; Obtaining a third loss function according to a first target detection result and a second target detection result of the sample detection area; the first target detection result is obtained by the student model performing target detection according to the first multi-source sample perception data, and the second target detection result is obtained by the teacher model performing target detection according to the second multi-source sample perception data; Obtaining a fourth loss function according to noise data of each predicted bounding box in the first target detection result; Performing multi-stage knowledge distillation training on the student model according to the first loss function, the second loss function, the third loss function and the fourth loss function; The target detection model is obtained according to the student model after multiple iterations of training.

3. The target detection method based on multi-stage knowledge distillation according to claim 2 is characterized in that: The step of obtaining a change value between the global entropy value of the first multi-source sample perception data and the global entropy value of the second multi-source sample perception data comprises: For each multi-source sample perception data in the first multi-source sample perception data and the second multi-source sample perception data, dividing the multi-source sample perception data into a plurality of data blocks of the same size; Calculating the probability of occurrence of each pixel value in each of the data blocks; Calculating a local entropy value of each of the data blocks according to the probability; Summing the local entropy values ​​of all data blocks in the multi-source sample perception data to obtain a global entropy value of the multi-source sample perception data; Traversing the first multi-source sample perception data and the second multi-source sample perception data to obtain a global entropy value of the first multi-source sample perception data and a global entropy value of the second multi-source sample perception data; The absolute value of the difference between the global entropy value of the first multi-source sample perception data and the global entropy value of the second multi-source sample perception data is calculated to obtain the change value.

4. The target detection method based on multi-stage knowledge distillation according to claim 2, characterized in that: The obtaining a first loss function according to a change value between a global entropy value of the first multi-source sample perception data and a global entropy value of the second multi-source sample perception data includes: According to the change value, the multimodal fusion features of each position point in the first multi-source sample perception data output by the student model, the multimodal fusion features of each position point in the second multi-source sample perception data output by the teacher model, the Gaussian distribution mask value of each position point, and the feature scale information corresponding to each position point, the difference between the foreground feature learning performance of the student model and the foreground feature learning performance of the teacher model is calculated to obtain the first loss function.

5. The target detection method based on multi-stage knowledge distillation according to claim 2, characterized in that: The acquiring a third loss function according to the first target detection result and the second target detection result of the sample detection area includes: According to the distance between each detected target and the preset target, obtaining the distance weight corresponding to each detected target; According to the distance weight, weighted fusion is performed on the errors between the classification results of each of the detection targets in the first target detection result and the classification results of each of the detection targets in the second target detection result to obtain a classification loss function; According to the distance weight, weighted fusion is performed on the errors between the regression results of each of the detection targets in the first target detection result and the regression results of each of the detection targets in the second target detection result to obtain a regression loss function; The classification loss function and the regression loss function are fused and calculated to obtain the third loss function.

6. The target detection method based on multi-stage knowledge distillation according to claim 2, characterized in that: The obtaining, according to the noise data of each predicted bounding box in the first target detection result, a fourth loss function comprises: Calculating a noise ratio according to the noise data of each predicted bounding box to obtain a noise ratio value of each predicted bounding box; Calculate the noise weight according to the noise ratio value to obtain the noise weight of each predicted bounding box; According to the noise weight, a weighted average calculation is performed on the confidence scores of each of the predicted bounding boxes in the first target detection result to obtain the fourth loss function.

7. The target detection method based on multi-stage knowledge distillation according to any one of claims 1 to 6, characterized in that: The feature extraction module includes a data alignment unit and a plurality of feature extraction units corresponding to different data types; The step of inputting the multi-source perception data into a feature extraction module of a target detection model to obtain a multi-source feature map includes: The multi-source perception data is input into the data alignment unit to obtain multi-source mapping data; the multi-source mapping data is obtained by the data alignment unit mapping the multi-source perception data to the data space and time space corresponding to any target detection device according to the external parameter matrix and the internal parameter matrix of any target detection device; Inputting each of the mapping data into a feature extraction unit corresponding to the data type of each of the perception data to obtain a feature map of each of the perception data; The feature maps of the perception data from multiple sources are integrated to obtain the multi-source feature map.

8. An object detection device based on multi-stage knowledge distillation, characterized in that: include: A data acquisition unit, used to acquire multi-source perception data of the area to be detected collected by multiple different target detection devices in the first environment; A first data processing unit, configured to input the multi-source perception data into a feature extraction module of a target detection model to obtain a multi-source feature map; A second data processing unit is used to input the multi-source feature map into the feature fusion module of the target detection model to obtain a multimodal fusion feature; A target detection unit, used for inputting the multimodal fusion feature into the detection module of the target detection model to obtain the target detection result of the area to be detected; The target detection model is obtained by multi-stage knowledge distillation training based on the first multi-source sample perception data, the second multi-source sample perception data and the target detection label of the sample detection area; The first multi-source sample perception data is obtained by multiple different sample detection devices collecting data on the sample detection area under the first environment, and the second multi-source sample perception data is obtained by multiple different sample detection devices collecting data on the sample detection area under the second environment; the environmental quality of the second environment is higher than the environmental quality of the first environment.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the target detection method based on multi-stage knowledge distillation as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the target detection method based on multi-stage knowledge distillation as described in any one of claims 1 to 7 is implemented.

Images