A deep learning-based real-time misclassification detection method, system, device and storage medium

Abstract

The application discloses a kind of real-time misclassification detection method, system, equipment and storage medium based on deep learning, including obtaining the video stream of the package to be detected on belt, slice and obtain the original image sequence to be detected;Lightweight target detection model is constructed, and then the package detection frame is labeled by labeling software, and the package training dataset is made, the lightweight target detection model is trained using the package training dataset, and the trained lightweight target detection model is obtained;Lightweight target detection model includes backbone network module, feature fusion module and prediction head, and the position information of optimal detection frame and the corresponding target class probability are output;The original image sequence to be detected is input into the trained lightweight target detection model, and the detection frame set is output;Then by introducing target tracking algorithm, the time sequence consistency correction is carried out on detection frame set using interframe motion correlation, and the package trajectory is obtained.The application effectively reduces processing time consumption and improves the precision of misclassification detection.

Description

Technical Field

[0001] This invention relates to the field of logistics parcel sorting technology, and in particular to a real-time misclassification detection method, system, device and storage medium based on deep learning. Background Technology

[0002] With the explosive growth of business volume in the express delivery and logistics industry, the accuracy and efficiency of parcel sorting have become key factors for industry development. Although automated sorting equipment has greatly improved sorting efficiency, missorting is still difficult to completely avoid due to the complexity of the logistics environment and the diversity of goods. The consequences of missorting parcels are not limited to fines, but also include lost packages, delays, and other problems, seriously affecting customer stability and service experience. Therefore, reducing missorting and improving sorting accuracy is particularly important.

[0003] Currently, the main method of automated sorting systems is to use a vision-based missorting detection system. This system includes target detection algorithms and target tracking algorithms. The target detection algorithm uses a mainstream non-lightweight detection model, which is time-consuming to deploy on smart machines, reduces real-time performance, and decreases belt conveyor speed and sorting efficiency. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a real-time misclassification detection method, system, device, and storage medium based on deep learning. This invention effectively reduces processing time and improves misclassification detection accuracy.

[0005] Technical Solution: To achieve the above objectives, the present invention provides a real-time misclassification detection method based on deep learning, comprising the following steps: (1) The video stream of the package to be inspected on the belt is obtained by using an industrial camera, and the original image sequence to be inspected is obtained by slicing it. (2) Construct a lightweight object detection model, then use annotation software to annotate the package detection boxes, create a package training dataset, use the package training dataset to train the lightweight object detection model, and obtain the trained lightweight object detection model; The lightweight object detection model includes a backbone network module, a feature fusion module, and a prediction head. The backbone network module performs initial convolution and pooling downsampling operations on the input raw image to generate an initial feature map. After progressive downsampling and feature extraction, the initial feature map outputs a multi-scale feature map with progressively decreasing resolution and progressively increasing channel number. The feature fusion module performs a uniform channel number transformation on the multi-scale feature maps output by the backbone network module, and then aggregates deep semantic features and shallow spatial features progressively through a bidirectional cross-level feature fusion path from top to bottom and bottom to top, outputting fused feature maps of various scales. The prediction head predicts the location information of the detection box and the probability of the target class based on the fused feature maps of different scales. Finally, a non-maximum suppression algorithm is used to select the optimal detection box, and the location information of the optimal detection box and the corresponding target class probability are output. The location information of the detection box includes the coordinates of the top-left corner of the detection box, the width of the detection box, and the height of the detection box. (3) Input the original image sequence to be detected into the trained lightweight target detection model and output the detection box set; then, by introducing the target tracking algorithm, use the inter-frame motion correlation to perform temporal consistency correction on the detection box set to obtain the package trajectory.

[0006] Optionally, the backbone network module includes a convolutional block, a pooling layer, and three cascaded ES modules connected in sequence. The original image is input into the backbone network module and undergoes initial convolution and pooling processing to obtain an initial feature map. The initial feature map is then passed through the three cascaded ES modules for progressive downsampling and feature extraction, outputting a first feature layer, a second feature layer, and a third feature layer, respectively. The ES module includes a depthwise separable convolutional layer and an SE attention mechanism module.

[0007] Optionally, the feature fusion module includes a convolutional layer and a CSP module corresponding to the three cascaded ES modules. The first feature layer, the second feature layer, and the third feature layer are respectively input to the convolutional layer. The number of channels of the output feature map of the convolutional layer is adjusted to a preset dimension, and then the feature is fused through the CSP module to obtain four fused feature maps of different scales. The third feature layer undergoes pointwise convolution processing through convolutional layers to adjust its channel count to a preset dimension, outputting a first intermediate feature map. The second feature layer undergoes pointwise convolution processing through convolutional layers to adjust its channel count to a preset dimension, outputting a second intermediate feature map. The second intermediate feature map is concatenated with the upsampled first intermediate feature map and then input into the CSP module for feature fusion to obtain a first fused feature map. The first feature layer undergoes pointwise convolution processing through convolutional layers to adjust its channel count to a preset dimension, outputting a third intermediate feature map. The third intermediate feature map is concatenated with the upsampled first fused feature map and then input into the CSP module for feature fusion to obtain a second fused feature map. The downsampled second fused feature map is concatenated with the first fused feature map and then input into the CSP module for feature fusion to obtain a third fused feature map. The downsampled third fused feature map is concatenated with the first intermediate feature map and then input into the CSP module for feature fusion to obtain a fourth fused feature map. The downsampled fourth fused feature map is added to the downsampled first intermediate feature map to obtain a fifth fused feature map. The second fused feature map is input into the first prediction head, the third fused feature map is input into the second prediction head, the fourth fused feature map is input into the third prediction head, and the fifth fused feature map is input into the fourth prediction head.

[0008] Optionally, in step (2), the non-maximum suppression algorithm is used to select the optimal detection box. Specifically, the detection boxes output by the detection head are arranged in descending order according to the target confidence score. The detection box with the highest confidence score is selected as the retained detection box. The intersection-union ratio of the remaining detection boxes and the retained detection boxes is calculated. Redundant detection boxes with an intersection-union ratio greater than a preset threshold are removed. The process is iterated until all detection boxes are processed. The location information of the optimal detection box corresponding to each target package and the target category probability are output.

[0009] Optionally, the target tracking algorithm in step (3) includes a Kalman filter and a Hungarian algorithm. The specific steps of the target tracking algorithm are as follows: (3.1) Maintain a trajectory pool. The trajectory pool is empty in the initial frame. The trajectory pool stores the trackers that have been established before the current frame. Each tracker encapsulates the corresponding package tracking trajectory and Kalman filter state. The Kalman filter state includes a state vector and its corresponding covariance matrix. The state vector contains the position parameters and position change rate parameters of the target detection box. The covariance matrix is ​​used to characterize the uncertainty of each state parameter and their correlation. Obtain the package tracking trajectory and Kalman filter state output and stored in the previous frame from the trajectory pool as the input of the current frame. For each package tracking trajectory in the current frame, use the Kalman filter to calculate the predicted state vector and predicted covariance matrix of the current frame based on the state vector and covariance matrix of the previous frame, combined with the state transition matrix and the process excitation noise covariance matrix, to obtain the set of tracking boxes of each package in the current frame, i.e., the set of predicted boxes. (3.2) Input the current frame image into the lightweight object detection model, output the detection results of all packages in the current frame, and form a set of detection boxes; (3.3) For the set of tracking boxes obtained in step (3.1) and the set of detection boxes obtained in step (3.2), each tracking box and each detection box is expanded outward along the x and y directions at the same ratio with their respective center points as the center, so as to increase the area of ​​each box and increase the probability that there is an intersection between the detection box and the tracking box. (3.4) Calculate the generalized intersection-union ratio (gIoU) between each detection box and each tracking box after expansion in step (3.3), and construct the cost matrix using the generalized intersection-union ratio (gIoU) as the similarity index; (3.5) Input the cost matrix constructed in step (3.4) into the Hungarian algorithm to solve the minimum cost matching between the detection box and the tracking box, and determine the correspondence between each detection box and each tracking box; (3.6) Based on the matching results of the Hungarian algorithm, all packages are divided into the following three categories: No matching tracking box: The tracking box failed to match any detection box; No matching detection box: The detection box failed to match any tracking box; Match successful: The detection box and the tracking box were successfully matched; (3.7) For a successfully matched package, the detection box matched in the current frame is used as the observation value and input into the Kalman filter to perform the update phase, that is, the prediction state vector and prediction covariance matrix obtained in step (3.1) are corrected using the observation value of the current frame to obtain the output tracking box of the current frame; at the same time, the tracking frame number of the package tracker is incremented by 1, the skipped frame number is reset to 0, and the ID of the package remains unchanged. (3.8) For unmatched detection boxes, determine that they are newly entered packages in the current frame and perform the following operations: use the detection box as the initial observation value, create a new tracker and assign it a unique identification number, and increment the total number of IDs by 1; at the same time, initialize the corresponding Kalman filter with the detection box, and use the detection box as the output tracking box of the current frame, as well as the first frame position of the tracker's starting tracking trajectory; add the newly created tracker to the trajectory pool; (3.9) For unmatched tracking boxes, increment the skipped frame count of the package tracker by 1, and determine whether the current skipped frame count has reached the preset skipped frame count threshold. If the threshold has not been reached, temporarily retain the tracker, and use the Kalman filter prediction result of the tracker, i.e. the tracking box output in step (3.1), as the output tracking box of the current frame. The ID of the package tracker remains unchanged. If the threshold has been reached, determine that the package has left the scene or has been permanently lost, and delete the tracker and its trajectory. (3.10) Output the current frame tracking box obtained from steps (3.7) to (3.9) and use it as the input for the next frame. Repeat steps (3.1) to (3.10) to achieve tracking of all wrapped targets in the original image sequence.

[0010] The present invention discloses a real-time misclassification detection system based on deep learning, comprising: The image acquisition module is used to acquire video streams of packages to be inspected on a belt using an industrial camera, and then slice them to obtain the original image sequence to be inspected. The detection model building module is used to build a lightweight object detection model. Then, the package detection boxes are labeled with annotation software to create a package training dataset. The lightweight object detection model is trained using the package training dataset to obtain the trained lightweight object detection model. The lightweight object detection model includes a backbone network module, a feature fusion module, and a prediction head. The backbone network module performs initial convolution and pooling downsampling operations on the input raw image to generate an initial feature map. After progressive downsampling and feature extraction, the initial feature map outputs a multi-scale feature map with progressively decreasing resolution and progressively increasing channel number. The feature fusion module performs a uniform channel number transformation on the multi-scale feature maps output by the backbone network module, and then aggregates deep semantic features and shallow spatial features progressively through a bidirectional cross-level feature fusion path from top to bottom and bottom to top, outputting fused feature maps of various scales. The prediction head predicts the location information of the detection box and the probability of the target class based on the fused feature maps of different scales. Finally, a non-maximum suppression algorithm is used to select the optimal detection box, and the location information of the optimal detection box and the corresponding target class probability are output. The location information of the detection box includes the coordinates of the top-left corner of the detection box, the width of the detection box, and the height of the detection box. The package trajectory tracking module is used to input the original image sequence to be detected into the trained lightweight target detection model and output a set of detection boxes. Then, by introducing a target tracking algorithm, the temporal consistency of the detection box set is corrected by using inter-frame motion correlation to obtain the package trajectory.

[0011] Optionally, the backbone network module includes a convolutional block, a pooling layer, and three cascaded ES modules connected in sequence. The original image is input into the backbone network module and undergoes initial convolution and pooling processing to obtain an initial feature map. The initial feature map is then passed through the three cascaded ES modules for progressive downsampling and feature extraction, outputting a first feature layer, a second feature layer, and a third feature layer, respectively. The ES module includes a depthwise separable convolutional layer and an SE attention mechanism module.

[0012] Optionally, the feature fusion module includes a convolutional layer and a CSP module corresponding to the three cascaded ES modules. The first feature layer, the second feature layer, and the third feature layer are respectively input to the convolutional layer. The number of channels of the output feature map of the convolutional layer is adjusted to a preset dimension, and then the feature is fused through the CSP module to obtain four fused feature maps of different scales. The third feature layer undergoes pointwise convolution processing through convolutional layers to adjust its channel count to a preset dimension, outputting a first intermediate feature map. The second feature layer undergoes pointwise convolution processing through convolutional layers to adjust its channel count to a preset dimension, outputting a second intermediate feature map. The second intermediate feature map is concatenated with the upsampled first intermediate feature map and then input into the CSP module for feature fusion to obtain a first fused feature map. The first feature layer undergoes pointwise convolution processing through convolutional layers to adjust its channel count to a preset dimension, outputting a third intermediate feature map. The third intermediate feature map is concatenated with the upsampled first fused feature map and then input into the CSP module for feature fusion to obtain a second fused feature map. The downsampled second fused feature map is concatenated with the first fused feature map and then input into the CSP module for feature fusion to obtain a third fused feature map. The downsampled third fused feature map is concatenated with the first intermediate feature map and then input into the CSP module for feature fusion to obtain a fourth fused feature map. The downsampled fourth fused feature map is added to the downsampled first intermediate feature map to obtain a fifth fused feature map. The second fused feature map is input into the first prediction head, the third fused feature map is input into the second prediction head, the fourth fused feature map is input into the third prediction head, and the fifth fused feature map is input into the fourth prediction head.

[0013] The present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement a real-time misclassification detection method based on deep learning as described above.

[0014] The present invention discloses a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement a real-time misclassification detection method based on deep learning as described above.

[0015] Beneficial Effects: Compared with existing technologies, the present invention has the following significant advantages: The lightweight target detection model of the present invention adopts a lightweight backbone network containing depthwise separable convolutions and bidirectional cross-level feature fusion, which significantly reduces the computational load and parameter count of the model while ensuring multi-scale target detection accuracy. It achieves real-time operation at a rate of 20 frames per second on edge computing devices such as H700 color cameras, significantly improving the real-time performance and deployment feasibility of target detection, and is suitable for industrial scenarios with strict requirements on timeliness and computing power. The present invention optimizes the similarity index and matching mechanism of the target tracking algorithm, reduces the frequency of package switching IDs, and effectively improves the accuracy of misclassification detection. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the process of the present invention; Figure 2 This is a flowchart illustrating the lightweight target detection model in this invention. Figure 3 This is a flowchart illustrating the target tracking algorithm in this invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0018] like Figure 1 As shown, a real-time misclassification detection method based on deep learning in this invention includes the following steps: (1) An industrial camera is used to acquire the video stream of the package to be inspected on the belt. After slicing, the original image sequence to be inspected is obtained. The original image is an RGB color image, and the data dimension of the original image is represented as 256×256×3.

[0019] (2) Construct a lightweight object detection model, then use annotation software to annotate the package detection boxes, create a package training dataset, use the package training dataset to train the lightweight object detection model, and obtain the trained lightweight object detection model.

[0020] The lightweight object detection model comprises a backbone network module, a feature fusion module, and a prediction head. The backbone network module consists of sequentially connected convolutional blocks, pooling layers, and three cascaded ES modules. The original image is input into the backbone network module, where initial convolution and pooling are performed to obtain an initial feature map. The initial feature map is then passed through the three cascaded ES modules for progressive downsampling and feature extraction, outputting a first feature layer C3, a second feature layer C4, and a third feature layer C5, respectively. The data dimensions of the first feature layer C3 are 8×8×96, the second feature layer C4 are 16×16×192, and the third feature layer C5 are 32×32×384. The ES module includes depthwise separable convolutional layers and an SE attention mechanism module.

[0021] The feature fusion module includes convolutional layers and CSP modules corresponding to the three cascaded ES modules. The first feature layer C3, the second feature layer C4, and the third feature layer C5 are respectively input to the convolutional layer. The number of channels of the output feature map of the convolutional layer is adjusted to the preset dimension, and then the feature is fused through the CSP module to obtain four fused feature maps of different scales.

[0022] The third feature layer C5 undergoes pointwise convolution processing through a 1×1 convolutional layer, adjusting its channel number to a preset dimension of 1, and outputs the first intermediate feature map; the second feature layer C4 undergoes pointwise convolution processing through a 1×1 convolutional layer, adjusting its channel number to a preset dimension, and outputs the second intermediate feature map; the second intermediate feature map is concatenated with the upsampled first intermediate feature map and then input into the CSP module for feature fusion to obtain the first fused feature map.

[0023] The first feature layer C3 undergoes pointwise convolution processing through a 1×1 convolutional layer, adjusting its channel count to a preset dimension of 1, and outputs a third intermediate feature map. The third intermediate feature map is then concatenated with the upsampled first fused feature map and input into the CSP module for feature fusion to obtain a second fused feature map.

[0024] The second fusion feature map, after being downsampled, is concatenated with the first fusion feature map and then input into the CSP module for feature fusion to obtain the third fusion feature map.

[0025] The third fused feature map, after being downsampled, is concatenated with the first intermediate feature map and then input into the CSP module for feature fusion to obtain the fourth fused feature map.

[0026] The fifth fusion feature map is obtained by adding the downsampled fourth fusion feature map to the downsampled first intermediate feature map.

[0027] The second fused feature map is input to the first predictive head, the third fused feature map is input to the second predictive head, the fourth fused feature map is input to the third predictive head, and the fifth fused feature map is input to the fourth predictive head. The first, second, third, and fourth predictive heads independently output the detection box position information and the target category probability. The first, second, third, and fourth predictive heads are all composed of two depthwise separable convolutions, which are used for result classification and localization.

[0028] Finally, the non-maximum suppression algorithm is used to select the optimal detection box, and the location information of the optimal detection box and the corresponding target class probability are output. The location information of the detection box includes the coordinates of the top left corner of the detection box, the width of the detection box, and the height of the detection box.

[0029] The optimal detection box is obtained by using the non-maximum suppression algorithm. Specifically, the detection boxes output by the detection head are sorted in descending order of target confidence score. The detection box with the highest confidence score is selected as the retained detection box. The intersection-union ratio (IUR) of the remaining detection boxes and the retained detection boxes is calculated. Redundant detection boxes with IUR greater than a preset threshold are removed. The process is iterated until all detection boxes have been processed. The location information of the optimal detection box corresponding to each target package and the target category probability are output.

[0030] In this invention, the backbone network module combines the depthwise separable convolution and SE attention mechanism module from MobileNetV3 with ShuffleNetV2, which reduces the amount of computation during feature extraction. The SE attention mechanism module assigns higher weights to key channel features, thereby improving the accuracy of feature extraction of the model.

[0031] (3) Input the original image to be detected into the trained lightweight target detection model and output the detection box set; then, by introducing the target tracking algorithm, use the inter-frame motion correlation to perform temporal consistency correction on the detection box set.

[0032] Target tracking algorithms include the Kalman filter and the Hungarian algorithm. The specific steps of a target tracking algorithm are as follows: (3.1) Kalman filter prediction A trajectory pool is maintained, initially empty, containing trackers established before the current frame. Each tracker encapsulates a corresponding package tracking trajectory and Kalman filter state. The Kalman filter state includes a state vector and its corresponding covariance matrix. The state vector contains the target detection box's position parameters and rate of change parameters. The covariance matrix characterizes the uncertainty of each state parameter and their correlation. The package tracking trajectories and Kalman filter states from the previous frame are retrieved from the trajectory pool and used as input for the current frame. For each package tracking trajectory in the current frame, the Kalman filter is used to calculate the predicted state vector and predicted covariance matrix of the current frame based on the previous frame's state vector and covariance matrix, combined with the state transition matrix and process excitation noise covariance matrix, thus obtaining the set of tracking boxes for each package in the current frame, i.e., the predicted box set.

[0033] (3.2) Detection box acquisition The current frame image is input into the lightweight object detection model, which outputs the detection results of all packages in the current frame, forming a set of detection boxes.

[0034] (3.3) Spatial expansion of detection box and prediction box For the set of tracking boxes obtained in step (3.1) and the set of detection boxes obtained in step (3.2), each tracking box and each detection box is expanded outward along the x and y directions at the same ratio, with their respective center points as the center, in order to increase the area of ​​each box and increase the probability that there is an intersection between the detection box and the tracking box.

[0035] (3.4) Construct the cost matrix based on the generalized intersection-union ratio gIoU Calculate the generalized intersection-union ratio (gIoU) between each detection box and each tracking box after the expansion in step (3.3), and use the generalized intersection-union ratio (gIoU) as the similarity index to construct the cost matrix.

[0036] (3.5) Optimal matching of Hungarian algorithm Input the cost matrix constructed in step (3.4) into the Hungarian algorithm to solve for the minimum cost matching between the detection box and the tracking box, and determine the correspondence between each detection box and each tracking box.

[0037] (3.6) Classification and processing of matching results Based on the matching results of the Hungarian algorithm, all packages are divided into the following three categories: No matching tracking box: The tracking box failed to match any detection box; No matching detection box: The detection box failed to match any tracking box; Match successful: The detection box and the tracking box were successfully matched.

[0038] (3.7) Successful matching branch processing For a successfully matched package, the matched detection box in the current frame is used as the observation value and input into the Kalman filter to perform the update phase. That is, the predicted state vector and predicted covariance matrix obtained in step (3.1) are corrected using the observation value of the current frame to obtain the output tracking box of the current frame. At the same time, the tracking frame number of the package tracker is incremented by 1, the skipped frame number is reset to 0, and the ID of the package remains unchanged.

[0039] (3.8) Handling of branches with unmatched detection boxes For unmatched detection boxes, determine that they are newly entered packages in the current frame and perform the following operations: use the detection box as the initial observation value, create a new tracker and assign it a unique identification number, incrementing the total number of IDs by 1; at the same time, initialize the corresponding Kalman filter with the detection box, and use the detection box as the output tracking box of the current frame, as well as the first frame position of the tracker's starting tracking trajectory; add the newly created tracker to the trajectory pool.

[0040] (3.9) Handling branches of mismatched tracking boxes For unmatched tracking boxes, increment the skipped frame count of the package tracker by 1, and determine whether the current skipped frame count has reached the preset skipped frame count threshold. If the threshold has not been reached, temporarily retain the tracker, and use the Kalman filter prediction result of the tracker, i.e. the tracking box output in step (3.1), as the output tracking box of the current frame. The ID of the package tracker remains unchanged. If the threshold has been reached, determine that the package has left the scene or has been permanently lost, and delete the tracker and its trajectory.

[0041] (3.10): Execute in a loop The current frame tracking box obtained from steps (3.7), (3.8) and (3.9) is output and used as the input for the next frame. Steps (3.1) to (3.10) are repeated to achieve continuous and stable tracking of all wrapped targets in the original image sequence.

[0042] In the lightweight target detection model of this invention, the depthwise separable convolution module significantly reduces computational cost during feature extraction, and the SE attention mechanism module assigns higher weights to key channel features, achieving both fast detection speed and accurate detection of packages on belts. Kalman filters and the Hungarian algorithm are introduced for target tracking. Packages are tracked by extracting images from the video stream frame by frame, ensuring the continuity of the same package across different frames. The frequency of package ID switching is reduced by optimizing the matching mechanism and the similarity index between the detection box and the prediction box, thereby improving misclassification detection accuracy.

[0043] Example 2: A real-time misclassification detection system based on deep learning according to the present invention includes: The image acquisition module is used to acquire the video stream of the package to be detected on the belt using an industrial camera, and then slice it to obtain the original image sequence to be detected; the image acquisition module specifically executes the steps (1) of a real-time misclassification detection method based on deep learning.

[0044] The detection model building module is used to build a lightweight object detection model. Then, the package detection boxes are labeled using annotation software to create a package training dataset. The lightweight object detection model is trained using the package training dataset to obtain the trained lightweight object detection model.

[0045] The lightweight object detection model includes a backbone network module, a feature fusion module, and a prediction head. The backbone network module performs initial convolution and pooling downsampling operations on the input raw image to generate an initial feature map. After progressive downsampling and feature extraction, the initial feature map outputs a multi-scale feature map with progressively decreasing resolution and progressively increasing channel number. The feature fusion module performs a uniform channel number transformation on the multi-scale feature maps output by the backbone network module, and then aggregates deep semantic features and shallow spatial features through a bidirectional cross-level feature fusion path from top to bottom and bottom to top, outputting fused feature maps of various scales. The prediction head predicts the location information of the detection box and the probability of the target category based on the fused feature maps of different scales. Finally, a non-maximum suppression algorithm is used to select the optimal detection box and output the location information of the optimal detection box and the corresponding target category probability. The location information of the detection box includes the coordinates of the upper left corner of the detection box, the width of the detection box, and the height of the detection box. The detection model construction module specifically executes the steps (2) of a real-time misclassification detection method based on deep learning.

[0046] The package trajectory tracking module is used to input the original image sequence to be detected into the trained lightweight target detection model and output a set of detection boxes; then, by introducing a target tracking algorithm, the detection box set is corrected for temporal consistency using inter-frame motion correlation to obtain the package trajectory. The package trajectory tracking module specifically executes the steps (3) of a real-time misclassification detection method based on deep learning.

[0047] Example 3: An electronic device of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement a real-time misclassification detection method based on deep learning as described above.

[0048] Example 4: A computer-readable storage medium of the present invention stores a computer program thereon, which is executed by a processor to implement a real-time misclassification detection method based on deep learning as described above.

Claims

1. A real-time misclassification detection method based on deep learning, characterized in that, Includes the following steps: (1) The video stream of the package to be inspected on the belt is obtained by using an industrial camera, and the original image sequence to be inspected is obtained by slicing it. (2) Construct a lightweight object detection model, then use annotation software to annotate the package detection boxes, create a package training dataset, use the package training dataset to train the lightweight object detection model, and obtain the trained lightweight object detection model; The lightweight object detection model includes a backbone network module, a feature fusion module, and a prediction head. The backbone network module performs initial convolution and pooling downsampling operations on the input raw image to generate an initial feature map. After progressive downsampling and feature extraction, the initial feature map outputs a multi-scale feature map with progressively decreasing resolution and progressively increasing channel number. The feature fusion module performs a uniform channel number transformation on the multi-scale feature maps output by the backbone network module, and then aggregates deep semantic features and shallow spatial features progressively through a bidirectional cross-level feature fusion path from top to bottom and bottom to top, outputting fused feature maps of various scales. The prediction head predicts the location information of the detection box and the probability of the target class based on the fused feature maps of different scales. Finally, a non-maximum suppression algorithm is used to select the optimal detection box, and the location information of the optimal detection box and the corresponding target class probability are output. The location information of the detection box includes the coordinates of the top-left corner of the detection box, the width of the detection box, and the height of the detection box. (3) Input the original image sequence to be detected into the trained lightweight target detection model and output the detection box set; then, by introducing the target tracking algorithm, use the inter-frame motion correlation to perform temporal consistency correction on the detection box set to obtain the package trajectory.

2. The real-time misclassification detection method based on deep learning according to claim 1, characterized in that, The backbone network module includes a convolutional block, a pooling layer, and three cascaded ES modules connected in sequence. The original image is input into the backbone network module and undergoes initial convolution and pooling processing to obtain an initial feature map. The initial feature map is then passed through the three cascaded ES modules for progressive downsampling and feature extraction, outputting the first feature layer, the second feature layer, and the third feature layer, respectively. The ES module includes depthwise separable convolutional layers and an SE attention mechanism module.

3. The real-time misclassification detection method based on deep learning according to claim 1, characterized in that, The feature fusion module includes convolutional layers and CSP modules corresponding to the three cascaded ES modules. The first feature layer, the second feature layer, and the third feature layer are respectively input to the convolutional layer. The number of channels of the output feature map of the convolutional layer is adjusted to a preset dimension. Then, the feature is fused through the CSP module to obtain four fused feature maps of different scales. in The third feature layer undergoes point-by-point convolution processing through a convolutional layer, adjusting its channel number to a preset dimension, and outputting the first intermediate feature map. The second feature layer undergoes pointwise convolution processing through a convolutional layer to adjust its channel number to a preset dimension, outputting a second intermediate feature map; the second intermediate feature map is then concatenated with the upsampled first intermediate feature map and input into the CSP module for feature fusion to obtain a first fused feature map. The first feature layer undergoes pointwise convolution processing through convolutional layers to adjust its channel count to a preset dimension, outputting a third intermediate feature map. This third intermediate feature map is then concatenated with the upsampled first fused feature map and input into the CSP module for feature fusion, yielding a second fused feature map. The downsampled second fused feature map is then concatenated with the first fused feature map and input into the CSP module for feature fusion, yielding a third fused feature map. The downsampled third fused feature map is then concatenated with the first intermediate feature map and input into the CSP module for feature fusion, yielding a fourth fused feature map. Finally, the downsampled fourth fused feature map is added to the downsampled first intermediate feature map to obtain a fifth fused feature map. The second fused feature map is input into the first prediction head, the third fused feature map is input into the second prediction head, the fourth fused feature map is input into the third prediction head, and the fifth fused feature map is input into the fourth prediction head.

4. The real-time misclassification detection method based on deep learning according to claim 1, characterized in that, In step (2), the non-maximum suppression algorithm is used to select the optimal detection box. Specifically, the detection boxes output by the detection head are arranged in descending order of target confidence score. The detection box with the highest confidence score is selected as the retained detection box. The intersection-union ratio of the remaining detection boxes and the retained detection boxes is calculated. Redundant detection boxes with intersection-union ratio greater than a preset threshold are removed. The process is iterated until all detection boxes are processed. The location information of the optimal detection box corresponding to each target package and the target category probability are output.

5. The real-time misclassification detection method based on deep learning according to claim 1, characterized in that, The target tracking algorithm in step (3) includes a Kalman filter and a Hungarian algorithm. The specific steps of the target tracking algorithm are as follows: (3.1) Maintain a trajectory pool. The trajectory pool is empty in the initial frame. The trajectory pool stores the trackers that have been established before the current frame. Each tracker encapsulates the corresponding package tracking trajectory and Kalman filter state. The Kalman filter state includes a state vector and its corresponding covariance matrix. The state vector contains the position parameters and position change rate parameters of the target detection box. The covariance matrix is ​​used to characterize the uncertainty of each state parameter and their correlation. Obtain the package tracking trajectory and Kalman filter state output and stored in the previous frame from the trajectory pool as the input of the current frame. For each package tracking trajectory in the current frame, use the Kalman filter to calculate the predicted state vector and predicted covariance matrix of the current frame based on the state vector and covariance matrix of the previous frame, combined with the state transition matrix and the process excitation noise covariance matrix, to obtain the set of tracking boxes of each package in the current frame, i.e., the set of predicted boxes. (3.2) Input the current frame image into the lightweight object detection model, output the detection results of all packages in the current frame, and form a set of detection boxes; (3.3) For the set of tracking boxes obtained in step (3.1) and the set of detection boxes obtained in step (3.2), each tracking box and each detection box is expanded outward along the x and y directions at the same ratio with their respective center points as the center, so as to increase the area of ​​each box and increase the probability that there is an intersection between the detection box and the tracking box. (3.4) Calculate the generalized intersection-union ratio (gIoU) between each detection box and each tracking box after expansion in step (3.3), and construct the cost matrix using the generalized intersection-union ratio (gIoU) as the similarity index; (3.5) Input the cost matrix constructed in step (3.4) into the Hungarian algorithm to solve the minimum cost matching between the detection box and the tracking box, and determine the correspondence between each detection box and each tracking box; (3.6) Based on the matching results of the Hungarian algorithm, all packages are divided into the following three categories: No matching tracking box: The tracking box failed to match any detection box; No matching detection box: The detection box failed to match any tracking box; Match successful: The detection box and the tracking box were successfully matched; (3.7) For a successfully matched package, the detection box matched in the current frame is used as the observation value and input into the Kalman filter to perform the update phase, that is, the prediction state vector and prediction covariance matrix obtained in step (3.1) are corrected using the observation value of the current frame to obtain the output tracking box of the current frame; at the same time, the tracking frame number of the package tracker is incremented by 1, the skipped frame number is reset to 0, and the ID of the package remains unchanged. (3.8) For unmatched detection boxes, determine that they are newly entered packages in the current frame and perform the following operations: use the detection box as the initial observation value, create a new tracker and assign it a unique identification number, and increment the total number of IDs by 1; at the same time, initialize the corresponding Kalman filter with the detection box, and use the detection box as the output tracking box of the current frame, as well as the first frame position of the tracker's starting tracking trajectory; add the newly created tracker to the trajectory pool; (3.9) For unmatched tracking boxes, increment the skipped frame count of the package tracker by 1, and determine whether the current skipped frame count has reached the preset skipped frame count threshold. If the threshold has not been reached, temporarily retain the tracker, and use the Kalman filter prediction result of the tracker, i.e. the tracking box output in step (3.1), as the output tracking box of the current frame. The ID of the package tracker remains unchanged. If the threshold has been reached, determine that the package has left the scene or has been permanently lost, and delete the tracker and its trajectory. (3.10) Output the current frame tracking box obtained from steps (3.7) to (3.9) and use it as the input for the next frame. Repeat steps (3.1) to (3.10) to achieve tracking of all wrapped targets in the original image sequence.

6. A real-time misclassification detection system based on deep learning, characterized in that, include: The image acquisition module is used to acquire video streams of packages to be inspected on a belt using an industrial camera, and then slice them to obtain the original image sequence to be inspected. The detection model building module is used to build a lightweight object detection model. Then, the package detection boxes are labeled with annotation software to create a package training dataset. The lightweight object detection model is trained using the package training dataset to obtain the trained lightweight object detection model. The lightweight object detection model includes a backbone network module, a feature fusion module, and a prediction head. The backbone network module performs initial convolution and pooling downsampling operations on the input raw image to generate an initial feature map. After progressive downsampling and feature extraction, the initial feature map outputs a multi-scale feature map with progressively decreasing resolution and progressively increasing channel number. The feature fusion module performs a uniform channel number transformation on the multi-scale feature maps output by the backbone network module, and then aggregates deep semantic features and shallow spatial features progressively through a bidirectional cross-level feature fusion path from top to bottom and bottom to top, outputting fused feature maps of various scales. The prediction head predicts the location information of the detection box and the probability of the target class based on the fused feature maps of different scales. Finally, a non-maximum suppression algorithm is used to select the optimal detection box, and the location information of the optimal detection box and the corresponding target class probability are output. The location information of the detection box includes the coordinates of the top-left corner of the detection box, the width of the detection box, and the height of the detection box. The package trajectory tracking module is used to input the original image sequence to be detected into the trained lightweight target detection model and output a set of detection boxes. Then, by introducing a target tracking algorithm, the temporal consistency of the detection box set is corrected by using inter-frame motion correlation to obtain the package trajectory.

7. The real-time misclassification detection system based on deep learning according to claim 6, characterized in that, The backbone network module includes a convolutional block, a pooling layer, and three cascaded ES modules connected in sequence. The original image is input into the backbone network module and undergoes initial convolution and pooling processing to obtain an initial feature map. The initial feature map is then passed through the three cascaded ES modules for progressive downsampling and feature extraction, outputting the first feature layer, the second feature layer, and the third feature layer, respectively. The ES module includes depthwise separable convolutional layers and an SE attention mechanism module.

8. The real-time misclassification detection system based on deep learning according to claim 6, characterized in that, The feature fusion module includes convolutional layers and CSP modules corresponding to the three cascaded ES modules. The first feature layer, the second feature layer, and the third feature layer are respectively input to the convolutional layer. The number of channels of the output feature map of the convolutional layer is adjusted to a preset dimension. Then, the feature is fused through the CSP module to obtain four fused feature maps of different scales. in The third feature layer undergoes point-by-point convolution processing through a convolutional layer, adjusting its channel number to a preset dimension, and outputting the first intermediate feature map. The second feature layer undergoes pointwise convolution processing through a convolutional layer to adjust its channel number to a preset dimension, outputting a second intermediate feature map; the second intermediate feature map is then concatenated with the upsampled first intermediate feature map and input into the CSP module for feature fusion to obtain a first fused feature map. The first feature layer undergoes pointwise convolution processing through convolutional layers to adjust its channel count to a preset dimension, outputting a third intermediate feature map. This third intermediate feature map is then concatenated with the upsampled first fused feature map and input into the CSP module for feature fusion, yielding a second fused feature map. The downsampled second fused feature map is then concatenated with the first fused feature map and input into the CSP module for feature fusion, yielding a third fused feature map. The downsampled third fused feature map is then concatenated with the first intermediate feature map and input into the CSP module for feature fusion, yielding a fourth fused feature map. Finally, the downsampled fourth fused feature map is added to the downsampled first intermediate feature map to obtain a fifth fused feature map. The second fused feature map is input into the first prediction head, the third fused feature map is input into the second prediction head, the fourth fused feature map is input into the third prediction head, and the fifth fused feature map is input into the fourth prediction head.

9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a real-time misclassification detection method based on deep learning as described in any one of claims 1-5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor to implement a real-time misclassification detection method based on deep learning as described in any one of claims 1-5.

Images