Transform target detection architecture inference acceleration system and method based on difficult sample consistency test

By introducing a difficult sample consistency verification mechanism into the Transformer object detection model, using the CNN small model to infer simple samples and handing over the difficult samples to Transformer large model for processing, the inference delay problem of the Transformer model in real-time applications is solved, and the implementation of the High-speed detection on mobile embedded devices takes into account both accuracy and real-time.

CN118968251APending Publication Date: 2024-11-15QUAN CHENG LABORATORY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410924933.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Transformer object detection model has high inference delay in real-time applications and is difficult to deploy in mobile embedded devices, especially in scenarios where high-speed detection is required.

Method used

Using an architecture based on difficult sample consistency test, the samples are inferred multiple times through the CNN small model and consistency test. If it is a simple sample, the results of the CNN small model are used. If it is a difficult sample, the transformer model is used for inference, which combines the The advantages of CNN and Transformer reduce the inference pressure of transformer models.

Benefits of technology

It effectively reduces the inference delay of the transformer model, improves the running speed on mobile embedded devices, takes into account real-time and accuracy, and reduces recognition performance within an acceptable range.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968251A_ABST
    Figure CN118968251A_ABST
Patent Text Reader

Abstract

The invention relates to a transform target detection architecture inference acceleration system and method based on difficult sample consistency check, and belongs to the field of machine learning target detection, and the system comprises a CNN small model S, a transform large model B and an inference result consistency check module. Firstly, difficult sample inspection in the field of visual target detection is carried out by utilizing and constructing a random simulation multiple inference mode, through the difficult sample inspection, simple samples are handed over to a high-speed small model, and the difficult samples are handed over to a low-speed large model, so that the inference pressure of the large model is shared, and the inference speed and the recognition precision are considered. According to the method, the advantages of the CNN algorithm and the transform technology are combined, and the inference speed of a transform large model architecture is effectively improved by means of a low-delay high-speed small model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a transformer target detection architecture inference acceleration system and method based on difficult sample consistency verification, and belongs to the field of machine learning target detection. Background Art

[0002] Object detection technology is one of the hottest research directions in computer vision, which aims to identify and locate specific target objects in images or videos. It is specifically used to automatically identify and locate the position and size of target objects in images or videos, and to classify or identify these targets and other related tasks. This technology extracts features from image or video data based on deep neural networks, and then uses classifiers and bounding box regressors to identify and locate objects. The basic supervision method of object detection usually includes three main parts: object localization, object classification, and object box regression. Object localization is responsible for accurately finding the position and size of the target in the image; object classification is to match the detected target with a predefined category; and object box regression is used to adjust the position and size of the target box according to the predicted position offset to improve the accuracy of detection.

[0003] Object detection technology has a wide range of applications in many fields, such as autonomous driving, security monitoring, image retrieval, and image analysis. In autonomous driving, it can help vehicles identify pedestrians, other vehicles, etc. to support safe driving decisions; in security monitoring, it can be used to detect abnormal behavior or identify specific targets in real time; in image retrieval, it can find relevant objects from images based on keywords, etc. This technology trains deep learning models to identify targets and marks the bounding box of the target in the image, thereby achieving automatic detection of the target. Object detection technology plays a key role in many practical applications. In addition, object detection technology has also been widely used in medical image analysis, face recognition, retail product recognition and other fields.

[0004] In many fields, real-time target detection algorithms play a key role. For example, in the decision-making and control process of self-driving cars, it is necessary to be able to process sensor data in real time, such as environmental information provided by cameras, radars, and lidars, as well as the vehicle's own status information. Through real-time algorithm processing and analysis of data, the self-driving system can identify road conditions, obstacles, and other traffic participants in real time, so as to make correct driving decisions. Therefore, how to deploy the fastest possible target detection and inference algorithm in such scenarios has become an important technology for realizing self-driving technology.

[0005] On the other hand, with the development of technology, the performance and accuracy of target detection algorithms are also constantly improving. For example, the continuous updating of the YOLO series of algorithms and the application of the Transformer method in recent years have brought higher performance and broader application prospects to target detection. The YOLO algorithm is a real-time target detection algorithm based on deep learning. Its core idea is to transform the target detection task into a single regression problem. The YOLO series of algorithms achieves high detection speed while maintaining high accuracy by optimizing the network structure and loss function. From the initial YOLOv1 to the subsequent YOLOv2, YOLOv3, and then to the latest YOLOv8, the algorithm has been continuously optimized and improved in performance. YOLOv8 inherits the advantages of previous versions and further introduces new technical features, such as Decoupled Head structure and Task Alignment Learning, thereby improving the accuracy and robustness of detection.

[0006] The Transformer method is a technology that has made significant breakthroughs in the field of natural language processing in recent years, and has also been gradually introduced into computer vision tasks. Transformer captures the dependencies between elements in a sequence through a self-attention mechanism, thereby achieving global modeling of the input data. In target detection tasks, the Transformer method can be used to build more powerful feature representations and context information modeling to improve the performance of detection algorithms. By incorporating the Transformer structure into the target detection model, its powerful modeling capabilities can be fully utilized to improve the detection of complex scenes and occluded targets. However, Transformer target detection may not be as fast as the CNN series (YOLO series algorithm) target detection algorithm in some cases, but in terms of accuracy, especially when processing global context information and complex pattern recognition tasks, Transformer generally exhibits higher accuracy.

[0007] In summary, the CNN series of object detection algorithms already have real-time application capabilities and faster detection speeds. Transformer-based technology usually has higher accuracy when processing global context information and complex pattern recognition tasks, but it is still a long way from being deployed on real-time application platforms. How to further reduce its inference latency has become a bottleneck restricting the deployment of transformer-based object detection models in real-time applications on mobile embedded devices. Summary of the invention

[0008] In view of the shortcomings of the prior art, the present invention provides a transformer target detection architecture inference acceleration system and method based on difficult sample consistency check, which can combine the advantages of CNN algorithm and transformer technology, take into account both real-time performance and accuracy, and then deploy the target detection model based on transformer technology in mobile embedded devices.

[0009] The present invention adopts the following technical solution:

[0010] A transformer target detection architecture inference acceleration system based on hard sample consistency check, including a CNN small model S, a transformer large model B and an inference result consistency check module;

[0011] The small CNN model S is used to detect targets on samples that have been replicated multiple times, and obtain multiple inference results;

[0012] The inference result consistency check module is used to calculate the consistency of multiple inference results inferred by the CNN small model S, and determine whether it is a difficult sample based on the consistency calculation results;

[0013] The large transformer model B is used to infer difficult samples.

[0014] A transformer target detection architecture inference acceleration method based on hard sample consistency check includes the following steps:

[0015] (1) The sample is first input into the CNN model S, which replicates the sample t times and batches them together for parallel inference to obtain t inference results.

[0016] (2) Send the t inference results to the inference result consistency verification module, calculate the consistency of the inference results, and determine whether the sample is a difficult sample;

[0017] (3) If it is a non-difficult sample, the structure inferred by the small CNN model S is used as the final result; if it is a difficult sample, the large transformer model B is used for inference and used as the final result.

[0018] Preferably, in step (1), the value of t can be determined according to the actual situation. According to the speed difference between the small model and the large model, the inference time of the sample CNN small model S with a single sample replicated t batches should be less than the inference time of the single sample 1 transformer large model B. For example, in yolov8, t = 4 samples, the inference time is 20ms, and the inference time of the large model RT-DETR for samples is 110ms. The former is much faster than the latter.

[0019] t is preferably 2-6.

[0020] Preferably, the CNN model S consists of a neural network f s The dropout layer is used to randomly discard data, so that the inference results of samples copied t times are different;

[0021] The CNN small model S is preferably yolov8.

[0022] Preferably, in step (2), the inference result consistency check module is composed of a visual feature similarity judgment logic, and the process of judging whether it is a difficult sample is as follows:

[0023] (2.1) First, for t inference results, according to the combination number C t 2 Take any two inference results {x ′ i ,x ′ j}, i and j correspond to the serial numbers of the two inference results, i, j∈[1,t];

[0024] (2.2) For the two results {x ′ i ,x ′ j}, take any inference result as the true label of another inference result, and calculate {x ′ i ,x ′ j}, the consistency metric function L may be cosine distance, Euclidean distance, VFL loss, DFL loss or other loss functions that quickly measure distribution distance, preferably cosine distance;

[0025] (2.3) For each pair of C obtained in step (2.2) t 2 The loss function calculation results are used to find the maximum consistency measurement function calculation result for consistency evaluation;

[0026] (2.4) For the maximum consistency metric function calculation result in step (2.3), if it is greater than the difficult sample consistency threshold Z, then this sample is considered a difficult sample, otherwise it is a non-difficult sample.

[0027] Preferably, in step (2.2), the cosine distance formula is:

[0028]

[0029] Where θ represents the vector angle.

[0030] Preferably, in step (2.4), the difficult sample consistency threshold Z can be adjusted according to the data situation, and can be 0.7 to 0.8, preferably 0.77.

[0031] Preferably, in step (3), the transformer large model B is composed of at least one transformer neural network f b Composition, target recognition is performed through the transformer architecture, and preferably, the transformer large model B is RT-DETR.

[0032] Preferably, the dropout probability of the dropout layer is p, which can be adjusted according to the data situation, and p is preferably 0.1.

[0033] For any details not provided in the present invention, please refer to the prior art.

[0034] The beneficial effects of the present invention are:

[0035] The present invention first utilizes and constructs a random simulation method to verify the consistency of multiple inferences to perform difficult sample testing in the field of visual target detection. Through this difficult sample testing, simple samples are given to a high-speed small model, and difficult samples are given to a slow large model, thereby sharing the inference pressure of the large model and taking into account both real-time and accuracy.

[0036] The present invention combines the advantages of CNN algorithm and transformer technology, and uses a low-latency and high-speed small model to effectively improve the inference speed of the transformer large model architecture while not significantly reducing the recognition performance, especially for difficult samples. This can reduce the operating deployment pressure of the target detection model based on transformer technology on mobile embedded devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings in the specification, which constitute a part of the present application, are used to provide further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0038] Figure 1 A schematic diagram of the transformer target detection architecture inference acceleration system based on hard sample consistency check of the present invention;

[0039] Figure 2 This is a flow chart of the transformer target detection architecture inference acceleration method based on difficult sample consistency check of the present invention. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the drawings in the implementation of this specification, but are not limited to this. Anything not fully described in the present invention shall be based on the conventional technology in the art.

[0041] Example 1

[0042] A transformer object detection architecture inference acceleration system based on hard sample consistency check, such as Figure 1 As shown, it includes a small CNN model S, a large transformer model B, and an inference result consistency check module;

[0043] The small CNN model S is used to detect targets on samples that have been replicated multiple times, and obtain multiple inference results;

[0044] The inference result consistency check module is used to calculate the consistency of multiple inference results inferred by the CNN small model S, and determine whether it is a difficult sample based on the consistency calculation results;

[0045] The large transformer model B is used to infer difficult samples.

[0046] Example 2

[0047] A transformer object detection architecture inference acceleration method based on hard sample consistency check, such as Figure 2 As shown, the following steps are included:

[0048] (1) The sample is first input into the CNN model S, which replicates the sample four times and concatenates them in the form of {x1, x2…, x4}, performs parallel inference, and obtains four inference results;

[0049] The CNN model S consists of a transformer neural network f s The dropout layer is used to randomly discard data, so that the inference results of the samples copied 4 times are different; the dropout layer has a dropout probability of p, and p is 0.1. The present invention introduces randomness into the CNN small model S at the same time, that is, introduces randomness through the dropout technology, and obtains the target detection model inference result through the calculation of the high-speed and low-latency target detection small model S module. The dropout layer has a dropout probability of p, and in this embodiment, p is 0.1, which can make the four inference results different without affecting the original inference accuracy and logic;

[0050] (2) Send the four inference results to the inference result consistency verification module to calculate the consistency of the inference results and determine whether the sample is a difficult sample;

[0051] The specific process is:

[0052] (2.1) First, for the four inference results, according to the combination number C4 2 Take any two inference results {x ′ i ,x ′ j}, i and j correspond to the serial numbers of the two inference results, i, j∈[1,t];

[0053] (2.2) For the two results {x ′ i ,x ′ j}, calculate {x ′ i ,x ′ j}'s consistency measurement function L;

[0054] (2.3) The consistency measurement function L is selected as the cosine distance, and the cosine distance formula is:

[0055]

[0056] Find the maximum cosine distance calculation result for consistency evaluation;

[0057] (2.4) If the maximum value of the cosine distance calculation result exceeds the threshold Z, the sample is considered a difficult sample, otherwise it is a non-difficult sample. In the actual computing environment, the threshold Z needs to be selected according to the characteristics of the data set and model or experimental method.

[0058] In this embodiment, the value of Z is 0.77.

[0059] (3) If it is a non-difficult sample, the structure inferred by the small CNN model S is used as the final result; if it is a difficult sample, the large Transformer model B is used for inference and used as the final result.

[0060] It is worth noting that the small CNN model S and the large transformer model B used in the present invention are both well-trained existing mature models. The process of inferring samples to obtain inference results can refer to the existing technology and is not the focus of the present invention.

[0061] The present invention is equivalent to using a small model to perform multiple inferences to find difficult samples. The premise for the effectiveness of this design is that the inference speed of the small model is much higher than that of the large transformer model. This is achievable in this field. For example, if the small model selects yolov8 and the large model selects RT-DETR or other DETR, the time it takes to copy the sample 4 times and batch it together and send it to the small model for inference is much less than the time it takes for the large model to infer once.

[0062] The present invention can achieve effective performance gains in the field of target detection transformer architecture, and achieve effective sample average inference speed improvement when the recognition performance of the high-performance target detection transformer model is not significantly reduced. In the target detection scenario, the transformer architecture achieves better recognition accuracy than the CNN architecture, but due to the large number of matrix operations of the transformer model, the model running speed cannot be compared with CNN. Therefore, the present invention uses a small CNN model S to reduce the inference cost of a large target detection transformer model with strong recognition performance but slow speed. The small model uses a random batch construction technology to achieve multiple inferences for the current single sample, and performs consistency checks on the inference results. If the inference results show a high degree of similarity, it means that this sample belongs to a simple sample. If the consistency of the inference results is weak, the transformer large model is used to infer the current sample. In the end, the large model only needs to infer some difficult samples with weak consistency, and the remaining samples are handed over to the small model for inference, which effectively accelerates the target detection model of the transformer architecture.

[0063] The test was conducted on the mobile embedded platform Jetson Agx Orin device, using the coco target detection data set validation set for testing, with a total of 5000 image samples. The CNN small model S uses yolov8-n, the transformer large model B uses RTDETR-x, t is 2 times, batchsize is 4, and the values ​​of each hyperparameter are as in Example 2. The speed improvement is shown in Table 1 below:

[0064] Table 1 Performance improvement results

[0065]

[0066] As shown in Table 1, compared with the calculation speed of 62.1ms of the first row of large models, the cascaded model method of the present invention can effectively reduce the average inference delay of samples. Under two different thresholds Z (0.77 and 0.80 respectively), the speed is effectively improved by 13.4% and 23.2% within the range of about 1% and 3% reduction in recognition performance. The number of large model inferences is reduced by 1516 and 2000 respectively, which effectively reduces the inference delay while ensuring that the recognition performance is still competitive.

[0067] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A transformer target detection architecture inference acceleration system based on hard sample consistency check, characterized in that: It includes a small CNN model S, a large transformer model B, and an inference result consistency check module; The small CNN model S is used to detect targets on samples that have been replicated multiple times, and obtain multiple inference results; The inference result consistency check module is used to calculate the consistency of multiple inference results inferred by the CNN small model S, and determine whether it is a difficult sample based on the consistency calculation results; The large transformer model B is used to infer difficult samples.

2. A transformer object detection architecture inference acceleration method based on hard sample consistency check, characterized in that: The steps include: (1) The sample is first input into the CNN model S, which replicates the sample t times and batches them together for parallel inference to obtain t inference results. (2) Send the t inference results to the inference result consistency verification module, calculate the consistency of the inference results, and determine whether the sample is a difficult sample; (3) If it is a non-difficult sample, the structure inferred by the small CNN model S is used as the final result; if it is a difficult sample, the large transformer model B is used for inference and used as the final result.

3. The transformer target detection architecture inference acceleration method based on hard sample consistency check according to claim 2 is characterized in that: In step (1), t is preferably 2-6.

4. The transformer target detection architecture inference acceleration method based on hard sample consistency check according to claim 2 is characterized in that: The CNN model S consists of a neural network f s The dropout layer is used to randomly discard data, so that the inference results of samples copied t times are different; The CNN small model S is preferably yolov8.

5. The transformer target detection architecture inference acceleration method based on hard sample consistency check according to claim 2, characterized in that: In step (2), the inference result consistency check module consists of a visual feature similarity judgment logic. The process of judging whether it is a difficult sample is as follows: (2.1) First, for t inference results, according to the number of combinations Take any two inference results {x′ i ,x′ j }, i and j correspond to the serial numbers of the two inference results, i, j∈[1,t]; (2.2) For the two inference results {x′ i ,x′ j }, calculate {x′ i ,x′ j }, thus obtaining The loss function calculation results; (2.3) For step (2.2) The loss function calculation results are used to find the maximum consistency measurement function calculation result for consistency evaluation; (2.4) For the maximum consistency metric function calculation result in step (2.3), if it is greater than the difficult sample consistency threshold Z, then this sample is considered a difficult sample, otherwise it is a non-difficult sample.

6. The method for accelerating transformer target detection architecture inference based on hard sample consistency check according to claim 5, characterized in that: In step (2.2), the consistency measurement function L is selected as the cosine distance, and the formula is:

7. The transformer target detection architecture inference acceleration method based on hard sample consistency check according to claim 5, characterized in that: In step (2.4), the difficult sample consistency threshold Z is set between 0.7 and 0.

8.

8. The method for accelerating transformer target detection architecture inference based on hard sample consistency check according to claim 7, characterized in that: In step (3), the transformer model B consists of at least one transformer neural network f bb Composition, target recognition is performed through the transformer architecture, and preferably, the transformer large model B is RT-DETR.

9. The method for accelerating transformer target detection architecture inference based on hard sample consistency check according to claim 8, characterized in that: The dropout probability of the dropout layer is p, and p is preferably 0.1.