Method for achieving NMS calculation in large graph slice reasoning post-processing based on Transform

Abstract

The invention discloses a method for realizing NMS (Network Management System) calculation in large graph slice reasoning post-processing based on Transform, and in order to realize NMS filtering operation, the method comprises the following steps: inputting coordinates, categories and confidence of a target frame detected by a deep learning network as semantic information into a Transform model, and learning to obtain filtered coordinates and other information; secondly, in order to improve the model reasoning speed, the model structure of transformer is simplified; therefore, end-to-end reasoning of the target detection network is realized in a real sense, and the process of post-processing of the model on a CPU is omitted.

Description

Technical Field

[0001] The present invention relates to the field of object detection technology, and specifically relates to a method for realizing NMS calculation in the post-processing of large image slicing inference based on Transformer. Background Art

[0002] Object detection is a key task in computer vision, and its goal is to identify all objects of interest in an image or video and determine the category and location of each object (usually represented in the form of a bounding box). Object detection has a wide range of applications in fields such as autonomous driving, intelligent monitoring, human-computer interaction, and medical image analysis. Currently, the mainstream deep learning models based on the CNN architecture mainly have two categories. The first is the two-stage detection network exemplified by the R-CNN series, and the second is the one-stage network exemplified by the YOLO series.

[0003] However, these two types of models need to perform NMS calculation in the post-processing of the model, aiming to filter out the detection boxes at the same position with low confidence. Summary of the Invention

[0004] Object of the Invention: The object of the present invention is to solve the deficiencies existing in the prior art, and provide a method for realizing NMS calculation in the post-processing of large image slicing inference based on Transformer, which processes the last link (i.e., non-maximum suppression NMS) in the detection model based on deep learning, and is a general method for processing NMS.

[0005] Technical Solution: A method for realizing NMS calculation in the post-processing of large image slicing inference based on Transformer of the present invention includes the following steps:

[0006] Step 1, construct a training data set, and the specific method is as follows:

[0007] Step 1.1, cut the high-resolution image with overlap to obtain a number of sub-images, and obtain the true label information of each sub-image according to the label information of the original image before cutting;

[0008] Step 1.2: Input each sub-image into the trained detection model (such as YOLO series or Faster-RCNN), and output the corresponding detection results, including the coordinates, category and confidence of the target; map the coordinates of the detection box based on the sub-image to the real coordinates in the original image through coordinate transformation (to ensure that the coordinates of all targets are in the same coordinate system); coordinate transformation is used because the coordinates of the detection result are based on the sub-image obtained by cutting, rather than the coordinates of the original image. For example, the coordinates of the upper left corner of the cut sub-image relative to the original image are (200,200), and the upper left and lower right coordinates of the detected target are (20,20), (40,40), then the coordinates should be modified to (220,220), (240,240);

[0009] Step 1.3, then write the test results obtained in step 1.2 to the same line in the same file (such as Figure 5 As shown in the figure, the detection results of different targets are separated by commas, and the detection results of different subgraphs are separated by semicolons (the purpose of commas and semicolons is to play a semantic guidance role, thereby reducing the learning difficulty of the model). At the same time, the IOU hyperparameter threshold for calculating NMS is set; so far, the above-obtained data is used as the input value X, the input value X is input into the NMS code to run, and the output is used as the Y value for supervised learning of the model; the IOU hyperparameter is input into the model as the IOU threshold for calculating NMS. After training, the model can dynamically adjust the calculation output of NMS by setting the IOU threshold in actual operation;

[0010] Step 2: Train the Tranformer model. Specific method:

[0011] Step 2.1: Slim down the original Tranformer model, and the slimmed-down Tranformer model adopts a single-head attention mechanism;

[0012] Step 2.2: Use GPU to train the slimmed-down Tranformer model, calculate the cross loss between the input X, the output Y' after the model passes through the model, and the real Y, and then let the slimmed-down Tranformer model learn how to calculate NMS;

[0013] Step 3: Infer the target detection model. The specific method is:

[0014] Step 3.1: Integrate the coordinates, labels, and confidence information of all sub-images obtained through the pre-trained detection model inference, and finally set the IOU of this inference to obtain complete input information;

[0015] Step 3.2: Input the data into the Transformer model trained in Step 2. Through the NMS calculation process simulated by the deep learning model, obtain the output results of the new tokens. Then, map the output token information through the vocabulary to obtain the filtered target coordinates, labels, confidence levels, etc. Further, in Step 1.1, the overlapping cutting is performed in the way of a sliding window. When moving the window for cutting, ensure that there are overlapping areas between the front and back, and between the upper and lower windows (there are overlapping areas between the front and back windows, between the upper and lower windows, and there are also overlaps among the four windows).

[0016] Further, in Step 2.1, the original Transformer model is slimmed down. The single-head attention mechanism is adopted and the positional encoding is removed, greatly reducing the number of parameters of the original model. The slimmed-down Transformer model has the following structure:

[0017] The Input Embedding module for performing embedding operations on the input words, the single-head self-attention mechanism, the Masked Attention with a mask, the Feed Forward neural network, and the Softmax activation function for multi-classification tasks.

[0018] Further, the specific formula for calculating the cross-loss function in Step 2.2:

[0019]

[0020] Further, the method for integrating the coordinate, label, and confidence information in Step 3 is: merge and splice the three types of data into a string.

[0021] Beneficial effects: The present invention uses deep learning technology to implement the NMS operation, directly using the GPU to achieve it, reducing the CPU occupancy rate, and at the same time being able to make full use of the resources of the GPU. Through the NMS operation, the entire detection is made into a complete end-to-end model, and no additional CPU calculation is required, improving the overall running speed.

[0022] At the same time, to be able to implement the filtering operation of NMS, the present invention takes the coordinates, categories, and confidence levels of the target boxes detected by the deep learning network as semantic information and inputs them into the slimmed-down Transformer model to learn and obtain the filtered coordinate and other information. In addition, using the slimmed-down Transformer model can improve the model inference speed. Brief Description of the Drawings

[0023] Figure 1 is the overall network framework diagram of the present invention;

[0024] Figure 2It is a diagram of the model training framework;

[0025] Figure 3 It is a diagram of the model inference framework;

[0026] Figure 4 It is a schematic diagram of the thinned Transformer model in the embodiment;

[0027] Figure 5 It is a format diagram of the input data during model training in the embodiment;

[0028] Figure 6 It is the effect diagram of the embodiment. Detailed implementation manners

[0029] The technical solution of the present invention will be described in detail below, but the protection scope of the present invention is not limited to the described embodiments.

[0030] The existing NMS calculation is implemented by the CPU. In order to release the CPU resources and make full use of the high efficiency of the multi-threaded calculation of the GPU, the present invention introduces Transformer to implement the NMS calculation, which improves the speed of model post-processing to a certain extent. As the most mainstream basic framework of deep learning at present, the Transformer model has become the preferred framework for deep learning NLP tasks because it is good at processing sequence inputs. NMS has two uses in the current object detection tasks. The first one is that in the mainstream CNN detection network, since multiple detection boxes are generated for the same object, NMS is needed to filter out the detection boxes with lower confidence. The second one is that in the inference process of high-resolution images, in order to improve the detection accuracy of small objects, the large image is usually cut into sub-images with overlapping regions, and the cut sub-images are used for actual inference. Since the cutting is overlapping, it will inevitably lead to the repeated detection of the objects in the overlapping regions. Finally, NMS is also needed to filter the object boxes detected by multiple sub-images.

[0031] In order to implement the filtering operation of NMS, the present invention takes the coordinates, categories, and confidence levels of the object boxes detected by the deep learning network as semantic information and inputs them into the transformer model to learn the information such as the coordinates after filtering. Secondly, in order to improve the model inference speed, the present invention simplifies the model structure of the transformer. Thus, the end-to-end inference of the object detection network is truly realized, and the process of post-processing the model on the CPU is omitted.

[0032] As Figures 1 to 3 shown, the method for implementing NMS calculation in the post-processing of large image slicing inference based on Transformer in this embodiment includes the following steps:

[0033] Step 1. Construct a training dataset. The specific method is as follows:

[0034] Step 1.1. Cut the high-resolution image with overlap to obtain several sub-images, and obtain the true label information of each sub-image according to the label information of the original image before cutting.

[0035] Step 1.2. Input each sub-image into the trained detection model, and output the corresponding detection results, including the coordinates, categories, and confidence levels of the targets; map the coordinates of the detection boxes based on the sub-images to the true coordinates in the original image through coordinate transformation.

[0036] Step 1.3. Then write the detection results obtained in Step 1.2 into the same line in the same file, separate the detection results of different targets with commas, and separate the detection results of different sub-images with semicolons; at the same time, set the IOU threshold for calculating NMS.

[0037] So far, use the above-obtained data as the input value X, input the input value X into the NMS code to run, and output the Y value for the supervised learning of the model; run the NMS code written on the CPU, input X into it, and obtain the value after NMS filtering.

[0038] Step 2. Train the Tranformer model. The specific method is as follows:

[0039] Step 2.1. Slim down the original Tranformer model. The slimmed-down Tranformer model uses a single-head attention mechanism.

[0040] Step 2.2. Use the GPU to train the slimmed-down Tranformer model, calculate the cross loss between the Y' value output by the model after inputting X and the true Y value, and then let the slimmed-down Tranformer model learn how to perform NMS calculations; the calculation process of the current mainstream NMS is mainly carried out on the cpu. To release the precious resources of the cpu and implement the end-to-end inference task of the model, this embodiment uses the slimmed-down Transformer model (as Figure 4 shown, the multi-head attention is changed to single-head attention, and the position encoding is removed at the same time) to implement the NMS calculation process, which can make full use of the performance of the GPU.

[0041] Step 3. Perform inference on the target detection model. The specific method is as follows:

[0042] Step 3.1. Integrate the coordinate, label, and confidence information obtained by inferring all sub-images through the pre-trained detection model, and then set the IOU value for this inference to obtain the complete input information.

[0043] As Figure 5As shown in the figure, in this embodiment, the coordinates, labels, and confidence levels are combined and spliced into a single string, that is, x1, x2, y1, y2, cls, conf, and IOU are converted into specific numerical values, where the first four are coordinates, cls is the class name, conf is the confidence level of the target, and IOU is the intersection over union ratio;

[0044] Step 3.2: Input the data obtained in Step 1.3 into the model. After the NMS calculation simulated by the deep learning model, new token information is obtained, and then the output token information is mapped through the vocabulary to obtain the filtered target coordinates, labels, and confidence levels.

[0045] In Step 1.1 of this embodiment, the original image is overlapped and cut using a sliding window method. When moving the window for cutting, ensure that there is an overlapping area between the front and back, and the upper and lower windows, as Figure 6 shown in the figure. There are four sliding window slices in the figure, and there are overlapping areas among the four windows.

[0046] In summary, the present invention can release CPU resources while making full use of the high efficiency of the multi-threaded computing of the GPU. By introducing Transformer to implement the NMS calculation, the speed of the model post-processing is improved to a certain extent. The detection boxes with low confidence levels are filtered out through NMS, and at the same time, the target boxes detected by multiple sub-images are filtered through NMS.

Claims

1. A method for implementing NMS calculation in large graph slice reasoning post-processing based on Transformer, characterized in that: The following steps are involved: Step 1: Build a training data set. The specific method is: Step 1.1, cut the high-resolution image with overlaps to obtain several sub-images, and obtain the real label information of each sub-image based on the label information of the original image before cutting; Step 1.2: Input each sub-image into the trained detection model and output the corresponding detection results, including the coordinates, category and confidence of the target; map the coordinates of the sub-image-based detection box to the real coordinates in the original image through coordinate transformation; Step 1.3, then write the detection results obtained in step 1.2 into the same line in the same file, separate the detection results of different targets by commas and separate the detection results of different sub-graphs by semicolons; at the same time, set the IOU threshold for calculating NMS; At this point, the above data is used as the input value X, the input value X is input into the NMS code to run, and the output is used as the Y value for supervised learning of the model; Step 2: Train the Tranformer model. Specific method: Step 2.1: Slim down the original Tranformer model, and the slimmed-down Tranformer model adopts a single-head attention mechanism; Step 2.2: Use GPU to train the slimmed-down Tranformer model, calculate the cross loss between the Y' value output by the model after the input X passes through the model and the actual Y value, and then let the slimmed-down Tranformer model learn how to calculate NMS; Step 3: Infer the target detection model. The specific method is: Step 3.1: Integrate the coordinates, labels, and confidence information of all sub-images inferred by the pre-trained detection model, and then set the IOU value of this inference to obtain complete input information; Step 3.2: Input the data obtained in step 1.3 into the Tranformer model trained in step 2. After NMS calculation simulated by the deep learning model, new token information is obtained. Then, the output token information is mapped through the vocabulary to obtain the filtered target coordinates, labels and confidence.

2. The method for implementing NMS calculation in large graph slice reasoning post-processing based on Transformer according to claim 1 is characterized in that: Step 1.1 uses a sliding window to perform overlapping cutting of the original image. When moving the window for cutting, ensure that the front and rear windows have overlapping areas, and the upper and lower windows have overlapping areas.

3. The method for implementing NMS calculation in large graph slice reasoning post-processing based on Transformer according to claim 1 is characterized in that: The slimmed-down Tranformer model adopts a single-head attention mechanism and removes position encoding; it also includes an Input Embedding module for input word embedding operations and an activation function Softmax for multi-classification tasks.

4. The method for implementing NMS calculation in large graph slice reasoning post-processing based on Transformer according to claim 1 is characterized in that: Step 2.2 Specific formula for calculating the cross loss function:

5. The method for implementing NMS calculation in large graph slice reasoning post-processing based on Transformer according to claim 1 is characterized in that: Information integration in step 3.1 refers to combining coordinates, labels, and confidence levels into a single string.

Images