Clue extraction method based on contour points and adaptive apparent characteristics

Abstract

The invention discloses a clue extraction method based on contour points and self-adaptive apparent features, which comprises the following steps: firstly, acquiring original image data, and performing feature extraction through a basic feature extraction network to generate a basic feature map; processing through a task execution network to obtain a result graph; for the result image, selecting a corresponding apparent feature vector for the detection target in a mapping matching mode in a post-processing part, enabling the apparent feature vector to correspond to the detection target, and finally obtaining the result image with feature clues to complete clue extraction; according to the method, on the basis of the multi-task parallel structure of the JDE normal form, the shape and the posture of the target can be described more detailedly and visually by adopting the self-adaptive target contour points. And the contour information of the target is captured through a representation mode of mapping matching, so that geometric features are enriched, and accurate identification and tracking of the target are facilitated. And an optimization scheme based on a bounding box clue extraction part is provided to alleviate the problem of misalignment between the detection target and the apparent characteristics.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision detection, and in particular to a clue extraction method based on contour points and adaptive surface features. Background Art

[0002] As an important branch of computer vision, multi-target tracking has shown great potential and application value in traffic scenes, intelligent monitoring and motion analysis. However, the current multi-target tracking technology still faces challenges, such as insufficient tracking robustness and operating efficiency under special conditions, which are directions that need further research and optimization. At present, the multi-target tracking task is mainly based on deep learning technology and is divided into two core parts: clue extraction and trajectory tracking. The clue extraction stage detects and identifies the target in the current video frame, extracts motion clues and encodes the apparent features; the trajectory tracking stage uses these clues to associate the historical trajectory with the target in the current frame, maintain and update the trajectory state. According to the different clue extraction processing processes, the multi-target tracking algorithm can be divided into three main technical routes: Tracking by Detection (TBD), Joint Detection and Embedding (JDE), and End to End (ETE) tracking paradigm.

[0003] The multi-target tracking algorithm based on the TBD paradigm has the overall characteristics of each module being independent and running sequentially, so that in terms of tracking performance, each independent module can be fully optimized without considering compatibility issues, and the performance of the entire tracker can be maximized; in terms of update iteration, this non-coupled structure has strong scalability. However, the serialized structure makes the tracker perform poorly in terms of operating efficiency: the apparent feature encoding based on the neural network requires feature extraction for each detected target, and the running time of the tracking algorithm will be superimposed with the number of targets in the scene, which will inevitably generate a large amount of computational redundancy, making it difficult for the tracking algorithm to achieve real-time operation.

[0004] In order to eliminate computational redundancy in the TBD paradigm and accelerate the running efficiency of the tracking algorithm, the researchers referred to the algorithm evolution route in the field of target detection to process the target detection task and the apparent feature encoding task in parallel, realizing the simultaneous extraction of motion clues and apparent clues, thus avoiding computational redundancy and forming the JDE paradigm. During reasoning, it only needs to be run once to obtain all tracking clues, which greatly optimizes the running efficiency; in terms of training, the part that needs to be trained in this type of algorithm exists in an end-to-end form, so it is easy to train.

[0005] However, the multi-task parallel structure also has many disadvantages: the bounding box in the JDE method cannot provide sufficiently accurate target positioning information and may be mixed with other redundant information. This will affect the subsequent apparent feature encoding and make the feature expression inaccurate. In the post-processing process, the method of directly selecting the detection target may cause the position and size of the detection target to be misaligned with the receptive field of the apparent feature. This alignment problem will make the extracted features unable to accurately reflect the actual position and size of the target, thus affecting the tracking effect.

[0006] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0007] In view of the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide a clue extraction method based on contour points and adaptive surface features, aiming to solve the problem that the method of directly selecting the detection target in the prior art causes the position and size of the detection target to be misaligned with the receptive field of the surface features, thereby affecting the tracking effect.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A clue extraction method based on contour points and adaptive surface features, comprising:

[0010] Step 1: Get the original image data, extract features through the basic feature extraction network, generate the basic feature map, and provide the feature map basis for the subsequent network;

[0011] Step 2: The basic feature map is processed by the task execution network to obtain the result map. The task execution network uses a feature enhancement component based on the spatial attention mechanism SAM and a DCN component based on adaptive shape alignment to form an apparent feature encoding task head.

[0012] Step 3: For the result image, in the post-processing part, a mapping matching method is used to select the corresponding apparent feature vector for the detection target, and the apparent feature vector is matched with the detection target, and finally a result image with feature clues is obtained to complete the clue extraction.

[0013] Furthermore, in step 1, the basic feature extraction network serves as the head of clue selection, including a backbone network. The backbone network performs preliminary processing on the original image data, reduces the feature dimension of the image, and improves the feature semantics of the image, that is, processes the original image data into effective feature data, and when the original image data flows in the backbone network, it will generate feature maps of various granularities, semantics, and receptive fields.

[0014] Furthermore, in the step 1, on the basic feature network, after the backbone network, a feature fusion network is also set. The feature fusion network adopts a feature pyramid network structure, and layers are set on the 8-fold, 16-fold, and 32-fold downsampled feature maps of the backbone network for feature fusion. Finally, three fused basic feature maps are output for subsequent task execution networks, among which small-resolution feature maps with large receptive fields and high-level semantic features are used to detect small targets, and conversely, large-resolution feature maps are used to detect large targets.

[0015] Furthermore, in the task execution network of step 2, the basic feature map is converted into a special feature map of the task head after passing through a task head composed of three non-linearly activated ordinary convolution components; the special feature map passes through a non-linearly activated ordinary convolution component and a linearly activated ordinary convolution component to obtain an intermediate result map with 2×9 channels as output, which can not only be converted into preliminary contour points of the target, but also used as a sampling offset of the deformable convolution component in the second stage.

[0016] Furthermore, after obtaining the intermediate result map, the DCN component performs a convolution operation with the dedicated feature map as input and the intermediate result as the sampling offset, and outputs a prediction result map of target refined contour points and detection confidence with 9×2+2 channels, where 9 represents the 9 contour points representing the target, the first 2 represents the sum coordinates of the contour points, and the second 2 represents the detection confidence.

[0017] Furthermore, in the post-processing part, the detection target is still obtained through confidence and NMS screening, and then the center point mapping selection CMS method is used to obtain the apparent features for the detection target. At the same time, the CMS method is further improved to obtain the adaptive apparent feature selection method AAFS based on contour points to obtain the center point coordinates of the detection target; finally, the center point is mapped to the apparent feature vector map according to the corresponding downsampling multiple, and four feature vectors are extracted from the apparent feature vector map using the bilinear interpolation method. The four feature vectors are obtained according to the following formula:

[0018]

[0019] Among them, E total is the apparent feature vector of the final output, is the number of divided parts, and E i is the apparent eigenvector corresponding to the center point of the i-th part, k i is the number of contour points in the i-th part, K ​​is the number of contour points describing the target, that is, the weight of each feature vector is determined by the number of contour points distributed in this part;

[0020] The four feature vectors are fused to obtain the apparent feature vector representing the target, which is then stored in the output set. The unmodified bounding box set and the updated feature vector set are returned, and finally a result map with feature clues is obtained to complete clue extraction.

[0021] Furthermore, the adaptive appearance feature selection method AAFS method is based on the center point mapping selection CMS method and combines the contour point information to operate: First, all the contour points of the detection target are used according to the following formula:

[0022]

[0023] Where P center is the center point coordinate, is the i-th coordinate, and n is the number of points involved in the calculation;

[0024] The coordinates of the center point of the target are calculated, and then the target is divided into four parts using the horizontal and vertical coordinates of the center point as dividing lines; for each part, the coordinates of the center point of the part are calculated using the contour points distributed therein and the center point of the target according to the above formula.

[0025] The technical solution adopted by the present invention has the following beneficial effects:

[0026] In the present invention, based on the multi-task parallel structure of the JDE paradigm, a more flexible target representation form is adopted, that is, the adaptive target contour points can describe the shape and posture of the target in more detail and intuitively. This representation method captures the contour information of the target, thereby providing rich geometric features, which helps to accurately identify and track the target. For the improvement of the task header part and the post-processing part, an optimization scheme based on the bounding box clue extraction part is proposed to alleviate the problem of misalignment between the detected target and the apparent features.

[0027] Secondly, the anchor points set by the present invention according to the resolution of the basic feature map can be considered as all the contour points of the target distributed in the same position. The FPN structure allocates targets of different scales to different levels, which alleviates the missed detection caused by "one anchor point can only be responsible for checking one target" and "multiple targets fall on the same anchor point";

[0028] Finally, the detection and recognition task heads of the present invention both adopt the basic module for generating special feature maps, which can realize the decoupling of the basic feature maps to a certain extent and alleviate the conflict problem of multi-task parallel training. At the same time, in order to solve the problem of apparent feature alignment, a feature enhancement component based on the spatial attention mechanism and a DCN component with adaptive shape alignment are used to form the apparent feature encoding task head, that is, a part of the detection task head. The feature enhancement component processes the basic feature map through pooling and ordinary convolution operations, and then generates a feature weight map using the Sigmoid function. The weight map is element-wise multiplied with the basic feature map, the feature value in the basic feature map is adjusted, the relevant features are enhanced, the irrelevant features are suppressed, and the DCN component is assisted to better aggregate the target apparent features. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart in an embodiment of the present invention;

[0030] Figure 2 is a structural schematic diagram of a method for selecting adaptive appearance features in an embodiment of the present invention;

[0031] Figure 3 is a schematic diagram of the structure of an enhanced feature module based on a spatial attention mechanism in an embodiment of the present invention;

[0032] Figure 4 is a schematic diagram of the structure of a clue extraction algorithm used in an embodiment of the present invention;

[0033] Figure 5 is a diagram of visual comparison results in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0035] A clue extraction method based on contour points and adaptive surface features. The overall process is as follows Figure 1 The specific structure is as shown in Figure 4 As shown, the specific steps include:

[0036] 1) The basic feature network performs feature extraction. As the head of clue selection, the basic feature extraction network is responsible for providing feature map basis for subsequent networks to complete the execution of various upstream tasks.

[0037] 2) For the basic feature extraction network, it often includes two parts: the backbone network and the feature fusion network. The backbone network is responsible for reducing the feature dimension of the image and improving the feature semantics of the image, that is, processing the original image data into effective feature data for subsequent task execution. When the image data flows in the backbone network, feature maps of various granularities, semantics, and receptive fields are generated to provide multi-level feature maps for the subsequent feature fusion network. This algorithm uses the DarkNet-53 network, in which the residual structure is used to deepen the number of network layers and avoid the problem of gradient disappearance or explosion. A feature fusion network is often set up after the backbone network, which is responsible for fusing the multi-level feature maps in the backbone network, improving the expressiveness of the feature maps, and distributing the processing of targets of different scales on feature maps of multiple resolutions to reduce the burden on each task head. This algorithm uses a feature pyramid network structure, which sets layers on the 8x, 16x, and 32x downsampled feature maps of the backbone network for feature fusion, and finally outputs three fused basic feature maps for subsequent task execution networks. Among them, small-resolution feature maps with large receptive fields and high-level semantic features are used to detect small targets, and conversely, large-resolution feature maps are used to detect large targets.

[0038] 3) The task execution network is set up on the three levels of the feature fusion network, using the same structure of the network, including the target detection task head and the target recognition task head, which are responsible for target positioning, detection confidence prediction, target classification and appearance feature encoding tasks respectively. At the same time, the decoupling task head of the YOLOX algorithm is used to separate the target classification task from the detection task head and place it in the recognition task head to alleviate the conflict problem of multi-task parallel training.

[0039] 4) The improvement of the task execution network is mainly reflected in the task head structure. The idea of ​​gradually refining the positioning is adopted to predict the contour points of the target, which is divided into two stages:

[0040] The basic feature map is transformed into a dedicated feature map of the task head after passing through a task head composed of three ordinary convolution components with nonlinear activations;

[0041] Then, the dedicated feature map enters the first stage, and after a nonlinearly activated ordinary convolution component and a linearly activated ordinary convolution component, an intermediate result map with 2×9 channels is obtained as the output of the first stage. The intermediate result can not only be converted into the preliminary contour points of the target, but also serve as the sampling offset of the deformable convolution component in the second stage; the second stage consists of a DCN component and a linearly activated ordinary convolution component. DCN is introduced in the second summary. The DCN component performs a convolution operation with the dedicated feature map as input and the intermediate result as the sampling offset, and outputs a prediction result map of target refined contour points and detection confidence with 9×2+2 channels, where 9 represents the 9 contour points representing the target, the first 2 represents the sum coordinates of the contour points, and the second 2 represents the detection confidence.

[0042] In general, the target representation form of the contour point is applied to three positions of the task head: the first is the initialization position of the contour point, that is, the anchor point set according to the resolution of the basic feature map, which can be considered as all the contour points of the target distributed in the same position. Here, the FPN structure distributes targets of different scales to different levels, alleviating the missed detection caused by "one anchor point can only be responsible for checking one target" and "multiple targets fall on the same anchor point"; the second position is the intermediate result of the first stage output, combined with the preset anchor point, the target preliminary contour point can be obtained; the third position is the prediction result of the second stage output, combined with the target preliminary contour point, the target refined contour point can be obtained. These three positions are based on the formula The refinement is performed in sequence. Indicates the target contour point at the previous position, It represents the predicted contour point offset at the current position, Rr is the position coordinate of the contour point after refinement, k is the number of the target contour point at the previous position, and there are n target contour points in total.

[0043] For the recognition task head, the workflow is as follows: the basic feature map passes through the basic module with the same structure as the above to obtain the recognition-specific feature map; then, a nonlinearly activated ordinary convolution component and a linearly activated ordinary convolution component are directly used to output the apparent feature encoding and classification result map with 128+n channels, where 128 represents the dimension of the apparent feature vector and indicates the number of categories.

[0044] Both the detection and recognition task heads use a basic module for generating dedicated feature maps, which can decouple the basic feature maps to a certain extent and alleviate the conflict problem of multi-task parallel training. At the same time, in order to solve the problem of apparent feature alignment, a feature enhancement component based on the spatial attention mechanism and a DCN component based on adaptive shape alignment are used to form the apparent feature encoding task head, such as Figure 3As shown in the figure, which is part of the detection task head, the feature enhancement component processes the basic feature map through pooling and ordinary convolution operations, and then generates a feature weight map using the Sigmoid function. The weight map is multiplied element-wise with the basic feature map to adjust the feature values ​​in the basic feature map, enhance relevant features, suppress irrelevant features, and assist the DCN component to better aggregate the target appearance features.

[0045] 5) Based on the algorithm model of bounding box target representation, in the task execution network part, a feature enhancement component based on spatial attention mechanism (SAM) and a DCN component with adaptive shape alignment are used to form the appearance feature encoding task head. SAM is used to enhance relevant features and suppress irrelevant features of the basic feature map input to the appearance feature encoding task head, so as to assist the subsequent DCN component to better complete the target appearance feature aggregation. The workflow of the feature enhancement component is to use pooling operations and ordinary convolution operations on the basic feature map, and then use the Sigmoid function to obtain the feature weight map, and attach the weight map to the basic feature map by element-wise multiplication, so as to adjust the numerical value of the feature value in the basic feature map.

[0046] 6) For the post-processing part, the detection target is still obtained through confidence and NMS screening, and then the center mapping selection (CMS) method shown in Algorithm 3.2 is used to obtain the apparent features for the detection target. At the same time, the CMS method is further improved to obtain the adaptive apparent feature selection method (Adaptive Apparent Feature Selection, AAFS) based on contour points, as shown in Figure 2 As shown in the figure, the AAFS method is based on the CMS method and combines the contour point information to operate: First, all the contour points of the detected target are used according to the formula (where P center is the center point coordinate, is the i-th coordinate, and n is the number of points involved in the calculation. ) Calculate the coordinates of the center point of the target, and then divide the target into four parts using the horizontal and vertical coordinates of the center point as the dividing line; for each part, use the contour points distributed in it and the center point of the target to calculate the coordinates of the center point of the part according to the above formula; finally, map the center points of the four parts to the apparent feature vector map according to the corresponding downsampling multiples, and use the bilinear interpolation method to extract four eigenvectors from the apparent feature vector map. These four eigenvectors are calculated according to the formula (Among them, E total is the apparent feature vector of the final output, is the number of divided parts, and E i is the apparent eigenvector corresponding to the center point of the i-th part, ki is the number of contour points in the i-th part, is the number of contour points describing the target, that is, the weight of each feature vector is determined by the number of contour points distributed in this part, and the apparent feature vector representing the target can be obtained by fusion.

[0047] The following is a verification experiment of this embodiment:

[0048] The method is compared on two public datasets: MOT15 and MOT20. The statistics of these datasets are shown in Table 1, Table 2 and Table 3:

[0049] Table 1 Data represents the ablation experiment of the apparent feature optimization scheme (MOT20 training set)

[0050]

[0051] Table 2 Data represents the ablation experiment of the apparent feature optimization scheme (MOT15 training set)

[0052]

[0053] Table 3 Comparative experiment (MO15 training set)

[0054]

[0055] In summary, as shown in Table 1, Table 2, and Table 3, the present invention demonstrates a good tracking effect and achieves a considerable balance between performance and functional real-time performance.

[0056] The implementation details are as follows:

[0057] The server used for the implementation was installed with Linux (Ubuntu) operating system, and the framework used was CUDA10.2 and Pytorch1.7.0. Among them, NVIDIA RTX 2080Ti×1 was used as the computing resource. During the model training process, the optimizer used the SGD optimization algorithm. The number of training rounds was set to 30, and the initial learning rate was set to 0.001.

[0058] The numerical comparison results are as follows:

[0059] It can be seen that when testing on the MOT15 training set with low target density, whether the target representation form is a bounding box or a contour point, the use of the feature enhancement component SAM and the apparent feature alignment module DCN, MM / AAFS can improve the continuity of the tracking trajectory, indicating that these optimization methods have an improvement effect on the quality of the apparent features. Experiment 10 is the complete clue extraction part of this chapter. Compared with the baseline algorithm of Experiment 1, it improves the MOTA index representing the comprehensive tracking performance by about 13%, the IDF1 index representing the tracking coherence by about 3%, and the number of IDs occurrences by about 150 times. Therefore, it can be proved that the above optimization scheme has alleviated the apparent feature misalignment problem existing in the baseline algorithm based on the JDE paradigm to a certain extent, effectively improving the quality of the apparent feature vector output by the algorithm, thereby promoting the improvement of the overall tracking performance. However, by comparing Experiments 1 and 6, we can find that the performance of the baseline algorithm using contour points has declined compared to the baseline algorithm using bounding boxes. This is because the latter is slightly insufficient in target detection performance, but the target representation form of contour points is more detailed than that of bounding boxes, which is more conducive to the performance of appearance feature encoding tasks. Therefore, the optimization algorithm using contour points has a more significant improvement in tracking continuity.

[0060] The test results on the MOT20 training set with extremely high target density are different from those on the MOT15 training set. From the data in the table, it can be seen that the target detector based on contour points is not suitable for use in scenes with very crowded targets. However, even if the multi-target tracking algorithm based on contour points has a significant performance degradation, the optimization scheme for apparent features still plays a role to a certain extent, alleviating the performance degradation of such algorithms.

[0061] Visual comparison results are as follows Figure 5 As shown in the figure, the visualization effect of multi-pedestrian target tracking based on the clue extraction algorithm based on contour points and adaptive surface features on different open source data sets is shown. The scenes in the figure are all captured from the monitoring perspective to provide high-quality video data streams. It is worth noting that under the condition of relatively high pedestrian density, the target tracking algorithm of the present invention presents a higher quality tracking effect.

[0062] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the schemes disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the claims.

Claims

1. A clue extraction method based on contour points and adaptive surface features, characterized in that: include: Step 1: Get the original image data, extract features through the basic feature extraction network, generate the basic feature map, and provide the feature map basis for the subsequent network; Step 2: The basic feature map is processed by the task execution network to obtain the result map. The task execution network uses a feature enhancement component based on the spatial attention mechanism SAM and a DCN component based on adaptive shape alignment to form an apparent feature encoding task head. Step 3: For the result image, in the post-processing part, a mapping matching method is used to select the corresponding apparent feature vector for the detection target, and the apparent feature vector is matched with the detection target, and finally a result image with feature clues is obtained to complete the clue extraction.

2. The clue extraction method based on contour points and adaptive surface features according to claim 1 is characterized in that: In the step 1, the basic feature extraction network serves as the head of clue selection, including a backbone network. The backbone network performs preliminary processing on the original image data, reduces the feature dimension of the image, and improves the feature semantics of the image, that is, processes the original image data into effective feature data. When the original image data flows in the backbone network, feature maps of various granularities, semantics, and receptive fields are generated.

3. The clue extraction method based on contour points and adaptive surface features according to claim 2 is characterized in that: In the step 1, a feature fusion network is provided after the backbone network on the basic feature network. The feature fusion network adopts a feature pyramid network structure, and layers are provided on the 8-fold, 16-fold, and 32-fold downsampled feature maps of the backbone network for feature fusion. Finally, three fused basic feature maps are output for subsequent task execution networks, wherein a small-resolution feature map with a large receptive field and high-level semantic features is used to detect small targets, and conversely, a large-resolution feature map is used to detect large targets.

4. The clue extraction method based on contour points and adaptive surface features according to claim 1, characterized in that: In the task execution network of step 2, the basic feature map is converted into a special feature map of the task head after passing through a task head composed of three non-linearly activated ordinary convolution components; the special feature map passes through a non-linearly activated ordinary convolution component and a linearly activated ordinary convolution component to obtain an intermediate result map with 2×9 channels as output, which can not only be converted into preliminary contour points of the target, but also used as a sampling offset of the deformable convolution component in the second stage.

5. The clue extraction method based on contour points and adaptive surface features according to claim 4 is characterized in that: After obtaining the intermediate result map, the DCN component performs a convolution operation with the dedicated feature map as input and the intermediate result as the sampling offset, and outputs a prediction result map of target refined contour points and detection confidence with 9×2+2 channels, where 9 represents the 9 contour points representing the target, the first 2 represents the sum coordinates of the contour points, and the second 2 represents the detection confidence.

6. The clue extraction method based on contour points and adaptive surface features according to claim 1, characterized in that: In the post-processing part, the detection target is still obtained through confidence and NMS screening, and then the center point mapping selection CMS method is used to obtain the apparent features for the detection target. At the same time, the CMS method is further improved to obtain the adaptive apparent feature selection method AAFS based on contour points to obtain the center point coordinates of the detection target; finally, the center point is mapped to the apparent feature vector map according to the corresponding downsampling multiple, and four feature vectors are extracted from the apparent feature vector map using bilinear interpolation. These four feature vectors are obtained according to the following formula: Among them, E total is the apparent feature vector of the final output, is the number of divided parts, and E i is the apparent eigenvector corresponding to the center point of the i-th part, k i is the number of contour points in the i-th part, K ​​is the number of contour points describing the target, that is, the weight of each feature vector is determined by the number of contour points distributed in this part; The four feature vectors are fused to obtain the apparent feature vector representing the target, which is then stored in the output set. The unmodified bounding box set and the updated feature vector set are returned, and finally a result map with feature clues is obtained to complete clue extraction.

7. The clue extraction method based on contour points and adaptive surface features according to claim 6 is characterized in that: The AAFS method is an adaptive surface feature selection method that combines contour point information based on the center point mapping selection CMS method: First, all contour points of the detection target are used according to the following formula: Where P center is the center point coordinate, (P i x ,P i y ) is the i-th coordinate, and n is the number of points involved in the calculation; The coordinates of the center point of the target are calculated, and then the target is divided into four parts using the horizontal and vertical coordinates of the center point as dividing lines; for each part, the coordinates of the center point of the part are calculated using the contour points distributed therein and the center point of the target according to the above formula.

Images