A single-target tracking method, device and equipment based on a SIMD instruction DSP and a medium

Abstract

The application discloses a single-target tracking method and device based on a SIMD instruction DSP, an apparatus and a medium. The method comprises the following steps: generating interpolation mapping relationship data required for image scaling by a CPU core; transmitting target region image data and the interpolation mapping relationship data to a DSP core by using DMA; and completing image scaling and target feature extraction based on a SIMD instruction by the DSP core. The method can fully utilize the vector parallel processing capability of the DSP, reduce the calculation time consumption of feature extraction, scaling processing and subsequent target detection process in the KCF algorithm; meanwhile, the method can improve the tracking stability in the target scale change scene by performing target detection on multiple candidate detection regions of different scales in the subsequent frame and determining the current frame target position and updating the target tracking model according to the maximum response result, and can reduce the delay and congestion caused by cross-core or cross-chip data interaction, and improve the real-time performance and engineering applicability of the single-target tracking method on the embedded DSP platform.

Description

Technical Field

[0001] This invention relates to the field of automatic tracking technology in image processing, specifically to a single-target tracking method, apparatus, device, and medium based on SIMD instruction DSP. Background Technology

[0002] Single-object tracking refers to a technique that provides an initial bounding box for a target in the first frame of a video and continuously locates the same target in subsequent image frames. The KCF (Kernel Correlation Filter) algorithm is a typical traditional correlation filter tracking algorithm. It achieves target model training and fast detection through circulant matrix, ridge regression, and fast Fourier transform. It has advantages such as relatively low computational cost, good real-time performance, no need for deep learning model pre-training, and ease of embedded deployment.

[0003] Existing KCF target tracking acceleration solutions typically rely on multi-core DSPs, FPGAs, or FPGA+DSP heterogeneous platforms. For example, multi-core DSPs can be used to divide tasks such as image preprocessing, feature extraction, and response calculation, or FPGAs can be used to perform HOG feature extraction and FFT acceleration, with the DSP handling subsequent tracking calculations. However, these solutions are often heavily dependent on specific hardware platforms, resulting in high portability costs. Furthermore, data interaction between multi-core processors or cross-chip transmission can easily introduce latency, potentially leading to processing congestion or frame drops in high frame rate scenarios.

[0004] In addition, while high-end multi-core DSPs have strong computing power, they also have high cost and power consumption; low-power DSP platforms are difficult to directly meet the real-time requirements of intensive operations such as image scaling, HOG / LAB feature extraction, and FFT frequency domain calculation in the KCF algorithm. Summary of the Invention

[0005] The main objective of this invention is to propose a single-target tracking method, apparatus, device, and medium based on SIMD instruction DSP, aiming to at least solve the technical problems mentioned in related technologies.

[0006] In a first aspect, the present invention provides a single-target tracking method based on SIMD instruction DSP, applied to an image processing apparatus including a CPU core, a DSP core, and a DMA module. The single-target tracking method includes: Acquire the video image to be tracked, and determine the target region based on the target bounding box specified in the first frame image; Based on the target scaling size corresponding to the target region, obtain the interpolation mapping relationship data required by the CPU core to generate bilinear interpolation; The image data and interpolation mapping relationship data of the target area are transferred to the DSP core through the DMA module, and the DSP core performs scaling processing on the target area based on SIMD instructions to obtain the scaled target image block. The DSP core obtains target features based on SIMD instructions; The target tracking model is obtained by training the model based on the target features; In subsequent frame images, target detection is performed on multiple candidate detection regions at different scales based on the target tracking model to determine the target tracking result of the current frame; The target tracking model is updated based on the target tracking results of the current frame, and the target tracking results of the current frame are used as the pre-detection boxes for the next frame.

[0007] In a second aspect, the present invention provides a single-target tracking device, including a CPU core, a DSP core, and a DMA module; The CPU core acquires the video image to be tracked and determines the target region based on the target box specified in the first frame image; The CPU core generates the interpolation mapping data required for bilinear interpolation based on the target scaling size corresponding to the target region. The DMA module moves the image data of the target region and the interpolation mapping relationship data to the storage space accessible by the DSP core, and the DSP core performs scaling processing on the target region based on SIMD instructions to obtain the scaled target image block; The DSP core extracts features from the scaled target image patch based on SIMD instructions to obtain target features; The DSP core trains a model based on the target features to obtain a target tracking model; The CPU core and the DSP core work together in subsequent frames to perform target detection on multiple candidate detection regions of different scales based on the target tracking model, and determine the target tracking result of the current frame; wherein, the DSP core is used to perform feature extraction and frequency domain calculation, and the CPU core is used to perform maximum response value search and determine the target tracking result; The DSP core updates the target tracking model based on the target features corresponding to the target tracking result of the current frame, and the CPU core uses the target box corresponding to the target tracking result of the current frame as the pre-detection box for the next frame.

[0008] A third aspect of the present invention provides an electronic device, the electronic device including a memory, a processor, and a bus; the bus is used to realize the connection and communication between the memory and the processor; the processor is used to execute a computer program stored in the memory; when the processor executes the computer program, it implements the steps of the single target tracking method based on SIMD instruction DSP of the first aspect.

[0009] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the single-target tracking method based on SIMD instruction DSP of the first aspect.

[0010] This invention discloses a single-target tracking method, apparatus, device, and medium based on SIMD instruction DSP. It generates interpolation mapping data required for image scaling via a CPU core and uses DMA to transmit the target region image data and interpolation mapping data to the DSP core. The DSP core then performs image scaling and target feature extraction based on SIMD instructions. This fully utilizes the vector parallel processing capabilities of the DSP, reducing the computational time of feature extraction, scaling, and subsequent target detection in the KCF algorithm. Simultaneously, by performing target detection on multiple candidate detection regions of different scales in subsequent frames and determining the target position in the current frame and updating the target tracking model based on the maximum response result, it improves tracking stability in scenarios with changing target scales. Compared to solutions relying on multi-core DSPs or FPGA+DSP heterogeneous platforms, this application reduces latency and congestion caused by cross-core or cross-chip data interaction, lowers hardware platform dependence, power consumption, and deployment costs, and improves the real-time performance and engineering applicability of the single-target tracking method on embedded DSP platforms. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A schematic diagram of the basic process of the single-target tracking method based on SIMD instruction DSP provided in the embodiments of this application; Figure 2 This is a schematic diagram of the overall algorithm flow corresponding to the single target tracking method based on SIMD instruction DSP provided in the embodiments of this application; Figure 3 This is a schematic diagram of the HOG feature extraction process in an embodiment of this application; Figure 4 This is a schematic diagram of the HOG feature and Lab feature fusion extraction process provided in the embodiments of this application; Figure 5 This is a schematic diagram of the neighborhood relationship for bilinear interpolation provided in an embodiment of this application; Figure 6 This is a schematic diagram of image segmentation provided in an embodiment of this application; Figure 7 HOG histogram of a single cell provided in the embodiments of this application; Figure 8 This is a schematic diagram of the HOG gradient normalization neighborhood provided in an embodiment of this application; Figure 9 This is a schematic diagram of histogram interpolation provided in an embodiment of this application; Figure 10 Lab histogram of a single cell provided in an embodiment of this application; Figure 11 This is a schematic diagram of the module connections of the single-target tracking device provided in the embodiments of this application; Figure 12 This is a schematic diagram of the internal structure and connections of the electronic device provided in the embodiments of this application.

[0013] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0014] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0016] In the description of the embodiments of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The term "multiple" means two or more, unless otherwise explicitly specified. The term "comprising" indicates the presence of the described feature, whole, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or sets thereof. The term "and / or" describes the relationship between related objects, indicating that three relationships may exist. For example, A and / or B may include three cases: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the related objects before and after are in an "or" relationship.

[0017] Unless otherwise defined, all technical terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art; the terms used in the embodiments of this application are for the purpose of describing specific embodiments only and are not intended to limit this application; the terms "comprising" and "having" and any variations thereof in the specification, claims and foregoing description of the drawings of this application are intended to cover non-exclusive inclusion.

[0018] Furthermore, terms such as "exemplary," "for example," and "optional" are used to indicate illustrative purposes. Any technical solution described by the above terms in the embodiments of this application should not be construed as being more preferred or advantageous than other technical solutions. Specifically, these terms are intended to present the relevant technical concepts in terms of specific implementation methods.

[0019] The relevant technologies are described below: Single-object tracking refers to a technique that, given an initial bounding box for a target in the first frame of a video, continuously locates the same target in every subsequent frame. Single-object tracking typically tracks only one specified target, without needing to distinguish other targets of the same category. It is widely used in scenarios such as video surveillance, unmanned equipment, robot vision, intelligent driving assistance, and embedded vision processing.

[0020] Kernel Correlation Filter (KCF) is a typical traditional correlation filter tracking algorithm. This algorithm transforms the target tracking problem into a classification problem between foreground targets and background regions, and trains the classifier using ridge regression. Since ridge regression typically involves matrix operations, direct computation leads to high computational complexity. The KCF algorithm constructs cyclically shifted samples and utilizes the property that cyclic matrices can be diagonalized by Fourier transform, shifting a large amount of computation to the frequency domain, thus significantly reducing the computational load and improving target detection speed. Furthermore, the KCF algorithm can introduce a Gaussian kernel function to enhance its ability to distinguish nonlinear target features.

[0021] Compared to deep learning tracking algorithms such as SiamFC, SiamRPN, ATOM, DiMP, and OSTrack, the KCF algorithm does not rely on large-scale pre-trained models and typically does not require GPU support. It boasts advantages such as small model size, low computational cost, fast startup speed, simple implementation, ease of debugging, and convenient embedded deployment. Therefore, the KCF algorithm still holds significant engineering application value in embedded vision devices with high requirements for power consumption, cost, and real-time performance.

[0022] However, the KCF algorithm still involves several computationally intensive steps in actual operation, such as image scaling, HOG feature extraction, LAB feature extraction, Gaussian kernel correlation calculation, fast Fourier transform, and response map search. Especially in high frame rate video or multi-scale detection scenarios, if the above calculations are performed serially by a general-purpose CPU, it can easily lead to long processing times for a single frame, making it difficult to meet real-time tracking requirements.

[0023] In existing technologies, some KCF acceleration solutions employ multi-core DSP platforms. For example, the main core performs image preprocessing, while multiple slave cores perform KCF response calculations in parallel. Other solutions use a heterogeneous architecture of FPGA and DSP, where the FPGA performs image acquisition, HOG feature extraction, or FFT acceleration, and the DSP performs subsequent correlation filtering calculations. While these solutions can improve computing speed to some extent, they typically suffer from strong hardware platform dependence, high portability costs, complex inter-core data interaction, and significant cross-chip transmission latency. In high frame rate scenarios, data congestion or frame drops may also occur.

[0024] Furthermore, while high-end multi-core DSPs offer strong computing power, they are costly and consume significant power under full load. Low-power DSP platforms, though more suitable for embedded devices, still struggle to meet the real-time requirements of the KCF algorithm if their SIMD instruction parallel processing capabilities are not fully utilized. Therefore, a single-target tracking method for SIMD instruction DSP platforms is urgently needed. This method aims to improve the real-time performance, energy efficiency, and embedded deployment applicability of KCF single-target tracking by rationally dividing the computational tasks between CPU and DSP cores and optimizing key processes such as image scaling, feature extraction, and frequency domain correlation calculations through parallelization.

[0025] To address at least one of the technical problems in related technologies, such as the large computational load and insufficient real-time performance of the KCF single-target tracking algorithm on embedded platforms, and the strong platform dependence, high data interaction latency, and high porting cost of heterogeneous solutions such as multi-core DSP or FPGA+DSP, this application provides a single-target tracking method based on SIMD instruction DSP.

[0026] This single-target tracking method can be applied to image processing devices including CPU cores and DSP cores, such as embedded vision processing devices, video surveillance devices, unmanned equipment vision tracking systems, robot vision systems, or vehicle-mounted vision processing systems.

[0027] Please see Figure 1 The single-target tracking method includes the following steps: Step S101: Acquire the video image to be tracked, and determine the target area based on the target box specified in the first frame image.

[0028] Specifically, the video images to be tracked include multiple frames arranged in chronological order. These can be a sequence of video frames acquired in real time by an image acquisition device, or a pre-stored sequence of video images. The first frame is used to initialize target tracking.

[0029] In step S101, the execution entity is the CPU core, which acquires the video image to be tracked and obtains the specified initial target bounding box in the first frame image. The initial target bounding box can be determined by the user based on the position of the target in the first frame image, or it can be provided by the target detection module based on the detection results of the first frame image.

[0030] Based on the position and size of the initial bounding box in the first frame, the target region containing the target to be tracked is determined, and this target region is used as the initial input for subsequent image scaling, target feature extraction, and target tracking model training. After the target tracking model is trained, for subsequent frames in the video image to be tracked, the position of the target to be tracked is determined frame by frame based on the target tracking model, and the corresponding target tracking results are obtained.

[0031] Step S102: Based on the target scaling size corresponding to the target area, obtain the interpolation mapping relationship data required for the CPU core to generate bilinear interpolation.

[0032] Specifically, in step S102, the execution entity (CPU core) determines the target scaling size corresponding to the target region from multiple preset scaling sizes based on the aspect ratio of the target bounding box specified in the first frame image. In this embodiment, the preset scaling sizes include 24×24, 24×16, and 24×8. Interpolation mapping relationship data is used to represent the positional correspondence between pixels in the scaled image and pixels in the original image; for example, it may include a lookup table of adjacent pixel positions required for bilinear interpolation. By pre-generating the interpolation mapping relationship data by the CPU core, the addressing computation load of the DSP core during the scaling process can be reduced.

[0033] It should be noted that bilinear interpolation is an image scaling method that estimates the pixel value of a target location based on the pixel values ​​of its four neighboring pixels. "Bilinear" means that linear interpolation is performed once in each of the two directions.

[0034] In one alternative implementation, the CPU core calculates the mapping position of each target pixel in the scaled target image block in the target area before scaling, based on the original size of the target area and the target scaling size.

[0035] For the target pixel P(i,j) in the scaled target image block, the horizontal scaling ratio and the vertical scaling ratio are determined according to the following formulas:

[0036]

[0037] in, and These represent the width and height of the target area before scaling, respectively. and These represent the width and height of the scaled target image patch, respectively. and These represent the horizontal scaling ratio and the vertical scaling ratio, respectively.

[0038] Based on the horizontal and vertical scaling ratios, the floating-point position (x,y) of the target pixel P(i,j) in the target region before scaling is determined according to the following formula:

[0039]

[0040] Based on the image coordinate definition shown in the attached figure, determine the adjacent x and y coordinates of the floating-point position (x, y) in the target area before scaling: ,

[0041] ,

[0042] in, This represents the integer x-coordinate to the left of or coinciding with the floating-point x-coordinate. This represents the integer x-coordinate to the right of or coinciding with the floating-point x-coordinate. This represents the integer y-coordinate obtained by rounding up the floating-point y-coordinate. This represents the integer y-coordinate obtained by rounding down the floating-point y-coordinate.

[0043] That is, (x1,y1), (x1,y2), (x2,y1), and (x2,y2) together determine the positions of the four neighboring pixels around the floating-point position (x,y) that participate in bilinear interpolation. This determines the four neighboring pixels participating in bilinear interpolation. Q 11 (x1,y1),Q 12 (x1,y2),Q 21 (x2,y1),Q 22 (x2, y2) Furthermore, the CPU core calculates the corresponding x1, x2, y1, and y2 for each scaled target pixel, and stores the obtained x1, x2, y1, and y2 as a first horizontal position lookup table, a second horizontal position lookup table, a first vertical position lookup table, and a second vertical position lookup table, respectively. The four position lookup tables constitute the interpolation mapping relationship data.

[0044] In subsequent scaling processing, the DSP core obtains the pixel values ​​of four adjacent pixels based on the interpolation mapping data, and first scales the pixel Q along the horizontal direction. 11 and pixel Q 21 Pixel Q 12 and pixel Q 22 Interpolation is performed to obtain intermediate interpolation results R1 and R2:

[0045]

[0046] in, This represents the pixel value at the intermediate interpolation point R1. This represents the pixel value at the intermediate interpolation point R2; Then, interpolate along the vertical axis at the intermediate interpolation points R1 and R2 to obtain the pixel value of the target pixel P:

[0047] This implementation pre-calculates the positional mapping relationship between the scaled target pixel and its neighboring pixels in the original target region using the CPU core. This allows the DSP core to directly retrieve the pixel data for bilinear interpolation based on four position lookup tables, reducing the computational load required for pixel coordinate transformation and neighbor pixel position calculation. Furthermore, this implementation uses SIMD instructions to load and process multiple input pixel values ​​and their corresponding position lookup data in parallel, enabling the DSP core to simultaneously perform bilinear interpolation operations on multiple target pixels, thereby improving the scaling speed of the target region.

[0048] In step S103, the image data and interpolation mapping relationship data of the target area are transferred to the DSP core through the DMA module. The DSP core performs scaling processing on the target area based on SIMD instructions to obtain the scaled target image block.

[0049] Specifically, in step S103, one execution unit (DMA module) is used to transfer image data and interpolation mapping relationship data of the target area to reduce the processing overhead caused by the CPU core participating in data transfer. Another execution unit (DSP core) determines the positions of adjacent pixels corresponding to each pixel to be interpolated according to the interpolation mapping relationship data, reads the pixel value data corresponding to multiple adjacent pixel positions, and loads the pixel value data and the corresponding interpolation calculation data into the vector register. Using SIMD instructions, weighted multiplication and summation operations are performed in parallel on multiple pixels to be interpolated to obtain the scaled target image block.

[0050] It should be understood that a target image patch is a local image data obtained by cropping the target region from a video image based on the target bounding box and scaling the target region to the target's scaled size. The target image patch contains image information about the target to be tracked and its surroundings, which is used for subsequent target feature extraction and target tracking model processing.

[0051] Step S104: The DSP core determines the target features based on the SIMD instructions.

[0052] Specifically, the execution unit (DSP core) can extract features from the scaled target image patch based on SIMD instructions to obtain target features. These target features can include HOG features and LAB features. By performing gradient calculation, orientation histogram statistics, feature normalization, and color feature statistics in parallel on the DSP core, feature extraction efficiency can be improved.

[0053] It should be noted that SIMD instructions enable the DSP core to process multiple data elements in parallel within the same instruction cycle. These multiple data elements include at least one of multiple pixel values, multiple gradient values, multiple directional dimensional feature values, or multiple color feature values. By using SIMD instructions to perform feature extraction operations in parallel, the data throughput of the DSP core can be improved, and the feature extraction time for a single frame of image can be reduced.

[0054] Optionally, the execution entity (DSP core) can perform feature extraction on the scaled target image block based on SIMD instructions to obtain target features including: the DSP core performs parallel gradient calculation on multiple pixels in the target image block based on SIMD instructions to determine the gradient magnitude and gradient direction of each pixel, and performs image unit division, gradient direction histogram statistics, neighborhood normalization and dimensionality reduction on the target image block to obtain HOG features; the DSP core converts the target image block from BGR color space to Lab color space based on SIMD instructions, calculates the distance between each pixel and multiple preset Lab features, and performs color feature histogram statistics on image units to obtain Lab features; the HOG features and Lab features are concatenated to obtain the target features.

[0055] Step S105: Train the model based on the target features to obtain the target tracking model.

[0056] Specifically, the DSP core, acting as the execution entity, can train a KCF model based on target features to obtain a target tracking model. The model training process may include constructing cyclic shift samples, calculating Gaussian kernel correlation, and solving the ridge regression model in the frequency domain. By converting the correlation calculation to frequency domain processing, the computational complexity of the model training process can be reduced.

[0057] Step S106: In subsequent frame images, target detection is performed on multiple candidate detection regions of different scales based on the target tracking model to determine the target tracking result of the current frame.

[0058] Specifically, the CPU core, acting as the execution entity, can generate multiple candidate detection regions of different scales based on the target bounding box of the previous frame in subsequent frames. These can be candidate regions of the same size as, slightly larger than, or slightly smaller than the target bounding box of the previous frame. The image data of each candidate detection region and its corresponding interpolation mapping data are transferred to the storage space accessible by the DSP core via the DMA module. Finally, the DSP core performs scaling and feature extraction on each candidate detection region based on SIMD instructions, obtaining the candidate region features corresponding to each region. For example, the candidate region features include the response map corresponding to each candidate detection region, and the candidate detection region corresponding to the largest response value is ultimately determined as the target tracking result for the current frame.

[0059] Step S107: Update the target tracking model based on the target tracking result of the current frame, and use the target tracking result of the current frame as the pre-detection box of the next frame.

[0060] Specifically, after determining the target tracking result of the current frame, the DSP core updates the target tracking model according to the target features corresponding to the target region of the current frame, so that the target tracking model can adapt to changes in the target's appearance, scale, or pose; the CPU core sets the target box corresponding to the target tracking result of the current frame as the pre-detection box of the next frame, so as to quickly determine multiple candidate detection regions in the next frame image.

[0061] As can be seen, the embodiments of this application generate interpolation mapping relationship data through the CPU core, and transmit the image data and interpolation mapping relationship data to the DSP core through DMA. The DSP core then performs computationally intensive operations such as image scaling and feature extraction based on SIMD instructions. This fully utilizes the parallel computing capabilities of the DSP core, reduces the computational burden of the CPU core and the data interaction latency, and improves the real-time performance and deployment efficiency of KCF single target tracking on the embedded platform.

[0062] In an optional implementation, step S104: obtaining target features obtained by the DSP core based on SIMD instructions includes at least: controlling the DSP core to extract features from the scaled target image block based on SIMD instructions to obtain target features; wherein, the target features include HOG features and LAB features.

[0063] It should be noted that HOG features are used to characterize the edge direction, contour structure, and texture distribution of local regions in a target image patch, thereby improving the adaptability of target appearance features to changes in illumination, local deformation, and background interference. LAB features are used to characterize the color distribution and texture information of each local region in a target image patch, and together with HOG features, constitute target features to improve the adaptability to changes in target appearance during target tracking.

[0064] Please see Figures 2 to 9 When extracting HOG features, the DSP core performs parallel processing on the scaled target image patch based on SIMD instructions.

[0065] Specifically, a 3×3 Sobel convolution operator is used to filter the target image patch to obtain the gradient Gx in the horizontal direction and the gradient Gy in the vertical direction for each pixel. Then, the corresponding gradient magnitude G and gradient direction θ are calculated based on the gradients in the horizontal and vertical directions.

[0066]

[0067] For a target image block containing multiple color channels, the gradient magnitude and gradient direction of the corresponding pixel in each color channel are calculated, and the maximum gradient magnitude and the gradient direction corresponding to the maximum gradient magnitude in each color channel are determined as the gradient magnitude and gradient direction of the pixel.

[0068] Furthermore, the gradient direction of each pixel is quantized to 9 directional dimensions within the range of 0° to 180° and 18 directional dimensions within the range of 0° to 360°, thereby obtaining 27-dimensional gradient direction data corresponding to each pixel (e.g., Figure 2 (As shown). The target image patch is divided into multiple image units (cells), each cell consisting of 4×4 pixels (e.g., ...). Figure 6 As shown in the figure). Based on the gradient direction and gradient magnitude of each pixel, the gradient magnitude is weighted and distributed to the corresponding direction dimension, and the weighted result is interpolated and distributed to adjacent cells; the gradient magnitudes of the same direction dimension in each cell are accumulated to obtain the 27-dimensional gradient direction histogram corresponding to that cell (as shown in the figure). Figure 7 (As shown).

[0069] Subsequently, the 27-dimensional gradient orientation histogram is normalized using the lower-right, upper-right, lower-left, and upper-left neighbors, including the current cell. Each neighborhood comprises four adjacent cells, and the normalization factor is determined based on the sum of the squares of the first nine dimensions of the gradient orientation histograms of the four cells. After four normalizations, each cell yields four sets of 27-dimensional normalized results, forming a 108-dimensional descriptor (e.g., ...). Figure 8 (As shown).

[0070] Furthermore, such as Figure 9 As shown, PCA dimensionality reduction is performed on the 108-dimensional descriptor to obtain 31-dimensional HOG features. Specifically, the 18-dimensional directed gradient descriptors corresponding to the four sets of normalized results are summed and multiplied by the first scaling factor to obtain 18-dimensional directed gradient features; the 9-dimensional undirected gradient descriptors corresponding to the four sets of normalized results are summed and multiplied by the first scaling factor to obtain 9-dimensional undirected gradient features; and the 18-dimensional directed gradient descriptors from the four sets of normalized results are accumulated and multiplied by the second scaling factor to obtain 4-dimensional texture features.

[0071] In one alternative implementation, the first scaling factor is: The second scaling factor is: .

[0072] Therefore, by combining 18-dimensional directed gradient features, 9-dimensional undirected gradient features, and 4-dimensional texture features, a 31-dimensional HOG feature is obtained.

[0073] In one alternative implementation, the DSP core processes 16 image blocks in parallel at a time based on SIMD instructions.

[0074] Please see Figure 2 , Figure 10When extracting HOG features, the DSP core controls the scaling target image patch to be converted from BGR color space to Lab color space based on SIMD instructions, and calculates the distance between the Lab color value of each pixel and 15 preset Lab cluster centers.

[0075] Specifically, the 15 pre-defined Lab cluster centers can be represented as follows: ; Each cluster center consists of one luminance component and two color components. For a pixel with a Lab color value of (l, a, b), calculate the distance between that pixel and the nth cluster center: ; Where n is the cluster center number, and n ranges from 0 to 14; , , and represent the three Lab components of the nth cluster center, respectively.

[0076] For each pixel, the minimum distance is determined from 15 distances, and the cluster center number corresponding to the minimum distance is determined as the Lab feature dimension number of that pixel. Subsequently, the Lab feature dimension numbers corresponding to each pixel are counted on an image unit cell basis to generate a Lab feature histogram containing 15 histogram intervals.

[0077] Arrange the 15-dimensional Lab feature histograms corresponding to each cell sequentially to obtain a size of Lab feature descriptor.

[0078] In an optional implementation, step S105: training the model based on the target features to obtain a target tracking model includes at least: training the KCF model based on the target features to obtain a target tracking model; wherein, the KCF model training includes constructing cyclic shift samples, calculating Gaussian kernel correlation, and solving the ridge regression model in the frequency domain.

[0079] Specifically, the target features corresponding to the target region in the first frame are used as training samples x. Cosine window weighting is applied to the training samples to reduce the boundary effect generated during the cyclic shift. A Gaussian label matrix y is generated with the target position corresponding to the training sample as the center. Multiple training samples are constructed by cyclic shifting the training samples. Among them, the sample corresponding to the target position is used as the positive sample, and the cyclically shifted sample that deviates from the target position is used as the negative sample.

[0080] For the training sample x and the feature samples to be calculated The kernel correlation between the two was calculated using a Gaussian kernel:

[0081] in, Represents the relationship between training sample x and feature sample x. Gaussian kernel correlation results between them; This indicates the number of elements in the training sample; represents the Gaussian kernel bandwidth parameter; c represents the feature dimension number of the target feature; The complex conjugate of the frequency domain data of the c-th template feature channel; exp( ): Performs exponential operation on each element within the parentheses; ⊙ indicates element-wise multiplication between corresponding frequency domain data; This represents the inverse fast Fourier transform.

[0082] When calculating the correlation results of the Gaussian kernel, the DSP kernel performs fast Fourier transform on the feature data of multiple feature dimensions respectively, and performs conjugate multiplication and accumulation on the frequency domain results of each feature dimension to transform the correlation operation based on cyclic shifted samples to the frequency domain.

[0083] In one specific implementation, the target features include 31-dimensional HOG features and 15-dimensional Lab features, totaling 46 feature dimensions; the feature map size of each feature dimension is 24×n, where n is 24, 16, or 8. The DSP core processes 16 feature dimensions in parallel at a time based on SIMD instructions and performs frequency domain calculations using floating-point fast Fourier transform.

[0084] Furthermore, Fast Fourier Transform (FFT) operations are configured according to the feature map size. The FFT for power-fourths lengths is optimized, while mixed-basis FFT is used for non-power-fourths lengths. During model initialization, the rotation factors required for the FFT are pre-calculated and cached, allowing the DSP core to directly read the rotation factors during subsequent model training and object detection.

[0085] After obtaining the Gaussian kernel correlation results, perform a fast Fourier transform on the Gaussian kernel correlation results and solve the ridge regression model in the frequency domain:

[0086] in, Represents the classifier coefficients in the frequency domain; This represents the Fourier transform result of the Gaussian label matrix y; λ represents the Fourier transform result of the Gaussian kernel correlation between the training sample x and itself; λ represents the ridge regression regularization parameter; division is the division of corresponding elements in the frequency domain.

[0087] Therefore, the template features and frequency domain classifier coefficients corresponding to the training samples are used as the target tracking model for target detection in subsequent frame images.

[0088] When updating the target tracking model based on the target tracking results of the current frame, the current model parameters and the new model parameters can be updated with weights according to a preset learning rate:

[0089]

[0090] in, This indicates that the model updates the learning rate; and These represent the frequency domain classifier coefficients before and after the update, respectively; This represents the frequency domain classifier coefficients obtained based on the target features of the current frame; and These represent the template features before and after the update, respectively. This indicates the target features corresponding to the target tracking result in the current frame.

[0091] In an optional implementation, step S106: Target detection is performed on multiple candidate detection regions of different scales based on the target tracking model in subsequent frame images to determine the target tracking result of the current frame. This includes at least: generating multiple candidate detection regions of different scales based on the target bounding box of the previous frame in subsequent frame images, and scaling and feature extraction are performed on each candidate detection region to obtain multiple candidate region features; frequency domain calculation is performed on the multiple candidate region features according to the target tracking model to obtain the response map corresponding to each candidate detection region; the candidate detection region corresponding to the maximum response value is determined from the multiple response maps, and this candidate detection region is determined as the target tracking result of the current frame.

[0092] Specifically, based on the target bounding box of the previous frame, a first candidate detection region with the same scale as the target bounding box of the previous frame, a second candidate detection region with a larger scale than the target bounding box of the previous frame, and a third candidate detection region with a smaller scale than the target bounding box of the previous frame can be generated. Each candidate detection region is then scaled, and the corresponding HOG features and Lab features are extracted to obtain the candidate region features.

[0093] For any candidate detection region, let x be the template feature in the target tracking model and z be the candidate region feature corresponding to the candidate detection region. Calculate the Gaussian kernel correlation between the template feature and the candidate region feature:

[0094] in, This represents the Gaussian kernel correlation result between template features and candidate region features; N represents the number of feature elements; c represents the Gaussian kernel bandwidth parameter; c represents the feature dimension. and These represent the Fourier transform results of the template feature and the candidate region feature in the c-th feature dimension, respectively; ) Represents complex conjugation; ⊙ represents element-wise multiplication; This represents the inverse fast Fourier transform.

[0095] Perform a Fourier transform on the Gaussian kernel correlation results and multiply them element-wise with the frequency domain classifier coefficients in the target tracking model to obtain the frequency domain response:

[0096] in, The Fourier transform result of the Gaussian kernel correlation result is represented. Represents the frequency domain classifier coefficients in the target tracking model. This represents the frequency domain response corresponding to the candidate detection region.

[0097] Perform an inverse Fourier transform on the frequency domain response and extract the real part of the inverse Fourier transform result to obtain the spatial domain response map:

[0098] Each response value in the response graph represents the confidence level that the target being tracked is present at the corresponding cyclic shift position. The maximum response value in the response graph is determined. and its corresponding location: Real( ) represents the real part extraction operation, used to extract the real part of the inverse Fourier transform result; f(z) represents the response value of the spatial domain response map corresponding to the candidate detection region;

[0099] The maximum response value corresponding to each candidate detection region is obtained, and the candidate detection region corresponding to the maximum response value with the largest value among multiple maximum response values ​​is determined as the target scale. The position of the maximum response value in the response map is determined as the position of the target in the candidate detection region, thereby obtaining the target box of the current frame.

[0100] In one specific implementation, the DSP core performs fast Fourier transforms, element-wise multiplication in the frequency domain, and inverse fast Fourier transforms in parallel across multiple feature dimensions based on SIMD instructions; the CPU core performs maximum response value search on each response map and can optimize sub-pixel positions based on response values ​​adjacent to the maximum response value. The target tracking result of the current frame includes the position and size of the target bounding box.

[0101] This application employs a collaborative processing approach using both CPU and DSP cores. The CPU core can be an SMCA53 core, and the DSP core can be a GMEC core. The CPU core is primarily used to generate the pixel location lookup table required for resizing and to perform maximum response value search and sub-pixel location optimization during fast detection. Other processing tasks, such as image scaling, feature extraction, model training, frequency domain calculation, and model updates, are mainly handled by the DSP core.

[0102] With arbitrary input image size and target region feature map sizes of 24×24, 24×16 and 24×8 respectively, the processing time for a single frame is shown in Table 1 below.

[0103] Table 1 Comparison of single-frame performance and time consumption

[0104] The corresponding algorithm processing frame rates are shown in Table 2 below.

[0105] Table 2 Performance Frame Rate Comparison

[0106] As shown in Tables 1 and 2, under the aforementioned test conditions, the A53 and GMEC collaborative processing solution achieves a single-frame processing speed approximately 28 to 34 times faster than the unoptimized TI C6678 solution. Specifically, it achieves the frame rates required for real-time video processing at feature map sizes of 24×24, 24×16, and 24×8.

[0107] Therefore, this application improves the computational efficiency of image scaling, target feature extraction, frequency domain model training, and fast detection by dividing tasks between CPU cores and DSP cores, DMA data transfer, and SIMD parallel processing of DSP cores, thereby realizing the real-time operation of the single target tracking algorithm.

[0108] Please see Figure 11 This application also provides a single-target tracking device, including a CPU core, a DSP core, a DMA module, an image scaling module, a feature extraction module, a model training module, a target detection module, and a model update module; The CPU core is used to acquire the video image to be tracked, determine the target region based on the target box specified in the first frame image, and generate the interpolation mapping relationship data required for bilinear interpolation based on the target scaling size corresponding to the target region. The DMA transfer module is used to transfer image data and interpolation mapping relationships of the target area to the DSP core; The image scaling module is used to scale the target area using the DSP core based on SIMD instructions to obtain a scaled target image block; The feature extraction module is used to extract features from scaled target image patches using the DSP core based on SIMD instructions to obtain target features; The model training module is used to train the model based on the target features to obtain the target tracking model; The target detection module is used to perform target detection on multiple candidate detection regions of different scales in subsequent frame images based on the target tracking model, and to determine the target tracking result of the current frame; The model update module is used to update the target tracking model based on the target tracking results of the current frame, and use the target tracking results of the current frame as the pre-detection box for the next frame.

[0109] Figure 12 An electronic device provided by an embodiment of the present invention is shown. This electronic device can be used to implement the single-target tracking method based on SIMD instruction DSP in any of the foregoing embodiments. The electronic device includes: The system includes a memory 1201, a processor 1202, a bus 1203, and a computer program stored in the memory 1201 and executable on the processor 1202. The memory 1201 and the processor 1202 are connected via the bus 1203. When the processor 1202 executes the computer program, it implements the single-target tracking method based on SIMD instruction DSP described in the foregoing embodiments. The number of processors can be one or more.

[0110] The memory 1201 can be a high-speed random access memory (RAM) or a non-volatile memory, such as a disk storage device. The memory 1201 is used to store executable program code, and the processor 1202 is coupled to the memory 1201.

[0111] Furthermore, embodiments of this application also provide a computer-readable storage medium, which may be disposed in the electronic device in the above embodiments, and the computer-readable storage medium may be a memory.

[0112] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the single-target tracking method based on SIMD instruction DSP in the aforementioned embodiments. Furthermore, the computer-readable storage medium can also be a USB flash drive, external hard drive, read-only memory (ROM), RAM, magnetic disk, or optical disk, or any other medium capable of storing program code.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0114] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0115] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0116] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, external hard drives, ROM, RAM, magnetic disks, or optical disks.

[0117] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0118] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0119] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A single-target tracking method based on SIMD instruction DSP, characterized in that, Single-target tracking methods, applied to image processing devices including CPU cores, DSP cores, and DMA modules, include: Acquire the video image to be tracked, and determine the target region based on the target bounding box specified in the first frame image; Based on the target scaling size corresponding to the target region, obtain the interpolation mapping relationship data required by the CPU core to generate bilinear interpolation; The image data and interpolation mapping relationship data of the target area are transferred to the DSP core through the DMA module, and the DSP core performs scaling processing on the target area based on SIMD instructions to obtain the scaled target image block. The DSP core obtains target features based on SIMD instructions; The target tracking model is obtained by training the model based on the target features; In subsequent frame images, target detection is performed on multiple candidate detection regions at different scales based on the target tracking model to determine the target tracking result of the current frame; The target tracking model is updated based on the target tracking results of the current frame, and the target tracking results of the current frame are used as the pre-detection boxes for the next frame.

2. The single-target tracking method based on SIMD instruction DSP as described in claim 1, characterized in that, Interpolation mapping data includes pixel position lookup tables; Based on the target scaling size corresponding to the target region, the interpolation mapping relationship data required for the CPU core to generate bilinear interpolation includes: Based on the aspect ratio of the target bounding box, the target scaling size corresponding to the target region is determined from multiple preset scaling sizes, and the pixel position lookup table required for bilinear interpolation generated by the CPU core is obtained.

3. The single-target tracking method based on SIMD instruction DSP as described in claim 2, characterized in that, The target features obtained by the DSP core based on SIMD instructions include: The DSP core extracts features from the scaled target image patch based on SIMD instructions to obtain target features; among them, target features include HOG features and LAB features.

4. The single-target tracking method based on SIMD instruction DSP as described in claim 3, characterized in that, The target tracking model is trained based on the target features, resulting in the following models: The target tracking model is obtained by training the KCF model based on the target features. The KCF model training includes constructing cyclic shift samples, calculating Gaussian kernel correlation, and solving the ridge regression model in the frequency domain.

5. The single-target tracking method based on SIMD instruction DSP as described in claim 4, characterized in that, In subsequent frames, target detection is performed on multiple candidate detection regions at different scales based on the target tracking model. The target tracking result for the current frame includes: In subsequent frames, multiple candidate detection regions of different scales are generated based on the target bounding box of the previous frame, and each candidate detection region is scaled and its features are extracted to obtain multiple candidate region features. Based on the target tracking model, frequency domain calculations are performed on the features of multiple candidate regions to obtain the response map corresponding to each candidate detection region. The candidate detection region corresponding to the maximum response value is determined from multiple response maps, and this candidate detection region is determined as the target tracking result of the current frame.

6. The single-target tracking method based on SIMD instruction DSP as described in claim 2, characterized in that, The scaling of the target region by the DSP core based on SIMD instructions to obtain the scaled target image patch includes: The positions of adjacent pixels corresponding to each pixel to be interpolated are determined based on the interpolation mapping relationship data. Read the pixel value data corresponding to multiple adjacent pixel positions, and load the pixel value data and the corresponding interpolation calculation data into the vector register; The target image patch is obtained by performing weighted multiplication and summation operations in parallel on multiple pixels to be interpolated using SIMD instructions.

7. The single-target tracking method based on SIMD instruction DSP as described in claim 3, characterized in that, The DSP core extracts features from the scaled target image patch based on SIMD instructions, obtaining target features including: The DSP core performs gradient calculations in parallel on multiple pixels in the target image block based on the SIMD instructions. The gradient magnitude and gradient direction of each pixel are determined, and the target image block is subjected to preset processing to obtain HOG features; wherein, the preset processing includes at least one of image unit division, gradient direction histogram statistics, neighborhood normalization and dimensionality reduction processing. The DSP core converts the target image block from the BGR color space to the Lab color space based on SIMD instructions, calculates the distance between each pixel and multiple preset Lab features, and performs color feature histogram statistics on image units to obtain Lab features. The target feature is obtained by concatenating HOG features and Lab features.

8. A single-target tracking device, characterized in that, Includes CPU core, DSP core, and DMA module; The CPU core acquires the video image to be tracked and determines the target region based on the target box specified in the first frame image; The CPU core generates the interpolation mapping data required for bilinear interpolation based on the target scaling size corresponding to the target region. The DMA module moves the image data of the target region and the interpolation mapping relationship data to the storage space accessible by the DSP core, and the DSP core performs scaling processing on the target region based on SIMD instructions to obtain the scaled target image block; The DSP core extracts features from the scaled target image patch based on SIMD instructions to obtain target features; The DSP core trains a model based on the target features to obtain a target tracking model; The CPU core and the DSP core work together in subsequent frames to perform target detection on multiple candidate detection regions of different scales based on the target tracking model, and determine the target tracking result of the current frame; wherein, the DSP core is used to perform feature extraction and frequency domain calculation, and the CPU core is used to perform maximum response value search and determine the target tracking result; The DSP core updates the target tracking model based on the target features corresponding to the target tracking result of the current frame, and the CPU core uses the target box corresponding to the target tracking result of the current frame as the pre-detection box for the next frame.

9. An electronic device, characterized in that, Includes memory, processor, and bus; The bus is used to enable communication between memory and processor; The processor is used to execute computer programs stored in memory; When the processor executes a computer program, it implements the steps in the single-target tracking method based on SIMD instruction DSP according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the single-target tracking method based on SIMD instruction DSP according to any one of claims 1 to 7.

Images