An edge-oriented lightweight multi-target tracking method and device
By building a lightweight multi-target tracking model on edge devices, integrating target detection and embedding learning, and using 8-bit fixed-point quantization and model pruning techniques, the problem of slow speed and high power consumption of traditional multi-target tracking methods on edge devices is solved, achieving high-performance, low-power real-time multi-target tracking with high accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING BENYUAN DATA TECH CO LTD
- Filing Date
- 2023-07-13
- Publication Date
- 2026-07-31
AI Technical Summary
Traditional multi-target tracking methods are slow and power-consuming on edge devices, cannot achieve real-time tracking, are costly, and cannot maintain high accuracy in complex scenarios.
A lightweight multi-target tracking method is adopted. By fusing target detection and embedding learning in a shared neural network, and using 8-bit fixed-point quantization and model pruning techniques, a lightweight multi-target tracking model is constructed to achieve global optimization.
It achieves high-performance, low-power real-time multi-target tracking on edge devices, reducing costs and improving tracking accuracy in complex scenarios.
Smart Images
Figure CN117132618B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multi-target tracking technology, and more specifically to a lightweight multi-target tracking method and apparatus for edge-oriented applications. Background Technology
[0002] Traditional multi-target tracking methods incorporate CNNs for appearance feature extraction models, such as... Figure 1 As shown. For each detected target, a convolutional neural network is first used to extract apparent features, and then the cosine distance between each target feature is calculated; alternatively, the Hungarian algorithm is used for feature matching, which significantly improves accuracy but also increases processing time. Because the network model uses floating-point representation for weights, power consumption is high and multi-target tracking is slow, making it unsuitable for edge computing devices. From... Figure 1 It can be seen that in traditional multi-target tracking methods, target detection, feature extraction, and target association are independent of each other, which makes the tracking time of such multi-target tracking methods long and unable to be optimized globally. Therefore, the accuracy of multi-target tracking decreases in complex construction site scenarios. On the other hand, since the weights of the model built by this scheme are floating-point type, the power consumption is large. Therefore, the inference of the model built by this scheme requires the support of server-side GPU graphics cards, which makes this scheme costly and unable to be widely used in products. Summary of the Invention
[0003] Purpose of the invention: The present invention discloses a lightweight multi-target tracking method and apparatus for edge-oriented applications, which can partially solve the above-mentioned technical problems.
[0004] Technical solution: In a first aspect, embodiments of this disclosure propose a lightweight multi-target tracking method for edge computing, the method comprising: Acquire the target image to be identified; The target image is input into a multi-target tracking model to obtain the target tracking result; The multi-target tracking model includes a learning detection network and a feature encoding network; The learning and detection network includes a first convolutional block, a first activation function, a first gated attention unit, an entropy-invariant function module, and first to third detection channels. The first convolutional block, the first activation function, and the first gated attention unit are cascaded to form a first granularity feature extraction module, used to extract first granularity features of the input image. The entropy-invariant function module is used to evenly distribute attention to the weights of all detection channels. The first detection channel obtains a second granularity feature based on the first granularity feature. The second detection channel obtains a third granularity feature based on the first granularity feature and the second granularity feature. The third detection channel obtains a fourth granularity feature based on the first granularity feature and the third granularity feature. The feature encoding network is used to encode the second granularity feature, the third granularity feature, and the fourth granularity feature respectively to obtain the output result of the multi-target tracking model.
[0005] As an optional implementation of the lightweight multi-target tracking method for edge-oriented devices described in this disclosure, the first detection channel includes a second convolutional block and a second gated attention unit; the second convolutional block is used to extract features from the first granularity features, and the second gated attention unit is used to perform feature fusion based on a gated attention mechanism on the features extracted by the second convolutional block to obtain the second granularity features; The second detection channel includes a first convolution and upsampling module, a second activation function, and a third gated attention unit; the first convolution and upsampling module is used to extract and upsample the second granular feature; the second activation function is used to process the first granular feature and the sampling result of the first convolution and upsampling module; the third gated attention unit is used to perform feature fusion based on a gated attention mechanism on the processing result of the second activation function to obtain the third granular feature; The third detection channel includes a second convolution and upsampling module, a third activation function, and a fourth gated attention unit; the second convolution and upsampling module is used to extract features and upsample the processing result of the second activation function; the third activation function is used to process the first granular feature and the sampling result of the second convolution and upsampling module; the fourth gated attention unit is used to perform feature fusion based on a gated attention mechanism on the processing result of the third activation function to obtain the fourth granular feature.
[0006] As an optional implementation of the lightweight multi-target tracking method for edge devices described in this disclosure, the feature encoding network includes a first encoding channel, a second encoding channel, and a third encoding channel; the first encoding channel, the second encoding channel, and the third encoding channel have the same structure, each including two parallel convolutional encoding sub-links, and the convolutional encoding sub-links include a third convolutional block and a feature encoding module cascaded in sequence.
[0007] As an optional implementation of the lightweight multi-target tracking method for edge-oriented devices described in this disclosure, the first to third convolutional blocks are deformable convolutional blocks, which can dynamically adjust the receptive field based on the size and position of the object.
[0008] As an optional implementation of the lightweight multi-target tracking method for edge-oriented devices described in this disclosure, the first to third activation functions are LeakyReLU functions, with the following expressions:
[0009] in, x Indicates the input signal. This represents the weighting coefficient.
[0010] As an optional implementation of the lightweight multi-object tracking method for edge-oriented embodiments of this disclosure, the multi-object tracking model further includes three parallel regression heads, which are used to estimate the heatmap, object center offset, and bounding box size, respectively.
[0011] As an optional implementation of the lightweight multi-target tracking method for edge computing described in this disclosure, the multi-target tracking model is trained end-to-end based on a loss function. The expression for the loss function is:
[0012] in, This represents the offset loss between the heatmap center and the object center. This represents the bounding box loss.
[0013] As an optional implementation of the lightweight multi-target tracking method for edge computing described in this disclosure, the multi-target tracking model is further compressed using int8 fixed-point quantization before training. The expression for int8 fixed-point quantization is:
[0014] in, s The scaling factor is 'range', the set of activation values for the weights is 'range', and 'bits' is the quantization bit width, which is 8 bits. The value is the quantized value, and clip is the stage function. z As a symmetrical point, The maximum value of the quantization. For the minimum value of quantization, This indicates the result of fixed-point quantization compression.
[0015] As an optional implementation of the lightweight multi-target tracking method for edge-oriented applications described in this disclosure, after compressing the multi-target tracking model, the multi-target tracking model is further pruned: During the training of the multi-object tracking model, a sparsified scaling factor is obtained. Channels with scaling factors below a specified threshold are pruned. Finally, the resulting multi-object tracking model is retrained to restore the accuracy lost due to pruning.
[0016] Secondly, embodiments of this disclosure provide an apparatus including a processor and a memory; the memory stores a computer program, and the processor executes the computer program to implement the lightweight multi-target tracking method for edge-oriented applications.
[0017] Thirdly, embodiments of this disclosure also propose a computer-readable storage medium storing a computer program that can be executed by a processor to implement the lightweight multi-target tracking method for edge devices.
[0018] Beneficial effects: Compared with the prior art, the present invention has the following advantages: This invention adopts a modeling method that integrates object detection and embedding learning into the same network. Detection and encoding are learned simultaneously in a shared neural network. A multi-task learning approach is used to set the loss function, enabling global optimization of the multi-object tracking model. This solves the problem that traditional multi-object tracking methods cannot achieve global optimization. Furthermore, because the model weights are quantized using 8-bit fixed-point quantization and combined with model pruning and compression techniques, the high power consumption and inability to run on edge devices of traditional methods are overcome, thus achieving cost savings. Attached Figure Description
[0019] Figure 1 This is a schematic diagram of the principle of a traditional multi-target tracking method. Figure 2 This is a diagram showing the backbone network structure of the multi-target tracking model involved in the embodiments of this disclosure; Figure 3 This is a structural diagram of the gated attention unit involved in an embodiment of this disclosure; Figure 4 This is a schematic diagram of the multi-task learning loss function involved in the embodiments of this disclosure. Implementation
[0020] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. However, it should be understood that the present invention can be implemented in various forms. The exemplary and non-limiting embodiments shown in the drawings and described below are not intended to limit the invention to the specific embodiments illustrated.
[0021] It should be understood that, where technically feasible, the technical features listed above for different embodiments can be combined with each other to form other embodiments within the scope of this invention. Furthermore, the specific examples and embodiments described in this invention are non-limiting, and corresponding modifications can be made to the structures, steps, and order described above without departing from the protection scope of this invention.
[0022] Currently, multi-target tracking technology is a key common technology in application scenarios such as autonomous driving and smart construction sites. Its main task is to simultaneously locate multiple targets of interest in a given image sequence and correlate them temporally to form a trajectory. Multi-target tracking models face problems of slow inference and low accuracy in edge computing resource-constrained environments. Therefore, this disclosure addresses the problem of multi-target tracking models being unable to run in real-time at the edge and having low accuracy from the perspective of modeling and model compression. It achieves high-performance, high-precision, and low-power real-time inference computing for multiple targets. To achieve this objective, this disclosure proposes a lightweight multi-target tracking method for edge computing, which includes the following steps: Acquire the target image to be identified; The target image is input into a multi-target tracking model to obtain the target tracking result.
[0023] Please refer to Figure 2 , Figure 2 The overall structure diagram of the multi-target tracking model is shown, including the learning detection network and the feature encoding network; The learning and detection network includes a first convolutional block, a first activation function, a first gated attention unit, an entropy-invariant function module, and first to third detection channels. The first convolutional block, the first activation function, and the first gated attention unit are cascaded to form a first granularity feature extraction module, used to extract first granularity features of the input image. The entropy-invariant function module is used to evenly distribute attention to the weights of all detection channels. The first detection channel obtains a second granularity feature based on the first granularity feature. The second detection channel obtains a third granularity feature based on the first granularity feature and the second granularity feature. The third detection channel obtains a fourth granularity feature based on the first granularity feature and the third granularity feature. The feature encoding network is used to encode the second granularity feature, the third granularity feature, and the fourth granularity feature respectively to obtain the output result of the multi-target tracking model.
[0024] In this embodiment, the first detection channel includes a second convolutional block and a second gated attention unit; the second convolutional block is used to extract features from the first granular feature, and the second gated attention unit is used to perform feature fusion based on the gated attention mechanism on the features extracted by the second convolutional block to obtain the second granular feature.
[0025] The second detection channel includes a first convolution and upsampling module, a second activation function, and a third gated attention unit; the first convolution and upsampling module is used to extract and upsample the second granular feature; the second activation function is used to process the first granular feature and the sampling result of the first convolution and upsampling module; the third gated attention unit is used to perform feature fusion based on the gated attention mechanism on the processing result of the second activation function to obtain the third granular feature.
[0026] The third detection channel includes a second convolution and upsampling module, a third activation function, and a fourth gated attention unit; the second convolution and upsampling module is used to extract features and upsample the processing result of the second activation function; the third activation function is used to process the first granular feature and the sampling result of the second convolution and upsampling module; the fourth gated attention unit is used to perform feature fusion based on a gated attention mechanism on the processing result of the third activation function to obtain the fourth granular feature.
[0027] The feature encoding network includes a first encoding channel, a second encoding channel, and a third encoding channel. The first encoding channel, the second encoding channel, and the third encoding channel have the same structure, each including two parallel convolutional encoding sub-links. Each convolutional encoding sub-link includes a third convolutional block and a feature encoding module cascaded in sequence.
[0028] In this embodiment, the first to third convolutional blocks are deformable convolutional blocks, which can dynamically adjust the receptive field based on the size and position of the object and alleviate alignment problems.
[0029] The entropy invariance function Softmax_plus in this embodiment has the following mathematical form: Entropy invariance means that it is insensitive to length n, so attention will be evenly distributed on all weights.
[0030] In this embodiment, the activation function is LeakyReLU, and its expression is:
[0031] in, x Indicates the input signal. This represents the weighting coefficient. The LeakyReLU function addresses the zero gradient problem for negative values by giving a very small linear component of the input x to the negative input 0.01x. LeakyReLU helps to extend the range of the ReLU function, typically with an alpha value around 0.01. The range of the LeakyReLU function is from negative infinity to positive infinity.
[0032] The structure of the gated attention unit (GAU) in this embodiment is as follows: Figure 3 As shown, the introduction of a gating mechanism to reduce the burden of self-attention is key to making the new model faster, more economical, and better. Compared to traditional attention models, gated attention units are characterized by small single heads and no softmax attention, making each GAU layer cheaper and lighter. More importantly, its quality is less dependent on attention accuracy, which is crucial for making multi-object tracking models faster, more economical, and better.
[0033] In this embodiment, the multi-object tracking model further includes three parallel regression heads, which are used to estimate the heatmap, object center offset, and bounding box size, respectively. Their principles are as follows: Figure 4 As shown. This implementation sets the loss function required for tracking and detection. The model learns both detection and tracking functions simultaneously, performing global optimization end-to-end. The mathematical form of the loss function is:
[0034] in, This represents the offset loss between the heatmap center and the object center. The bounding box loss is represented by the term "bounding box loss". Object detection is viewed as a center-based bounding box regression task on high-resolution feature maps. Specifically, three parallel regression heads are appended to the backbone network to estimate the heatmap, object center offset, and bounding box size, respectively. Each regression head is implemented by applying a 3×3 convolution (with 256 channels) to the output feature map of the backbone network, and then the final object is generated through a 1×1 convolutional layer. This regression head is responsible for estimating the location of the object center. A heatmap-based representation is used here, with a heatmap size of 1×H×W. The response decays exponentially with the distance between the location in the heatmap and the object center.
[0035] In this embodiment, the multi-target tracking model is compressed using int8 fixed-point quantization before training. The expression for int8 fixed-point quantization is:
[0036] in, s The scaling factor is 'range', the set of activation values for the weights is 'range', and 'bits' is the quantization bit width, which is 8 bits. The value is the quantized value, and clip is the stage function. z As a symmetrical point, The maximum value of the quantization. For the minimum value of quantization, This represents the result of fixed-point quantization compression. The floating-point operations involved in scaling and zero-point alignment are replaced with bit shifts so that we can use pure integer hardware.
[0037] Based on the constructed loss function, this embodiment trains a multi-object tracking model on the TensorRT database. During the training process, a self-supervised learning paradigm component is adopted. This paradigm aims to improve the feature extraction capability of the multi-object tracking model by designing auxiliary tasks to mine the representational characteristics of the data itself as supervision information for unlabeled data. Self-supervised learning can break the dependence on manual annotation and can efficiently train the multi-object tracking model even without labeled data.
[0038] As a preferred embodiment, this disclosure also proposes an apparatus including a processor and a memory; the memory stores a computer program, and the processor executes the computer program to implement the lightweight multi-target tracking method for edge-oriented applications.
[0039] Specifically, memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the training method for the face age estimation model and the program instructions / modules corresponding to the face age estimation method in the embodiments of this application. The processor can implement the lightweight multi-target tracking method and the multi-target tracking model training method in the embodiments of this application by running the non-transitory software programs, instructions, and modules stored in the memory. The memory may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0040] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
[0041] This disclosure also proposes a computer-readable storage medium, such as a memory including program code, which can be executed by a processor to complete the lightweight multi-target tracking method and the training of the multi-target tracking model in the above embodiments. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CDROM), magnetic tape, floppy disk, and optical data storage device, etc.
[0042] This disclosure also provides a computer program product comprising one or more lines of program code stored in a computer-readable storage medium. A processor of an electronic device reads the program code from the computer-readable storage medium and executes the program code to complete the implementation steps of the lightweight multi-target tracking method and the training of the multi-target tracking model provided in the above embodiments.
[0043] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0044] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.
Claims
1. An edge-oriented lightweight multi-target tracking method, characterized in that, The method includes: Acquire the target image to be identified; The target image is input into a multi-target tracking model to obtain the target tracking result; The multi-target tracking model includes a learning detection network, a feature encoding network, and three parallel regression heads; The learning and detection network includes a first convolutional block, a first activation function, a first gated attention unit, a softmax function module, and first to third detection channels. The first convolutional block, the first activation function, and the first gated attention unit are cascaded to form a first granularity feature extraction module, used to extract first granularity features of the input image. The softmax function module is used to evenly distribute attention to the weights of all detection channels. The first detection channel obtains a second granularity feature based on the first granularity feature. The second detection channel obtains a third granularity feature based on the first granularity feature and the second granularity feature. The third detection channel obtains a fourth granularity feature based on the first granularity feature and the third granularity feature. The feature encoding network is used to encode the second granularity feature, the third granularity feature, and the fourth granularity feature, respectively. The three parallel regression heads are used to estimate the heatmap, object center offset, and bounding box size based on the feature encoding output by the feature encoding network, respectively, to obtain the output results of the multi-object tracking model.
2. The method according to claim 1, characterized in that: The first detection channel includes a second convolutional block and a second gated attention unit; the second convolutional block is used to extract features from the first granularity features, and the second gated attention unit is used to perform feature fusion based on a gated attention mechanism on the features extracted by the second convolutional block to obtain the second granularity features; The second detection channel includes a first convolution and upsampling module, a second activation function, and a third gated attention unit; the first convolution and upsampling module is used to extract and upsample the second granular feature; the second activation function is used to process the first granular feature and the sampling result of the first convolution and upsampling module; the third gated attention unit is used to perform feature fusion based on a gated attention mechanism on the processing result of the second activation function to obtain the third granular feature; The third detection channel includes a second convolution and upsampling module, a third activation function, and a fourth gated attention unit; the second convolution and upsampling module is used to extract features and upsample the processing result of the second activation function; the third activation function is used to process the first granular feature and the sampling result of the second convolution and upsampling module; the fourth gated attention unit is used to perform feature fusion based on a gated attention mechanism on the processing result of the third activation function to obtain the fourth granular feature.
3. The method according to claim 2, characterized in that: The feature encoding network includes a first encoding channel, a second encoding channel, and a third encoding channel. The first encoding channel, the second encoding channel, and the third encoding channel have the same structure, each including two parallel convolutional encoding sub-links. Each convolutional encoding sub-link includes a third convolutional block and a feature encoding module cascaded in sequence.
4. The method according to claim 3, characterized in that: The first to third convolutional blocks are deformable convolutional blocks, which can dynamically adjust the receptive field based on the size and position of the object.
5. The method according to claim 3, characterized in that, The first to third activation functions are LeakyReLU functions, with the following expressions: ； in, x Indicates the input signal. This represents the weighting coefficient.
6. The method according to claim 1, characterized in that, The multi-object tracking model is compressed using int8 fixed-point quantization before training. The expression for int8 fixed-point quantization is: ； in, s The scaling factor is 'range', the set of activation values for the weights is 'range', and 'bits' is the quantization bit width, which is 8 bits. The value is the quantized value, and clip is the truncation stage function. x Represents raw floating-point data. z Zero point The maximum value of the quantization. For the minimum value of quantization, This indicates the result of fixed-point quantization compression. Indicates the maximum activation value. This represents the minimum activation value.
7. The method according to claim 6, characterized in that, After compressing the multi-target tracking model, the model is also pruned: During the training of the multi-object tracking model, a sparsified scaling factor is obtained. Channels with scaling factors below a specified threshold are pruned. Finally, the resulting multi-object tracking model is retrained to restore the accuracy lost due to pruning.
8. An apparatus, characterized in that, It includes a processor and a memory; the memory stores a computer program, and the processor executes the computer program to implement the method of any one of claims 1 to 7.