Target tracking and identifying system and method based on Raycore microprocessor

Abstract

The invention discloses a target tracking and identifying system and method based on a Raycore microprocessor, and belongs to the technical field of target tracking and identifying, and the method specifically comprises the steps that a Core0 collects image data from an external sensor in real time and transmits the image data to a Core1, the Core1 receives infrared image data collected by the Core0, firstly, the directional derivative of an image is calculated based on a Facet model, and then the directional derivative of the image is calculated based on a Facet model; the method comprises the following steps: firstly, acquiring directional derivative information, generating a penalty factor in combination with the directional derivative information, then constructing a three-layer double-local contrast model, completing construction of an LCWMD target detection model in combination with the penalty factor, performing preliminary analysis on an image by using the model, capturing initial position information of a target, and if the target is not shielded, sending an identification result to a long-term tracking module by Core2 for long-term tracking; if the target is shielded, the Core2 sends an identification result to the short-term tracking module for short-term tracking, under the condition that the target disappears, the SAM segmentation module extracts edge information of the shielded object and returns the result to the Core0, and the Core0 continues to collect data until the target appears again.

Description

Technical Field

[0001] The present invention belongs to the technical field of target tracking and recognition, and particularly relates to a target tracking and recognition system and method based on a Rockchip microprocessor. Background Art

[0002] Target tracking and recognition technologies are widely used in fields such as UAV monitoring, infrared imaging, and remote sensing data analysis. In these applications, high requirements are placed on the real-time performance and accuracy of target detection and tracking. Traditional target detection methods usually rely on a single computational model, such as target localization based on traditional algorithms or recognition methods based on deep learning. However, these methods often face problems such as low computational efficiency, poor real-time performance, and insufficient adaptability to dynamic environmental changes in complex backgrounds.

[0003] To solve these problems, the present invention proposes a target tracking and recognition system and method based on a Rockchip microprocessor. The system makes full use of the multi-core parallel computing power of the RK3588 processor, the high-performance computing power of the NPU, and its excellent performance in AI applications. It can provide high real-time performance and accurate target detection and tracking under complex backgrounds, occlusion, and dynamic change conditions, and is widely applicable to application scenarios such as target monitoring, UAV detection, and real-time tracking. Among them, Core0 is implemented by the ARM Cortex-A55 CPU core, and Core1 and Core2 are both implemented by the ARM Cortex-A76 CPU core, ensuring the real-time performance and computational efficiency of the algorithm, while also having strong environmental adaptability and power consumption optimization capabilities. The short-term tracking module, long-term tracking module, and SAM segmentation module are all implemented by three AI application NPU cores independently developed by Rockchip, giving full play to the advantages of the NPU core in AI applications and providing a reliable solution for the practical application of target tracking and recognition technologies. Summary of the Invention

[0004] The purpose of the present invention is to provide a target tracking and recognition system and method based on a Rockchip microprocessor, which solves the problems of low computational efficiency, poor real-time performance, and insufficient adaptability to dynamic environmental changes in complex backgrounds caused by the over-reliance of existing traditional target detection methods on a single computational model.

[0005] To achieve the above object, the present invention provides an object tracking and recognition system based on Rockchip microprocessors, which includes three Cortex cores and three AI application NPU cores independently developed by Rockchip. Image acquisition, object positioning, feature extraction and recognition are realized through an efficient object detection and tracking process and a deep learning network; the three Cortex cores include Core0, Core1 and Core2; the three AI application NPU cores independently developed by Rockchip include a SAM segmentation module, a long-term tracking module and a short-term tracking module; Core0 collects image data from an external sensor in real time and transmits it to Core1. Core1 receives the infrared image data collected from Core0, first calculates the directional derivative of the image based on the Facet model, and generates a penalty factor in combination with the directional derivative information. Subsequently, by constructing a three-layer dual local contrast model and combining the penalty factor, the LCWMD object detection model is constructed. The model is used to perform a preliminary analysis of the image to capture the initial position information of the object. If the object is not occluded, Core2 will transmit the recognition result to the long-term tracking module for long-term tracking and recognition; if the object is occluded, Core2 will transmit the recognition result to the short-term tracking module for short-term tracking. In the case where the object disappears, the SAM segmentation module of the NPU core will extract the edge information of the occluder and return the result to Core0, and Core0 will continue to collect data until the object reappears.

[0006] The present invention also provides a method for an object tracking and recognition system based on Rockchip microprocessors, including the following steps: Step 1, data acquisition and transmission; Core0 is connected to an external sensor, controls the external sensor to collect each frame of image, and transmits the collected image data to Core1 through an interface in real time for subsequent processing; when receiving the segmentation result of the occluder boundary from Core2, it will control the external sensor to collect images at its edge and wait for a signal of the reappearance of the object; when receiving the recognition result of the reappearance of the object from Core2, it will continue to control the external sensor to track and collect the object image according to the recognition result; Step 2, constructing the LCWMD object detection model and capturing the initial object position; Core1 uses the collected infrared image data to calculate multi-directional derivative features and perform three-layer dual local contrast modeling based on the Facet model, so as to construct the LCWMD object detection model; Step 3, using Core2 to judge and classify the object calculation results output by Core1; when the object is recognized, it is judged whether the object is occluded; when the object is not recognized, the object recognition result is transmitted to the SAM segmentation module of the NPU core through Core2, and the segmentation result is transmitted to Core0, and step 1 is executed again.

[0007] Preferably, multi-directional derivative features The specific expression is as follows: ; Among them, represents the second-order directional derivative in the direction, where i and j are and represents the offset , and respectively represent the results after filtering the derivative maps in two different directions.

[0008] Preferably, the expression of the three-layer double local contrast model is as follows: ; Among them, represents the average gray value of the corresponding layer, represents the maximum value among the average gray values of 8 background blocks, represents the average gray value of the u-th background block in the layer, where u is 1, 2, 3, 4, 5, 6, 7, 8,

[0009] Preferably, the expression of the LCWMD target detection model is as follows: ; ; ; In the formula, and represent the mean and variance of is a given constant, and the best range obtained from experiments is 0.3 to 0.8, represents the number of pixels that satisfy the condition.

[0010] Preferably, the specific content of determining whether the target is occluded in step 3 is as follows: When the target is occluded, the following conditions are satisfied: ; In the formula, represents the value of the in the frame, represents the value of the in the frame, represents the set low threshold coefficient, and the best range obtained from experiments is 0.2 to 0.5. Denotes the set high threshold coefficient, and the optimal range obtained from experiments is 0.7 to 0.9; When the target is not occluded, the following conditions are satisfied: 。

[0011] Preferably, when the target is occluded, Core2 sends the target recognition result to the short-term tracking module of the NPU core. The short-term tracking module extracts features and trains the target, specifically: First, the target bounding box is extracted through the SAM segmentation module in the NPU core. Then, the short-term tracking module extracts samples to train the short-term network. After that, the short-term network is used to recognize the target from the received image data, and the recognition result is returned to Core2 for verification and classification.

[0012] Preferably, when the target is not occluded, Core2 transmits the recognition result to the long-term tracking module of the NPU core for long-term tracking and recognition, specifically: First, the target bounding box is extracted through the SAM segmentation module in the NPU core. Then, the long-term tracking module extracts samples to train the long-term network. After that, the long-term network is used to recognize the target from the received image data, and the recognition result is returned to Core2 for verification and classification.

[0013] Preferably, when the target is not recognized, the following conditions are satisfied: 。

[0014] Therefore, the present invention adopts the above-mentioned target tracking and recognition system and method based on Rockchip microprocessors. Through the combination of the Rockchip RK3588 CPU core and the NPU core, and by utilizing the parallel computing power of the Rockchip RK3588 CPU core and the deep learning module of the NPU core, the system achieves efficient real-time processing under multiple frames of images, meeting the high real-time requirements. At the same time, in complex environments such as occlusion and dynamic changes, it can continuously and accurately track the target. Through the cooperation of long-term and short-term convolutional neural networks, when the target disappears or is occluded, the system can accurately predict the target position and perform short-term or long-term tracking, effectively reducing errors.

[0015] Next, through the drawings and embodiments, the technical solutions of the present invention will be further described in detail. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a calculation flow chart between the RK3588 CPU core and the NPU core module in the Rockchip high-performance intelligent application processor of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts fall within the scope of protection of the present invention.

[0018] Please refer to Figure 1 , a target tracking and recognition system based on Rockchip microprocessors, including three Cortex cores and three AI application NPU cores independently developed by Rockchip. It realizes image acquisition, target positioning, feature extraction and recognition through an efficient target detection and tracking process and a deep learning network; the three Cortex cores include Core0, Core1 and Core2; the three AI application NPU cores independently developed by Rockchip include a SAM segmentation module, a long-term tracking module and a short-term tracking module; in addition, the three Cortex cores are specifically two high-performance Cortex-A76 CPU cores and one high-efficiency Cortex-A55 CPU core; the AI application NPU core independently developed by Rockchip uses the Rockchip RK3588 CPU core. Core0 collects image data from external sensors in real time and transmits it to Core1. Core1 receives the infrared image data collected from Core0, first calculates the directional derivative of the image based on the Facet model, and generates a penalty factor in combination with the directional derivative information. Subsequently, by constructing a three-layer double local contrast model and combining the penalty factor, the construction of the LCWMD target detection model is completed. The model is used to preliminarily analyze the image and capture the initial position information of the target. If the target is not occluded, Core2 will transmit the recognition result to the long-term tracking module for long-term tracking and recognition; if the target is occluded, Core2 will transmit the recognition result to the short-term tracking module for short-term tracking. In the case of the target disappearing, the SAM segmentation module of the NPU core will extract the edge information of the occluder and return the result to Core0, and Core0 will continue to collect data until the target reappears; A method for a target tracking and recognition system based on Rockchip microprocessors includes the following steps: Step 1, data acquisition and transmission; Core0 is connected to an external sensor, controls the external sensor to collect each frame of image, and transmits the collected image data to Core1 through an interface in real time for subsequent processing; when receiving the occluder boundary segmentation result from Core2, it will control the external sensor to collect images at its edge and wait for the signal of the target reappearing; when receiving the recognition result of the target reappearing from Core2, it will continue to control the external sensor to track and collect the target image according to the recognition result; Step 2: Construct the LCWMD target detection model and capture the initial target position; Core1 uses the collected infrared image data to calculate multi-directional derivative features and perform three-layer dual local contrast modeling based on the Facet model, thereby constructing the LCWMD target detection model; among them, the multi-directional derivative features The specific expression is as follows: ; where represents the second-order directional derivative in the direction, are i, j, , and respectively represent the results after filtering the derivative maps in two different directions.

[0019] The expression of the three-layer dual local contrast model is as follows: ; where represents the average gray value of the corresponding layer, represents the maximum value among the average gray values of 8 background blocks, represents the average gray value of the u-th background block in the layer, where u is 1, 2, 3, 4, 5, 6, 7, 8,

[0020] The expression of the LCWMD target detection model is as follows: ; ; ; In the formula, and represent the mean and variance of is a given constant, and the optimal range obtained from experiments is 0.3 to 0.8, represents the number of pixels that satisfy the condition.

[0021] Step 3: Use Core2 to judge and classify the target calculation results output by Core1; when the target is recognized, judge whether the target is occluded; when the target is not recognized, send the target recognition result to the SAM segmentation module of the NPU core through Core2, and transmit the segmentation result to Core0, then return to execute Step 1; specifically: when Core2 judges that the target is not recognized or calls SAM to segment the target border, transfer the image data to the SAM segmentation module in the NPU core. Based on the pre-trained large segmentation model, this module extracts the edges and segments the occluders in the image, accurately locating the occluded area. By calling the trained model parameters, the SAM module can quickly complete the segmentation of the occluder boundary and transmit the segmentation result to Core0, providing key information support for subsequent target re-recognition. Then return the recognized result to Core2 for verification and classification; among them, the specific content of judging whether the target is occluded is as follows: When the target is occluded, the following conditions are met: ; In the formula, represents the value of the th frame, represents the value of the th frame, represents the set low threshold coefficient, and the best range obtained from experiments is 0.2 to 0.5, represents the set high threshold coefficient, and the best range obtained from experiments is 0.7 to 0.9; When the target is not occluded, the following conditions are met: .

[0022] When the target is not recognized, the following conditions are met: .

[0023] When the target is occluded, Core2 sends the target recognition result to the short-term tracking module of the NPU core. The short-term tracking module extracts features and trains the target. Specifically: first, extract the target border through the SAM segmentation module in the NPU core, then use the short-term tracking module to extract samples to train the short-term network, and then use the short-term network to recognize the target for the received image data, and return the recognition result to Core2 for verification and classification.

[0024] When the target is not occluded, Core2 transmits the recognition result to the long-term tracking module of the NPU core for long-term tracking and recognition. Specifically: First, the target bounding box is extracted through the SAM segmentation module in the NPU core. Then, the long-term tracking module is used to extract samples to train the long-term network. After that, the long-term network is used to perform target recognition on the received image data, and the recognition result is returned to Core2 for verification and classification.

[0025] Therefore, the present invention adopts the above-mentioned target tracking and recognition system and method based on Rockchip microprocessors. Through the combination of the Rockchip RK3588 CPU core and the NPU core, and by utilizing the parallel computing power of the Rockchip RK3588 CPU core and the deep learning module of the NPU core, the system realizes efficient real-time processing under multiple frames of images, meeting the high real-time requirements. At the same time, in complex environments such as occlusion and dynamic changes, it can continuously and accurately track the target. Through the cooperation of long-term and short-term convolutional neural networks, when the target disappears or is occluded, the system can accurately predict the target position and perform short-term or long-term tracking, effectively reducing errors. This technical solution has broad application prospects especially in the fields of UAV monitoring, infrared imaging, intelligent monitoring, etc., and can significantly improve the accuracy, real-time performance, and adaptability of target detection and tracking.

[0026] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that they can still modify or equivalently replace the technical solutions of the present invention, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A target tracking and recognition system based on Rockchip microprocessors, characterized in that: It includes three Cortex cores and three AI application NPU cores independently developed by Rockchip; The three Cortex cores include Core0, Core1, and Core2; the three AI application NPU cores independently developed by Rockchip include the SAM segmentation module, the long-term tracking module, and the short-term tracking module; Core0 collects image data from external sensors in real time and transmits it to Core1. Core1 receives the infrared image data collected from Core0, first calculates the directional derivative of the image based on the Facet model, and generates a penalty factor in combination with the directional derivative information. Subsequently, by constructing a three-layer dual local contrast model and combining the penalty factor, the LCWMD target detection model is constructed. Using this model, the image is preliminarily analyzed to capture the initial position information of the target. If the target is not occluded, Core2 transmits the recognition result to the long-term tracking module for long-term tracking and recognition; If the target is occluded, Core2 transmits the recognition result to the short-term tracking module for short-term tracking. In the case where the target disappears, the SAM segmentation module of the NPU core extracts the edge information of the occluder and returns the result to Core0, and Core0 will continue to collect data until the target reappears.

2. A method for a target tracking and recognition system based on a Rockchip microprocessor as described in claim 1, characterized in that, It includes the following steps: Step 1, data collection and transmission; Core0 connects to an external sensor, controls the external sensor to collect each frame of image, and transmits the collected image data to Core1 through an interface in real time for subsequent processing; when receiving the occluder boundary segmentation result from Core2, it will control the external sensor to collect images at its edge and wait for the signal of the target reappearance; when receiving the recognition result of the target reappearance from Core2, it will continue to control the external sensor to track and collect the target image according to the recognition result; Step 2, construct the LCWMD target detection model and capture the initial target position; Core1 uses the collected infrared image data to calculate the multi-directional derivative features and construct a three-layer dual local contrast model based on the Facet model, so as to construct the LCWMD target detection model; Step 3, use Core2 to judge and classify the target calculation results output by Core1; When the target is recognized, it is judged whether the target is occluded; When the target is not recognized, the target recognition result is sent to the SAM segmentation module of the NPU core through Core2, and the segmentation result is transmitted to Core0, and step 1 is executed again.

3. The method of a target tracking and recognition system based on Rockchip microprocessor according to claim 2, characterized in that, Multi-directional derivative feature The specific expression is as follows: ； Among them, represents the second-order directional derivative in the direction, where i and j represent the offset , and respectively represent the results after filtering the derivative maps in two different directions.

4. The method of an object tracking and recognition system based on Rockchip microprocessor according to claim 3, characterized in that Three-layer double local contrast model The expression is as follows: ； Among them, represents the average gray value of the corresponding layer, represents the maximum value among the average gray values of 8 background blocks, represents the average gray value of the u-th background block in the layer, where u is 1, 2, 3, 4, 5, 6, 7, 8, is the offset set to 1, and TC, TA, and LB respectively represent the TC core layer, TA attenuation layer, and LB background layer.

5. A method for a target tracking and recognition system based on a Rockchip microprocessor according to claim 4, characterized in that, The expression of the LCWMD target detection model is as follows: ； ； ； In the formula, and represent the mean value and variance of, is a given constant, represents the number of pixels satisfying the 6. The method of a target tracking and recognition system based on Rockchip microprocessor according to claim 5, characterized in that, The specific content of judging whether the target is occluded in step 3 is as follows: When the target is occluded, the following conditions are met: ； wherein, represents the value of the th frame, represents the value of the th frame, represents the set low threshold coefficient, represents the set high threshold coefficient; When the target is not occluded, the following conditions are met: 。 7. The method of a target tracking and recognition system based on Rockchip microprocessor according to claim 6, characterized in that: When the target is occluded, Core2 sends the target recognition result to the short-term tracking module of the NPU core. The short-term tracking module extracts features and trains the target. Specifically: First, the target bounding box is extracted through the SAM segmentation module in the NPU core. Then, the short-term tracking module extracts samples to train the short-term network. After that, the short-term network is used to perform target recognition on the received image data, and the recognition result is returned to Core2 for verification and classification.

8. The method of an object tracking and recognition system based on Rockchip microprocessor according to claim 7, characterized in that: When the target is not occluded, Core2 transmits the recognition result to the long-term tracking module of the NPU core for long-term tracking and recognition. Specifically: First, the target bounding box is extracted through the SAM segmentation module in the NPU core. Then, the long-term tracking module extracts samples to train the long-term network. After that, the long-term network is used to perform target recognition on the received image data, and the recognition result is returned to Core2 for verification and classification.

9. The method of a target tracking and recognition system based on Rockchip microprocessor according to claim 8, characterized in that, When the target is not recognized, the following conditions are met: 。

Images