Photoelectric turntable signal processing system and method based on multi-core heterogeneous architecture

Abstract

The invention relates to a photoelectric turntable signal processing system and method based on a multi-core heterogeneous architecture. The system comprises an FPGA, a multi-core DSP and an ARM processor integrated with a neural network inference engine. The FPGA collects visible light and infrared videos, and pre-processed images are distributed to the DSP and the ARM in parallel through SRIO and LVDS special channels; the ARM operates an improved YOLOv4-tiny model to carry out target identification, and position information is sent to the DSP to guide tracking initialization; the method comprises the steps that the FPGA collects and distributes images in parallel, the ARM recognizes a target and guides the DSP to track, the DSP calculates the miss distance and integrates the memory tracking module, and Kalman prediction maintains output when the target is lost. According to the scheme, identification and tracking are deeply coordinated, the end-to-end delay is less than or equal to 20 ms, the recovery rate within 2 seconds of loss is greater than or equal to 95%, the volume power consumption is reduced by more than 30%, and the method is suitable for military / police photoelectric reconnaissance, security monitoring, unmanned aerial vehicle load and other scenes.

Description

Technical Field

[0001] This invention relates to the field of optoelectronic information processing and intelligent control technology, and in particular to an optoelectronic turntable signal processing system and method based on a multi-core heterogeneous architecture. Background Technology

[0002] Traditional optoelectronic turntable signal processing systems generally adopt a separate "acquisition-transmission-processing" architecture. The image acquisition unit, target recognition unit, tracking control unit, and servo drive unit are usually physically or logically independent modules that interact with each other via standard buses (such as Ethernet and PCIe). This architecture results in long data transmission paths and numerous stages between modules, introducing significant communication delays and processing latency, making it difficult to meet the requirements for continuous and stable tracking of high-speed, highly maneuverable targets.

[0003] Furthermore, with the development of deep learning technology, object detection and recognition methods based on convolutional neural networks have been applied in many scenarios. However, when applying such technologies to embedded systems like optoelectronic turntables, which have stringent requirements for real-time performance, robustness, and power consumption, existing solutions still face significant bottlenecks: First, there is a contradiction between high-precision recognition models and the requirements of embedded computing power and real-time performance. To improve recognition accuracy in complex scenarios, existing technologies often employ deep neural network models with a huge number of parameters and high computational complexity. While such models can run on general-purpose computing platforms (such as GPUs), they are difficult to deploy directly on the resource-constrained embedded processing units (such as ARM application processors) used in optoelectronic turntables. Although there have been attempts with lightweight models such as the YOLO series, the balance between accuracy loss, inference speed, and power consumption has not been fundamentally resolved, making it difficult to simultaneously meet the system requirements of high recognition rates and high processing frame rates.

[0004] Secondly, the coordination between the target recognition function and subsequent tracking and control is insufficient. In the existing optoelectronic turntable system architecture, the target recognition module and the tracking algorithm module are usually designed as independent or serial functional units. The recognition results are only used for initial target acquisition or periodic verification, and there is a lack of real-time, close status feedback and closed-loop coordination mechanism between the two modules. When the target is temporarily lost due to occlusion, rapid movement, or background interference, the system cannot make quick and autonomous decisions and recover based on the context information provided by the recognition module or the status feedback from the tracking module, resulting in limited overall system intelligence and continuous stable tracking capability.

[0005] Furthermore, the multi-sensor data fusion processing capability is weak. To improve environmental adaptability, modern optoelectronic turntables often integrate multi-spectral sensors such as visible light and infrared. Existing technologies for processing multiple video streams mostly remain at the level of independent processing and simple superposition, failing to achieve efficient and real-time heterogeneous data fusion at the feature layer or decision layer. This prevents the full utilization of the complementary advantages of multi-source information, and system performance is prone to sharp declines when the imaging quality of a single spectral band is poor. Therefore, existing technological solutions are insufficient to meet the integrated requirements of high real-time performance, high reliability, and low power consumption intelligent processing in high-end optoelectronic turntable systems, urgently requiring an innovative system architecture and collaborative processing method. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a photoelectric turntable signal processing system based on a multi-core heterogeneous architecture, comprising: An FPGA is configured to acquire at least two raw video data streams from a photoelectric sensor and preprocess the raw video data, the preprocessing including at least decoding, YUV format conversion, and buffering; the FPGA is also configured to distribute the preprocessed first image data to a multi-core DSP via a first dedicated data path, and to distribute the preprocessed second image data to an ARM in parallel via a second dedicated data path; the first dedicated data path is a serial RapidIO interface, and the second dedicated data path is an LVDS interface; A multi-core DSP is configured to receive the first channel of image data through the serial RapidIO interface, execute a target tracking algorithm in parallel across multiple cores to calculate the miss distance, and output the miss distance to the servo control system through the FPGA; the multi-core DSP is internally configured to execute the target tracking task in parallel across multiple cores, including: The first kernel is used to extract HOG features from image data; The second core is used to perform discrete Fourier transform / inverse transform and correlation calculation; The third core is used to perform tracking template updates and average peak-related energy calculations. The multi-core DSP also includes a memory tracking module, configured to predict the position based on the motion state before the target is lost when the average peak correlation energy value is lower than a preset threshold, and maintain the miss distance output within a preset time window; the memory tracking module uses a Kalman filter algorithm to predict the target position based on the motion trajectory information of N consecutive frames before the target is lost. The ARM, which integrates a neural network inference engine, is configured to receive the second image data via the LVDS interface and run a lightweight target detection model after INT8 quantization. The model is a network structure based on an improved YOLOv4-tiny architecture, including a newly added stride=64 deep feature layer, used to perform target detection and classification on the image data and generate target location information. The ARM is also configured to send the target location information to the multi-core DSP to guide or update target tracking. A bidirectional status interaction channel is located between the multi-core DSP and the ARM, used to transmit tracking status information and identification control commands. The tracking status information includes the average peak correlation energy value. The ARM dynamically adjusts the identification strategy based on the received average peak correlation energy value, forming a bidirectional closed loop of identification and tracking.

[0007] In one embodiment of the present invention, the multi-core DSP is further configured to calculate the average peak correlation energy value in real time and compare it with a preset threshold to determine the tracking status; the preset threshold is 12, and when the average peak correlation energy value is lower than 12, it is determined that the target is lost.

[0008] In one embodiment of the present invention, the preset time window of the memory tracking module is 0.5 seconds to 2 seconds, during which the predicted miss distance is continuously output; if the target is recaptured within the time window, the normal tracking mode is restored, otherwise the tracking failure status is reported.

[0009] In one embodiment of the present invention, the ARM dynamically adjusts the recognition strategy based on the received average peak correlation energy value, including: when the average peak correlation energy value is higher than a preset threshold, the ARM shrinks the recognition area to the periphery of the tracking window to reduce computing power consumption; when the average peak correlation energy value is lower than the preset threshold, the ARM resumes full-image detection to improve the target re-capture probability.

[0010] In one embodiment of the present invention, the system is further configured to enter a search mode or report a loss status after a target loss timeout, wherein the timeout period can be configured to be 2 seconds.

[0011] The present invention also provides a signal processing method for an optoelectronic turntable, comprising the following steps: Step S1: The FPGA acquires multiple video data in parallel, performs preprocessing, and then distributes the first image data to the multi-core DSP through the serial RapidIO interface, and distributes the second image data to the ARM in parallel through the LVDS interface. Step S2: The ARM performs target recognition based on the neural network inference engine. The improved YOLOv4-tiny model with INT8 quantization is used to detect and classify targets in the image. The target location and category information with the highest confidence are output through non-maximum suppression. The target location information is sent to the multi-core DSP. Step S3: The multi-core DSP performs target tracking based on the target position information or the tracking status of the previous frame. The improved multi-scale kernel correlation filtering algorithm is run in parallel by multiple cores to calculate and output the miss distance in real time. At the same time, the average peak correlation energy value is calculated to evaluate the tracking stability. When the average peak correlation energy value is lower than the preset threshold, it is determined that the tracking has lost lock. Step S4: When the tracking loss judgment is triggered, the multi-core DSP starts the memory tracking module, predicts the target position based on the Kalman filter algorithm, and continuously outputs the predicted miss amount within a preset time window of 0.5 seconds to 2 seconds to maintain the servo system's pointing tracking of the target; the ARM dynamically adjusts the recognition strategy according to the received average peak correlation energy value to form a two-way closed loop of recognition and tracking.

[0012] In one embodiment of the present invention, the YOLOv4-tiny improved model in step S2 includes a newly added stride=64 deep feature layer, which is used to extract deep features with high semantic information to improve the small target recognition capability.

[0013] In one embodiment of the present invention, the preset threshold in step S3 is 12, and when the average peak correlation energy value is lower than 12, it is determined that the tracking has lost lock.

[0014] The technical solution of the present invention has the following advantages compared with the prior art: The photoelectric turntable signal processing system and method described in this invention achieves several significant advantages through a multi-core heterogeneous architecture and a deep collaborative mechanism. In terms of real-time performance, the dedicated SRIO and LVDS data paths built on the FPGA enable parallel zero-copy distribution of image data. Combined with the DSP's multi-core parallelized KCF algorithm, the end-to-end latency is controlled within 32.5ms, reducing it by more than 60% compared to the traditional discrete architecture, meeting the real-time tracking requirements for high-speed maneuvering targets. Regarding robustness and autonomy, the system pioneers a bidirectional closed loop that deeply integrates recognition and tracking: the ARM-based YOLOv4-tiny model adds a stride=64 deep feature layer, achieving a small target recognition rate of 88.7% even in low signal-to-noise ratio scenarios; the DSP-based integrated memory tracking module continuously predicts the target's position and outputs the miss distance for up to 2 seconds based on Kalman filtering when the target is lost, achieving an occlusion recovery success rate of up to 97%, completely changing the traditional system's failure-as-a-miss mode. In terms of integration and energy efficiency, the single board integrates acquisition, identification, tracking, and control functions, with a total power consumption of only 13.8W, a 40% reduction in size, and a 30% reduction in power consumption, making it particularly suitable for embedded platforms such as drones and automotive applications. Regarding intelligence, the ARM and DSP establish bidirectional state interaction through shared memory, guiding tracking initialization and optimizing identification strategies based on tracking status, forming a perception-decision-control closed loop, significantly enhancing the system's autonomous decision-making capabilities.

[0015] In summary, this invention has achieved groundbreaking progress in four dimensions: high real-time performance, high robustness, low power consumption, and high integration, and has outstanding value for industrial application. Attached Figure Description

[0016] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0017] Figure 1 This is a schematic block diagram illustrating the architectural principle of the optoelectronic turntable signal processing system based on a multi-core heterogeneous architecture according to the present invention. Figure 2 This is a flowchart of the FPGA software processing described in this invention; Figure 3 This is a flowchart of the DSP software processing described in this invention; Figure 4 This is the ARM processing flowchart described in this invention; Figure 5 This is a flowchart of the tracking algorithm processing in this invention; Figure 6 This is a diagram of the improved model network structure based on YOLOv4-tiny in this invention; Figure 7 This is a flowchart of the target detection and recognition algorithm in this invention; Figure 8 This is a flowchart of the target tracking algorithm software in this invention. Detailed Implementation Example 1

[0018] This embodiment provides a signal processing system for an optoelectronic turntable based on a multi-core heterogeneous architecture; such as Figure 1 As shown, the data and control interface connections between FPGA, DSP, and ARM, as well as external interfaces with sensors, servos, and displays are illustrated.

[0019] Specifically, an integrated multi-core heterogeneous processing board is selected, and the selection and configuration are as follows: Programmable Logic Unit (FPGA): Employs a Xilinx Kintex-7 series FPGA chip. This unit receives raw video data streams from a visible light HD camera (1920×1080 resolution, 30fps) and an uncooled infrared thermal imager (640×512 resolution, 25Hz frame rate) via dual CameraLink interfaces. The FPGA's internal logic implements video decoding, YUV422 color space conversion, and DDR3 cache management. Figure 2 As shown, after preprocessing, the FPGA distributes the image slice data to the digital signal processing unit at a rate of 5Gbps via the first dedicated data path, namely the serial RapidIO interface, at a rate of 4×Lane; at the same time, it distributes the full-resolution image data to the application processing unit in parallel via the second dedicated data path, namely the LVDS interface.

[0020] Digital Signal Processing Unit (DSP): Employs a TI TMS320C6678 octa-core digital signal processor. In this embodiment, the DSP receives a 128×128 pixel image slice from the FPGA via the SRIO interface. This slice is centered on the target and covers the search area. Figure 3 The diagram shown is a flowchart of the DSP software processing of this invention. It runs an improved multi-scale kernel correlation filter (KCF) tracking algorithm to calculate the miss distance in real time, and outputs the result to the servo controller via the FPGA's UART hard core.

[0021] The miss distance refers to the distance difference between the target and the center point. Its working principle is based on the feedback miss distance. First, the actual angle difference between the target position and the center point is calculated. Then, based on this angle difference, the motor is driven to move the device so that the center point of the device coincides with the target position, thereby achieving the effect that the device always follows the target position.

[0022] The DSP internally implements a multi-core parallelization strategy, with the specific task division as follows: Core 0: Responsible for extracting HOG features (histogram of oriented gradients) of image regions; Core 1: Performs discrete Fourier transform and inverse transform, frequency domain correlation calculation, and runs an improved multi-scale kernel correlation filter (KCF) tracking algorithm. The KCF algorithm is a tracker based on the ridge regression function problem. It uses kernel functions to calculate the degree of matching between the predicted region and the target, and takes the predicted region with the highest degree of matching as the new tracking target.

[0023] The KCF target tracking algorithm first extracts HOG features from the image information of the target region; then, it generates a large number of samples through cyclic displacement to train the target classifier; it uses the Gaussian kernel function to find the correlation response between the target sample and the test sample, and the coordinates of the point with the maximum response value are the latest position of the target; it uses discrete Fourier transform to transform the above process from the time domain to the frequency domain, which can greatly reduce the amount of computation and improve the computation speed; finally, it updates the classifier using the new target features.

[0024] Specific classifier training: Let a set of training samples be... , For the sample The corresponding regression expectation. The goal of classifier training is to find a function. Make the regularization risk function (1) To obtain the minimum value. Where: These are the classifier parameters; This is a regularization parameter, whose function is to prevent overfitting.

[0025] The optimal solution can be expressed as a linear combination of all input training samples, i.e. (2) Substituting equation (2) into equation (1) in kernel space and performing a discrete Fourier transform yields the following result: (3) The purpose of classifier training is to solve for the position filter. , Kernel Circular Matrix The first element.

[0026] Detection and Update; After training, the KCF algorithm uses the target position of the previous frame as the center and updates the target image patch in the new frame. The above is used for detection, and its relevant response expression is: (4) In the formula: This represents the kernel correlation between the target sample x and the detection region z; response The position where the maximum value is obtained represents the position of the tracked target in the new frame image.

[0027] The KCF algorithm uses linear interpolation to update the filter template and the target appearance template. The update method is as follows: (5) (6) In the formula: Indicates the update rate; t represents the number of frames; , These represent the filter template and the target appearance template from the previous frame, respectively.

[0028] Target tracking process The KCF target tracking algorithm may fail during target tracking due to factors such as occlusion or the target being out of the field of view. To ensure effective target tracking over a longer period, the algorithm includes a re-detection module, which re-detects the target's position after determining that the target has been lost due to occlusion or other factors.

[0029] To assess occlusion during tracking, the algorithm introduces the Mean Peak Correlation Energy (APCE), an indicator that reflects the degree of oscillation in the response map. Its expression is: (7) In the formula: Indicates the value of APCE; Indicates to Calculate the average value; , These represent the maximum and minimum values ​​in the response graph, respectively. This represents the value in row p and column q of the response graph. When occlusion occurs, the APCE value will decrease significantly. The target tracking anti-occlusion capability of this software is that it can track normally if the target tracking time is less than 2 seconds, and the corresponding APCE threshold is 12.

[0030] The target tracking function is developed based on the KCF tracking algorithm, and its software flow is shown in [link to software flowchart]. Figure 8 The target tracking function is implemented by calling the encapsulated tracking algorithm interface.

[0031] Core 2: Responsible for real-time calculation and tracking status determination of template updates, target appearance model updates, and average peak correlation energy (APCE). APCE (Average Peak Correlation Energy) is a key indicator used in target tracking (especially in correlation filtering-based tracking algorithms such as KCF and CSK) to quantify the quality of the correlation map and evaluate tracking reliability. Its core is to reflect the degree of matching between the target and the template through the "peak prominence" in the correlation map.

[0032] The remaining cores are used for task scheduling, shared memory management, and auxiliary computation.

[0033] This embodiment uses a double buffering mechanism to implement data pipeline, that is, the processing of the current frame and the preloading of the next frame are executed in parallel. At the same time, the overhead of inter-core copying is reduced by sharing DDR memory mapping. The measured single-frame KCF tracking time is reduced by about 40%.

[0034] Therefore, the KCF tracing algorithm used in the above-mentioned specific functions has undergone relevant adjustments and optimizations in its calculation process, and several strategies have been added to make the algorithm more robust in complex environments. The specific process is as follows: Tracker initialization: At the start of the first frame of tracking, target bounding box information is input. Based on the target bounding box information, the HOG features of the target region are calculated, and this target region is used as the target template. Then, its width and height are expanded to three times as the search region (i.e., a nine-square grid area centered on the target bounding box). The HOG features of the search region are calculated. A large number of positive and negative samples are collected in the search region through cyclic shifting. Based on the target's HOG features, a classifier is trained using the positive and negative sample set and the ridge regression algorithm to obtain the relevant filtering template.

[0035] Tracking phase: After the next frame of image is input, at the target's position in the previous frame, a search region is selected (in the same way as the search region selection in step a). The HOG features of the search region are calculated and smoothed by a cosine window to obtain the sample set to be detected. Then, the sample set to be detected is multiplied by the correlation filter template in step a) to obtain the response map of the current search region. The frequency domain response map is transformed by inverse Fourier transform to obtain the time domain response map. The position with the largest response value is the tracking prediction position.

[0036] Tracking template update: After predicting the position of the target, the target template and related filter templates are updated based on the position coordinate information. The update process is the same as the training process in step a).

[0037] Repeat steps b) and c) for continuous tracking.

[0038] During the tracking process, the average peak correlation energy (APEC) is continuously output as a threshold for judging the tracking status. If APEC < 12, the tracking status is judged as lost; if APEC ≥ 12, the tracking status is judged as normal. If the tracking status is normal, update the target position information and output the miss distance; If the tracking status is lost, report that the tracked target is lost and set the miss distance to ΔX: 4095, ΔY: 4095.

[0039] Furthermore, the DSP integrates a memory tracking module. For example... Figure 5The diagram shows the processing flowchart of the tracking algorithm of this invention. When the APCE value is lower than the preset threshold of 12 for several consecutive frames, the system determines that the target has been lost, and the DSP starts the memory tracking module (i.e., the functional unit in the software that implements the KCF tracking logic, including sub-modules such as initialization, tracking, updating, and status judgment). Based on the motion trajectory information of the target in the 10 consecutive frames before the target was lost, this module uses the Kalman filter algorithm to predict the position and continuously outputs the predicted miss distance within a configurable time window of 0.5 seconds to 2 seconds to maintain the servo system's pointing tracking of the target.

[0040] Application Processing Unit (ARM): Utilizes the HiSilicon Hi3559AV100 chip, which integrates a dedicated Neural Network Inference Engine (NNIE). The ARM unit runs the Linux operating system and receives full-resolution images distributed by the FPGA via the LVDS interface. Figure 4 The diagram shows the ARM processing flowchart of the present invention. In this embodiment, the ARM unit deploys an improved target detection model based on YOLOv4-tiny. This model adds a deep feature layer with stride=64 at the end of the original YOLOv4-tiny network to extract deep features with high semantic information. Compared with shallow features, this layer has a stronger class discrimination ability and significantly improves the recognition accuracy of small targets and complex backgrounds.

[0041] The specific deployment process of YOLOv4-tiny on NNIE is as follows: Developed based on the YOLO series algorithms, using the YOLOV4-tiny model, the trained model is quantized and converted using the HiSilicon conversion tool nnie_mapper, and then deployed by calling the NNIE (Neural Network Inference Engine) interface of the Hi3559AV100 platform.

[0042] That is, the network structure of the improved YOLOv4-tiny model is as follows: Figure 6 As shown, this model, based on the standard YOLOv4-tiny, adds a deep feature layer with stride=64 at the end of the network to extract deep features with high semantic information. This layer has stronger class discrimination ability compared to shallow features, significantly improving the recognition accuracy of small targets and complex backgrounds. The ARM forwards the recognized target location information to the DSP via the FPGA through the UART interface to guide or update target tracking.

[0043] Meanwhile, the specific structure of the improved YOLOv4-tiny-based model network used in this function is as follows: Figure 7As shown, its basic network for image feature extraction consists of multiple CSP components. Its improvement mainly involves adding a deep feature layer with a stride of 64 for deeper target feature extraction. Compared with shallow features, the features of this layer have stronger semantic information and can better distinguish the type of the target.

[0044] The deployment process includes: (a) Model training: Train the improved YOLOv4-tiny model on a custom dataset.

[0045] (b) Model conversion: Use the HiSilicon official tool nnie_mapper to perform INT8 quantization on the trained .caffemodel or .onnx model; (c) Generate an NNIE executable model file (.wk format) adapted to the Hi3559AV100 platform.

[0046] (d) Hardware deployment: (1) Load the .wk model by calling the NNIE inference engine API of Hi3559AV100; (2) Input a preprocessed image (such as scaling and normalization), and perform forward inference through NNIE; (3) The post-processing output detection box (including category, confidence level, and coordinates) is used by the subsequent tracking module.

[0047] In summary, YOLOv4-tiny was adopted as the basic object detection model, and its lightweight structure is suitable for embedded platforms.

[0048] Key improvements: A deep feature layer with stride=64 is added at the end of the network to extract deep features with high semantic information. Compared with shallow features, this layer has stronger class discrimination ability, significantly improving the recognition accuracy of small targets and complex backgrounds.

[0049] This embodiment provides a signal processing method for an optoelectronic turntable, including the following steps: S1: Data Acquisition and Parallel Distribution; The FPGA synchronously acquires visible light and infrared dual-channel video streams, performing preprocessing operations such as decoding, YUV conversion, and DDR buffering. For example... Figure 2 As shown, the FPGA sends a 128×128 pixel tracking area image slice to the DSP via the SRIO interface; and sends a full-resolution image to the ARM via the LVDS interface for target recognition.

[0050] S2: Target recognition and location guidance; ARM calls the NNIE inference engine to perform forward inference on the input image using the improved YOLOv4-tiny model. For example... Figure 4 and Figure 6 As shown, the model, after training, INT8 quantization, and conversion using the nnie_mapper tool, generates a .wk file and is deployed on the Hi3559AV100 platform. The inference results are fused into candidate boxes using non-maximum suppression, outputting the location and category information of the target with the highest confidence. The ARM relays the coordinate information to the DSP via the FPGA through the UART interface.

[0051] S3: Target tracking and miss distance calculation; after receiving coordinate information, the DSP initializes or updates the KCF tracking window. For example... Figure 3 and Figure 5 As shown, the DSP executes the improved multi-scale KCF algorithm under a multi-core parallel architecture, including: Core 0: Extracts HOG features of the image region; Core 1: Performs Discrete Fourier Transform / Inverse Transform and Frequency Domain Correlation Calculation; Core 2: Performs tracking template updates and average peak-correlated energy (APCE) calculations.

[0052] The miss distance calculated in real time by the DSP is output to the servo controller via the UART interface of the FPGA; at the same time, the APCE value is used for tracking stability evaluation, and the "target lost" flag is triggered when APCE < 12.

[0053] Furthermore, the ARM receives tracking status and miss distance information from the DSP and overlays OSD information, including the target bounding box, motion trajectory, lock status indicator, target category, and confidence level, onto the video stream. The ARM performs H.264 / H.265 hardware encoding through the VPSS / VENC module, and outputs the encoded video stream and system status data packets through dual gigabit Ethernet ports.

[0054] S4: Memory Tracking and Position Prediction; When the APCE value remains below the preset threshold of 12 and there are multiple consecutive frames without a match, the DSP determines that the target has been lost and activates the memory tracking module. This module uses the Kalman filter algorithm to predict the target position based on the motion trajectory information of the target for N consecutive frames before the target is lost (N=10 in this embodiment), and continuously outputs the predicted miss distance within a configurable time window of 0.5 seconds to 2 seconds to maintain the servo system's pointing tracking of the target.

[0055] If the target is recaptured within the preset time window (APCE rises above the threshold), the DSP resumes normal tracking mode; if it is not captured after the timeout, the system reports a "tracking failure" status and switches to global search mode.

[0056] Furthermore, to comprehensively evaluate the actual performance of the solutions in Examples 1 and 2, a standardized testing environment was established, and multi-dimensional, multi-scenario performance tests were conducted. The test content covered core indicators such as system latency, power consumption, target recognition rate, and tracking recovery success rate.

[0057] 1. Experimental Environment a) Hardware platform: Device under test: Multi-core heterogeneous processing board (integrated FPGA, 8-core DSP C6678, ARM).

[0058] Optical turntable: Equipped with a visible light high-definition camera (1920×1080@30fps) and an uncooled infrared thermal imager (640×512@25Hz).

[0059] Servo controller: Receives the miss distance command and has a control accuracy of ≤0.01°.

[0060] Target simulator: A high-precision six-degree-of-freedom motion platform that can simulate various maneuver trajectories such as straight lines, circles, and S-shapes.

[0061] Test PC: Used for data recording, command issuance, and result analysis.

[0062] b) Software environment: The test host computer runs on a QT display and control device to achieve automated script control and data acquisition.

[0063] 1) Test scenario: Scene A (static background): The target is moving in a straight line at a low, constant speed during a clear daytime, without any obstructions.

[0064] Scenario B (Dynamic Interference): The target moves through a city street scene with dynamic interference objects such as vehicles and pedestrians.

[0065] Scene C (Brief Occlusion): The target is completely occluded by a fixed obstacle (such as a wall or tree) for 2 seconds and then reappears.

[0066] Scene D (Low Signal-to-Noise Ratio): At night or in dense fog, the contrast between the target and the background is extremely low.

[0067] 2. Test Items and Data Recording 3. Results Analysis and Comparison 3.1 System latency and power consumption Test results show that the system's average end-to-end latency is 32.5ms across all scenarios, meeting the ≤60ms requirement. Specifically, FPGA data distribution takes approximately 5ms, ARM identification approximately 15ms, DSP tracking and prediction approximately 10ms, and the remainder is interface transmission and scheduling overhead. The system's average power consumption under full load is 13.8W, below the 15W design limit, demonstrating the energy efficiency advantages of the multi-core heterogeneous architecture.

[0068] 3.2 Target Recognition Rate Thanks to the addition of a stride=64 deep feature layer in the improved YOLOv4-tiny model, the system's ability to identify small targets is significantly enhanced. Overall, the system still achieved a recognition rate of 88.7% in the most challenging low signal-to-noise ratio scenarios, with an overall average recognition rate of 92.8%, which is better than the acceptance criteria.

[0069] 3.3 Tracking robustness and recovery capability The effectiveness of the KCF algorithm's multi-core parallelization and APCE criterion was fully verified in the tests. In 100 tests of Scene C (2-second occlusion), the system successfully recovered tracking 97 times, achieving a recovery success rate of 97%, far exceeding the technical target of 95%.

[0070] Conclusion: The test data fully demonstrates that the present invention, through the deep integration of multi-core heterogeneous architecture, the improved YOLOv4-tiny model, and the KCF tracking strategy with memory function, successfully achieves the design goals of high real-time performance, low power consumption, high recognition rate, and strong robustness. All key performance indicators meet or exceed industry standards and are significantly better than traditional discrete architecture solutions.

[0071] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A signal processing system for an optoelectronic turntable based on a multi-core heterogeneous architecture, characterized in that, include: An FPGA is configured to acquire at least two raw video data streams from a photoelectric sensor and preprocess the raw video data, the preprocessing including at least decoding, YUV format conversion, and buffering; the FPGA is also configured to distribute the preprocessed first image data to a multi-core DSP via a first dedicated data path, and to distribute the preprocessed second image data to an ARM in parallel via a second dedicated data path; the first dedicated data path is a serial RapidIO interface, and the second dedicated data path is an LVDS interface; The multi-core DSP is configured to receive the first channel image data through the serial RapidIO interface, execute the target tracking algorithm in parallel to calculate the miss distance, and output the miss distance to the servo control system through the FPGA. The multi-core DSP is internally configured to perform target tracking tasks in parallel across multiple cores, including: The first kernel is used to extract HOG features from image data; The second core is used to perform discrete Fourier transform / inverse transform and correlation calculation; The third core is used to perform tracking template updates and average peak-related energy calculations. The multi-core DSP also includes a memory tracking module, configured to predict the position based on the motion state before the target is lost when the average peak correlation energy value is lower than a preset threshold, and maintain the miss distance output within a preset time window; the memory tracking module uses a Kalman filter algorithm to predict the target position based on the motion trajectory information of N consecutive frames before the target is lost. The ARM, which integrates a neural network inference engine, is configured to receive the second image data via the LVDS interface and run a lightweight target detection model after INT8 quantization. The model is a network structure based on an improved YOLOv4-tiny architecture, including a newly added stride=64 deep feature layer, used to perform target detection and classification on the image data and generate target location information. The ARM is also configured to send the target location information to the multi-core DSP to guide or update target tracking. A bidirectional status interaction channel is located between the multi-core DSP and the ARM, used to transmit tracking status information and identification control commands. The tracking status information includes the average peak correlation energy value. The ARM dynamically adjusts the identification strategy based on the received average peak correlation energy value, forming a bidirectional closed loop of identification and tracking.

2. The photoelectric turntable signal processing system according to claim 1, characterized in that: The multi-core DSP is also configured to calculate the average peak correlation energy value in real time and compare it with a preset threshold to determine the tracking status; the preset threshold is 12, and when the average peak correlation energy value is lower than 12, it is determined that the target is lost.

3. The photoelectric turntable signal processing system according to claim 1, characterized in that: The preset time window of the memory tracking module is 0.5 seconds to 2 seconds. Within this time window, the predicted miss distance is continuously output. If the target is recaptured within the time window, the normal tracking mode is restored; otherwise, a tracking failure status is reported.

4. The photoelectric turntable signal processing system according to claim 1, characterized in that: The ARM dynamically adjusts the recognition strategy based on the received average peak correlation energy value, including: when the average peak correlation energy value is higher than a preset threshold, the ARM shrinks the recognition area to the periphery of the tracking window to reduce computing power consumption; when the average peak correlation energy value is lower than the preset threshold, the ARM resumes full-image detection to improve the target re-capture probability.

5. The photoelectric turntable signal processing system according to claim 1, characterized in that: The system is also configured to enter search mode or report the loss status after the target is lost and the timeout period can be configured to 2 seconds.

6. A photoelectric turntable signal processing method, applied to the system described in any one of claims 1-5, characterized in that; Includes the following steps: Step S1: The FPGA acquires multiple video data in parallel, performs preprocessing, and then distributes the first image data to the multi-core DSP through the serial RapidIO interface, and distributes the second image data to the ARM in parallel through the LVDS interface. Step S2: The ARM performs target recognition based on the neural network inference engine. The improved YOLOv4-tiny model with INT8 quantization is used to detect and classify targets in the image. The target location and category information with the highest confidence are output through non-maximum suppression. The target location information is sent to the multi-core DSP. Step S3: The multi-core DSP performs target tracking based on the target position information or the tracking status of the previous frame. The improved multi-scale kernel correlation filtering algorithm is run in parallel by multiple cores to calculate and output the miss distance in real time. At the same time, the average peak correlation energy value is calculated to evaluate the tracking stability. When the average peak correlation energy value is lower than the preset threshold, it is determined that the tracking has lost lock. Step S4: When the tracking loss judgment is triggered, the multi-core DSP starts the memory tracking module, predicts the target position based on the Kalman filter algorithm, and continuously outputs the predicted miss amount within a preset time window of 0.5 seconds to 2 seconds to maintain the servo system's pointing tracking of the target; the ARM dynamically adjusts the recognition strategy according to the received average peak correlation energy value to form a two-way closed loop of recognition and tracking.

7. The photoelectric turntable signal processing method according to claim 6, characterized in that: The improved YOLOv4-tiny model described in step S2 includes a newly added deep feature layer with stride=64, which is used to extract deep features with high semantic information to improve the ability to recognize small targets.

8. The photoelectric turntable signal processing method according to claim 6, characterized in that: The preset threshold mentioned in step S3 is 12. When the average peak correlation energy value is lower than 12, it is determined that the tracking has lost lock.

Images