An adaptive target tracking system based on embedded intelligent heterogeneous computing

Abstract

The application provides an adaptive target tracking system and method based on embedded intelligent heterogeneous computing, which is used for identifying and extracting a target from multiple original images to track the target. The system comprises a computing device and a master device, wherein the computing device is configured with multiple embedded intelligent heterogeneous computing board cards to improve its computing power, and in the processing process, the extraction range is adaptively adjusted according to the distance of the target, and all pixels in the range are extracted from the original image to form an accurately segmented target image. The technical scheme of the application solves multiple problems such as limited computing power, limited data transmission rate, inaccurate front-end image dynamic segmentation and target extraction of the existing edge embedded intelligent image target computing scheme by combining the high-performance hardware architecture design and the innovative image processing mode.

Description

Technical Field

[0001] This invention relates to the field of target tracking technology, specifically to the cross-disciplinary field of embedded intelligent heterogeneous computing and target tracking, and more specifically, to an adaptive target tracking system based on embedded intelligent heterogeneous computing. Background Technology

[0002] With the development of artificial intelligence, AI chips, and edge embedded intelligent computing technology, edge embedded intelligent computing is driving equipment across various industries from "automation" to "intelligence," and from "informatization" to "intelligentization." It integrates powerful computing capabilities (hereinafter referred to as computing power) and AI algorithm models into the equipment, transforming it from a simple execution terminal into an intelligent entity with autonomous perception, decision-making, and collaborative capabilities. However, existing edge embedded intelligent image target computing solutions still face many core technological bottlenecks that urgently need to be overcome when applied in industrial or military equipment, including:

[0003] High-performance computing bottleneck. Complex deep learning models (such as convolutional neural network models used for object detection, recognition, and segmentation) have extremely high requirements for fixed-point / floating-point computing power when running on the edge. However, the computing power of existing general-purpose X86, ARM, FPGA, and DSP architecture processors is difficult to meet the algorithm's requirements for trillion-times-per-second (TOPS) and above computing performance under strict power consumption and size constraints.

[0004] The bottleneck of high-speed, high-bandwidth bus interconnection and transmission for large-scale images. In edge intelligent sensing computing, there are a large number of image / video data I / O interaction requirements. The bandwidth and latency of traditional communication interfaces such as GbE Ethernet and USB interfaces have become bottlenecks for improving system performance.

[0005] Bottlenecks exist in dynamic image segmentation and adaptive target extraction in complex scenes. In target perception and tracking scenarios, complex background interference such as clouds and mountains often exists in the image. Existing embedded vision processing solutions typically transmit the complete original image directly to the CPU or NPU for target detection and tracking, resulting in a large amount of invalid background data occupying bus bandwidth during transmission and wasting computing resources. In addition, traditional image segmentation algorithms generally use fixed-size image blocks for processing, which cannot adapt to changes in target pixel scale caused by changes in target distance, and may also lead to too much invalid background in the segmented target image.

[0006] In summary, existing edge-embedded intelligent image target computing solutions suffer from multiple problems, including limited computing power, limited data transmission rates, and inaccurate front-end image dynamic segmentation and target extraction, resulting in low efficiency in target recognition and extraction. Therefore, a heterogeneous intelligent computing solution combining hardware and software is urgently needed to support large-scale image data transmission and achieve accurate dynamic image segmentation and adaptive target region extraction in complex background scenes.

[0007] It should be noted that the background information presented here is only for illustrating relevant information about the present invention to aid in understanding the technical solution of the present invention, and does not imply that the relevant information is necessarily prior art. The relevant information was submitted and disclosed together with the present invention, and should not be considered prior art unless there is evidence that the relevant information was disclosed before the filing date of the present invention. Summary of the Invention

[0008] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide an adaptive target tracking system and method based on embedded intelligent heterogeneous computing.

[0009] The objective of this invention is achieved through the following technical solution:

[0010] According to a first aspect of the present invention, an adaptive target tracking system based on embedded intelligent heterogeneous computing is provided. This system is used to identify and extract a target from multiple original images for tracking the target, each original image carrying its corresponding target distance information. The system includes a computing device and a main control device, wherein:

[0011] The computing device includes multiple embedded intelligent heterogeneous computing boards, one communication board, and a connection backplane. Each embedded intelligent heterogeneous computing board is used to adaptively adjust the extraction range according to preset rules based on the distance of the target in each original image, and extract all pixels within that range from the original image to form an accurately segmented target image. The connection backplane is used to connect each embedded intelligent heterogeneous computing board and the communication board respectively. The communication board is used to realize data transmission between the computing device and the main control device.

[0012] The main control device is used to send multiple raw images to the computing device for processing, and to receive the target image corresponding to each raw image to synthesize target tracking video.

[0013] This solution achieves at least the following beneficial technical effects: 1. The system adjusts the extraction range to different sizes according to targets at different distances, overcoming the problem of traditional methods struggling to simultaneously ensure the segmentation accuracy of both near and far targets when extracting target images. 2. Higher segmentation accuracy allows for smaller final extracted target image sizes, avoiding the transmission of large amounts of useless background data during target image transmission, thereby saving communication resources and reducing communication bottlenecks.

[0014] Optionally, each embedded intelligent heterogeneous computing board is configured to process each original image according to preset rules as follows:

[0015] The target binary mask image is generated for each original image using the image grayscale statistical binarization segmentation method. The size of the grayscale statistical window is adaptively adjusted according to the distance to the target, and the size of the grayscale statistical window is smaller as the target distance increases.

[0016] The target is identified and segmented from the corresponding original image based on the target binary mask image to determine the extraction range.

[0017] Extract all pixels within this range from the original image to compose an accurately segmented target image.

[0018] Optionally, each embedded intelligent heterogeneous computing board has three different grayscale statistical window sizes (3×3, 5×5, and 7×7) and near-distance and far-distance thresholds preset. Each embedded intelligent heterogeneous computing board is configured to adaptively adjust the size of the grayscale statistical window by comparing the target distance with the near-distance and far-distance thresholds. Specifically: if the target distance is greater than or equal to the preset far-distance threshold, the size of the grayscale statistical window is adjusted to 3×3; if the target distance is less than or equal to the preset near-distance threshold, the size of the grayscale statistical window is adjusted to 7×7; otherwise, the size of the grayscale statistical window is adjusted to 5×5.

[0019] Optionally, each embedded intelligent heterogeneous board is configured with a CPU unit, an FPGA unit, and an NPU unit, wherein: the CPU unit is used to schedule image data across devices, boards, and units; the FPGA unit is configured to preprocess the original image to generate a target binary mask image and target location and size, the preprocessing including median filtering for noise reduction, image grayscale statistical binarization segmentation, image connected component parsing, and target location marking and size parameter extraction; the NPU unit is configured with an image target detection model, which is used to identify and extract targets from the original image based on the target location and size generated by the FPGA unit.

[0020] This solution can achieve at least the following beneficial technical effects: through the combination of high-speed interconnect hardware architecture, heterogeneous computing boards, and FPGA preprocessing algorithms, a high-efficiency and accurate target recognition solution can be achieved through software and hardware collaboration.

[0021] Optionally, the FPGA unit is configured with a raw image circular buffer queue and a target binary mask image circular buffer queue, wherein: the raw image circular buffer queue includes multiple sequentially arranged buffer areas, wherein each raw image is sequentially written to a different buffer area for the CPU to read, and the CPU reads the raw image from the previously written buffer area; the target binary mask image circular buffer queue includes multiple sequentially arranged buffer areas, wherein each target binary mask image generated by the FPGA unit is sequentially written to a different buffer area for the CPU to read, and the CPU reads the target binary mask image from the previously written buffer area.

[0022] This solution can achieve at least the following beneficial technical effects: it can avoid data conflicts caused by image writing and reading access to the same storage area, thus solving the problems of frame loss and screen tearing in high frame rate video transmission scenarios.

[0023] Optionally, the FPGA unit is configured to perform median filtering denoising as follows:

[0024] T1. Set a sliding filter window in the original image.

[0025] T2. The sliding filter window traverses the entire original image row by row and column by column. Each slide performs the following processing: T21. Traverse all pixels within the current coverage area of ​​the filter window to obtain the grayscale value of each pixel, and sort the grayscale values ​​of all pixels within the current coverage area in ascending order. T22. Take the median value from all the ascending-sorted grayscale values ​​and assign it to the center pixel within the current coverage area of ​​the filter window.

[0026] Optionally, the FPGA unit is configured to perform image grayscale statistical binarization segmentation based on two-level grayscale statistics, including:

[0027] S1. Construct a pixel operation matrix centered on the pixel to be processed. It includes a central sub-block, multiple inner sub-blocks distributed adjacent to the central sub-block, and multiple outer sub-blocks distributed adjacent to all inner sub-blocks. The size of all sub-blocks is equal to the size of the grayscale statistics window.

[0028] S2. The sliding grayscale statistics window calculates the inner grayscale statistics value corresponding to the area occupied by all inner layer sub-blocks and the outer grayscale statistics value corresponding to the area occupied by all outer layer sub-blocks, and uses the product of the inner grayscale statistics value and the outer grayscale statistics value as the pixel feature value of the pixel to be processed.

[0029] S3. Calculate the pixel feature values ​​of all pixels in the original image to obtain the feature response map, and calculate the global mean and global standard deviation of the feature response map, and calculate the adaptive segmentation threshold as follows:

[0030]

[0031] Where T represents the adaptive segmentation threshold; The global mean of the characteristic response map; The empirical coefficient corresponding to the global mean of the characteristic response map is 2; The global standard deviation representing the characteristic response map; The empirical coefficient corresponding to the global standard deviation of the characteristic response map is 40.

[0032] S4. Compare each pixel feature value with the adaptive segmentation threshold one by one to generate the target binary mask image.

[0033] This scheme can achieve at least the following beneficial technical effects: the dual-level image grayscale statistical binarization segmentation divides the pixels surrounding the pixel to be processed into two layers (inner and outer layers) to calculate the grayscale statistical value of each layer and then further calculates the pixel feature value. Compared with the traditional binarization segmentation based on the single-layer grayscale statistical value to calculate the pixel feature value, this method can more accurately distinguish large-scale background local interference such as large-scale cloud layers and mountainous backgrounds, and avoid over-segmentation, missed detection, and high false alarm rate that are prone to occur in complex scenes.

[0034] Optionally, in S2, the inner layer grayscale statistics and outer layer grayscale statistics are calculated as follows: S21, calculate the grayscale mean of each sub-block. S22, calculate the absolute difference between the grayscale mean of the central sub-block and the grayscale mean of each inner layer sub-block, and multiply the absolute differences calculated on the same pair of paths to obtain the median value on each pair of paths, and take the minimum median value as the inner layer grayscale statistics; where the pair of paths refers to the line connecting the center pixel of the central sub-block and the center pixel of each inner layer sub-block. S23, take the minimum grayscale mean among all outer layer sub-blocks as the outer layer grayscale statistics.

[0035] This scheme can achieve at least the following beneficial technical effects: the inner and outer gray-scale statistical values ​​calculated using this scheme can be used in subsequent calculations to make the final target segmentation result more accurate.

[0036] Optionally, the FPGA unit is configured to perform image connectivity analysis as follows: M1. Perform a full-frame scan of the target binary mask image row by row and pixel by pixel to identify all bright pixels, where a bright pixel is defined as a pixel with a brightness value of 1. M2. Using an eight-neighbor connectivity rule, spatially adjacent bright pixels are grouped into the same connected component, and a unique ID is assigned to each connected component as the target ID corresponding to that connected component. M3. Compare the number of pixels in each connected component with a preset minimum pixel count threshold, retaining connected components with a pixel count greater than or equal to the minimum pixel count threshold as valid connected components, and discarding the remaining connected components.

[0037] Optionally, the FPGA unit is configured to perform target location marking and size parameter extraction in the following manner:

[0038] Traverse all pixels within each valid connected component to determine the center pixel coordinates of each valid connected component, and calculate the minimum horizontal coordinate, maximum horizontal coordinate, minimum vertical coordinate, and maximum vertical coordinate of each pixel within each valid connected component to solve for the size of the bounding rectangle of that valid connected component; where the difference between the maximum horizontal coordinate and the minimum horizontal coordinate is the target width, and the difference between the maximum vertical coordinate and the minimum vertical coordinate is the target height.

[0039] The target ID, center pixel coordinates, and bounding rectangle size of each valid connected component are packaged and cached to complete the target location marking and size parameter extraction.

[0040] Optionally, the communication board is a RapidIO bus to fiber optic communication board, which is configured with multiple optical modules, each of which serves an embedded intelligent heterogeneous board. The main control device is configured with multiple fiber optic-RapidIO to PCIe bus communication boards, each of which is connected to an optical module on the RapidIO bus to fiber optic communication board via a fiber optic link.

[0041] This solution can achieve at least the following beneficial technical effects: by designing a separate optical communication link to the main control device for each embedded intelligent heterogeneous board through multiple optical modules and multiple fiber-optic RapidIO to PCIe bus communication boards, the data transmission speed can be improved and mutual interference can be avoided.

[0042] Optionally, the connection backplane is a standard VPX backplane, which includes one RapidIO bus to fiber optic communication board slot and multiple embedded intelligent heterogeneous board slots, wherein: each embedded intelligent heterogeneous board slot is configured with a power interface, a RapidIO interface, a Camera Link interface, a network port, and a serial port; the RapidIO bus to fiber optic communication board slot is configured with a power interface, a RapidIO interface, a network port, and a serial port; wherein the same type of interface between the RapidIO bus to fiber optic communication board slot and the embedded intelligent heterogeneous board slot is connected through a backplane circuit.

[0043] Optionally, the standard VPX backplane is 3U in size and is equipped with one RapidIO bus to fiber optic communication board slot and three embedded intelligent heterogeneous board slots.

[0044] This solution can achieve at least the following beneficial technical effects: the 3U size can provide enough embedded intelligent heterogeneous board slots to ensure the computing power of the system, and it is also easy to deploy in mobile vehicles, ships and other equipment, making it more practical.

[0045] According to a second aspect of the present invention, an image target tracking method is provided, the method comprising the following steps: M1, acquiring multiple original images; M2, inputting all the original images into an adaptive target tracking system based on embedded intelligent heterogeneous computing as described in the first aspect of the present invention, to obtain a target tracking video.

[0046] Compared with existing technologies, the advantages of this invention are as follows: the solution of this invention effectively solves the problems of computing power bottleneck, large-scale image data transmission bottleneck, real-time image segmentation and target extraction accuracy bottleneck in existing edge embedded intelligent image target computing solutions by combining heterogeneous hardware architecture with innovative image processing methods. It improves the efficiency of target recognition and extraction, and provides an effective solution for edge intelligent perception, intelligent tracking and intelligent decision-making tasks, and has good prospects for practical application and promotion. Attached Figure Description

[0047] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0048] Figure 1 This is a schematic diagram of the system composition of an adaptive target tracking system based on embedded intelligent heterogeneous computing according to an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the system architecture of an adaptive target tracking system based on embedded intelligent heterogeneous computing according to an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the embedded intelligent heterogeneous computing board architecture of the adaptive target tracking system based on embedded intelligent heterogeneous computing according to an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the preprocessing process of the original image by the FPGA unit of the adaptive target tracking system based on embedded intelligent heterogeneous computing according to an embodiment of the present invention.

[0052] Figure 5 This is a schematic diagram of the RapidIO bus-to-fiber optic communication board architecture of an adaptive target tracking system based on embedded intelligent heterogeneous computing according to an embodiment of the present invention.

[0053] Figure 6 This is a schematic diagram of the 3U, 4-slot standard VPX backplane architecture of an adaptive target tracking system based on embedded intelligent heterogeneous computing according to an embodiment of the present invention.

[0054] Figure 7 This is a schematic diagram of the fiber-optic-RapidIO to PCIe bus communication board architecture of an adaptive target tracking system based on embedded intelligent heterogeneous computing according to an embodiment of the present invention.

[0055] Figure 8 This is a schematic diagram of a median filtering denoising method for an adaptive target tracking system based on embedded intelligent heterogeneous computing according to an embodiment of the present invention;

[0056] Figure 9 This is a schematic diagram of the adaptive switching method of the processing kernel parameters based on the target distance in an adaptive target tracking system based on embedded intelligent heterogeneous computing according to an embodiment of the present invention.

[0057] Figure 10 This is a schematic diagram of an image dual-level grayscale statistical binarization segmentation method for an adaptive target tracking system based on embedded intelligent heterogeneous computing according to an embodiment of the present invention;

[0058] Figure 11 This is a schematic diagram of the image connected component analysis and target position marking and size parameter extraction method of the adaptive target tracking system based on embedded intelligent heterogeneous computing according to an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0060] As mentioned in the background section, existing edge-embedded intelligent image target computing solutions suffer from multiple problems, including limited computing power, limited data transmission rate, inaccurate front-end image dynamic segmentation, and target extraction, resulting in low efficiency in target recognition and extraction.

[0061] The inventor analyzed each of the above problems one by one:

[0062] First, the limitation of computing power mainly stems from the contradiction between the constraints of deployment space and power consumption at the edge and the scale of data. The edge needs to process large-scale image data in real time (especially high-definition visible light, infrared, and SAR synthetic aperture image / video stream data), but edge locations such as vehicles, ships, and high-altitude observation posts lack sufficient space and power for deploying computing equipment, and therefore do not possess the massive computing power of a central computing center to process such large-scale image data. Therefore, the application of intelligent heterogeneous computing architectures (such as CPU+FPGA+ASIC, CPU+dedicated AI acceleration IP cores) should be considered at the edge. This architecture can provide high-performance and high-capacity support for edge intelligent perception, intelligent tracking, and intelligent decision-making tasks.

[0063] Secondly, data transmission rate limitations mainly include intra-device interconnect bandwidth bottlenecks, inter-device collaboration latency, and transmission distance constraints. Intra-device interconnect bandwidth bottlenecks refer to the need for high-speed data exchange between multiple computing units (such as multiple AI accelerator cards), storage units, and control units within a single chassis or device. Insufficient interconnect bandwidth creates a "data bottleneck," causing powerful computing units to become idle due to "starvation." Inter-device collaboration latency and transmission distance constraints refer to the extreme requirements placed on the bandwidth latency and physical transmission distance constraints of interconnect buses in equipment systems composed of multiple device nodes (such as payload equipment and PC main control computing equipment), where devices need to transmit large amounts of raw or intermediate processed data via high-speed buses. To address these constraints, technologies such as inter-board RapidIO (Serial RapidIO), PCIe (PCI Express), and inter-device fiber optic interconnects are needed to build long-distance, interference-resistant, high-bandwidth, and low-latency internal and external data transmission channels to truly unleash the collaborative performance of multi-card computing clusters.

[0064] Finally, one of the key reasons for the inaccuracy of dynamic image segmentation and target extraction is that traditional solutions do not incorporate target distance information into their processing. Specifically, distant targets have a small pixel ratio, and the fixed-size block processing method in traditional solutions easily introduces background noise; while close targets have a large pixel ratio, and the fixed-size block processing method struggles to extract the target contour completely. This leads to poor target-background segmentation in complex backgrounds, directly reducing the stability of subsequent target extraction and tracking. Furthermore, most existing solutions implement segmentation algorithms at the CPU software level, but CPUs are not adept at handling this type of computation. Therefore, it is necessary to consider utilizing the parallel computing capabilities of FPGAs, combined with target distance information measured by external ranging devices such as lasers and radar, to achieve real-time and accurate extraction of suspected target regions from the source image. This avoids the transmission and computation of massive amounts of invalid background data, thereby improving the real-time performance and stability of intelligent target recognition and tracking in heterogeneous computing systems.

[0065] Based on the above analysis, this invention provides an adaptive target tracking system based on embedded intelligent heterogeneous computing to overcome the shortcomings of existing edge embedded intelligent computing solutions. This system is a high-performance, high-capacity intelligent heterogeneous computing platform with long-distance, anti-interference, high-speed bus communication capabilities and adaptive image target recognition and extraction capabilities. It can be applied to edge intelligent perception, intelligent tracking, and intelligent decision-making tasks.

[0066] In summary, the system provided by this invention is used to identify and extract a target from multiple original images for tracking the target, each original image carrying its corresponding target distance information. (See reference...) Figure 1 The system includes computing devices and main control devices, wherein:

[0067] The computing device includes multiple embedded intelligent heterogeneous computing boards, one communication board, and a connection backplane. Each embedded intelligent heterogeneous computing board adaptively adjusts its extraction range according to preset rules based on the distance to the target in each original image, and extracts all pixels within that range from the original image to form an accurately segmented target image. The connection backplane connects each embedded intelligent heterogeneous computing board and the communication board respectively. The communication board enables data transmission between the computing device and the main control device.

[0068] The main control device is used to send multiple raw images to the computing device for processing, and to receive the target image corresponding to each raw image to synthesize target tracking video.

[0069] The system of this invention features multiple computing boards, enabling high-performance intelligent computing. Furthermore, the system adaptively adjusts the image extraction range to different sizes based on targets at varying distances, overcoming the difficulty of simultaneously ensuring accurate segmentation of both near and far targets in traditional methods. Higher segmentation accuracy results in smaller final extracted target images, avoiding the transmission of large amounts of useless background data when transmitting the target image to the main control device. This saves communication resources, reduces communication bottlenecks, and significantly reduces the computational load on the main control device.

[0070] To better understand this invention, the following detailed description, in conjunction with specific embodiments, covers both hardware architecture and image processing methods. It should be noted that some hardware units or components (such as embedded intelligent heterogeneous computing boards, optical modules, connection backplane slots, communication boards on the main control device, etc.) are shown in specific quantities in the accompanying drawings. This is for easier understanding of the drawings and the invention, and does not imply that these hardware units or components can only be configured in the quantities shown in the drawings. In actual deployment, they can be flexibly increased or decreased according to the computing needs of the scenario.

[0071] I. Introduction to Hardware Architecture.

[0072] According to one embodiment of the present invention, the overall architecture of the system is as follows: Figure 2As shown: The main control device preferably has multiple fiber-optic-RapidIO to PCIe bus communication boards, while the computing device's communication board is preferably a RapidIO bus to fiber-optic communication board. Each RapidIO bus to fiber-optic communication board is connected to a single fiber-optic-RapidIO to PCIe bus communication board via a fiber optic link. Furthermore, each RapidIO bus to fiber-optic communication board is configured with multiple optical modules, each logically serving a single embedded intelligent heterogeneous computing board to avoid bandwidth bottlenecks and interference, and connecting to a single fiber-optic-RapidIO to PCIe bus communication board. In addition to the fiber optic links, serial and network cable links are also configured between the main control device and the RapidIO bus to fiber-optic communication boards. The fiber optic links are primarily used for transmitting image data to leverage their high bandwidth and low latency advantages; the serial and network cable links are mainly used by the main control device for configuring and maintaining the computing device. Inside the computing device, each embedded intelligent heterogeneous computing board is connected to a RapidIO bus to fiber-optic communication board via a connection backplane (preferably a standard VPX backplane). It should also be noted that the raw images processed by the computing device can come not only from the main control device, but also from external image acquisition devices (also known as photoelectric payloads, such as infrared and visible light cameras). In this case, a Camera Link interface needs to be configured on the embedded intelligent heterogeneous computing board to connect to the external image acquisition device.

[0073] The architecture and functions of each hardware component in the system are described below.

[0074] (I) Embedded intelligent heterogeneous computing board.

[0075] like Figure 3 As shown, the hardware components of the embedded intelligent heterogeneous computing board include a CPU processor unit, an FPGA (Field-Programmable Gate Array) unit, an NPU (Neural Processing Unit), a high-speed RapidIO-to-PCIe bus communication unit, a power supply module, and a VPX connector unit. An ARM processor is preferred as the CPU, which, combined with the FPGA and NPU, forms the core computing unit. High-speed RapidIO-to-PCIe bus interconnect communication technology is used to interact with the backplane. Specifically:

[0076] The hardware components of the CPU processor unit include an ARM CPU processor (such as the FT2000 / 4 microprocessor), DDR4 memory, a network port module and a serial port module, as well as peripheral devices (such as NOR FLASH memory, a 48MHz LVCMOS clock module, an 88E1111 PHY chip network port module, etc.). The embedded ARM CPU processor unit achieves high-speed interconnection with the FPGA (Field-Programmable Gate Array) unit and the NPU unit through built-in two PCIe x16 buses. The embedded ARM CPU processor unit acts as the PCIe RC root node, while other units act as PCIe Endpoint nodes. The embedded ARM CPU processor unit serves as the main control unit, responsible for running the embedded Linux operating system, intelligent computing middleware, RapidIO driver, and data transmission application software. Its role in the system is threefold: first, to enable high-speed RapidIO bus access and caching of external image / video data streams; second, to enable high-speed PCIe bus data communication with FPGA field-programmable gate array units; and third, to enable real-time distribution and scheduling of cached image data streams in memory for processing by FPGA and NPU units, thereby fully utilizing the high-efficiency, high-computing-power multi-core computing resources of FPGA and NPU to achieve high-throughput, high-real-time inference computing.

[0077] The hardware components of an FPGA (Field-Programmable Gate Array) unit include an FPGA main chip (preferably an FPGA K-series or higher chip, such as the SMQ7K325TFFG900), DDR3 memory circuitry (e.g., using four SM41J256M16M chips with a single 256M×16bit specification to form a 2GB high-capacity cache memory), PCIe interconnect interface circuitry (e.g., one PCIe 2.0 x4 interconnect interface circuit), a high-speed GTX interface, a high-speed Camera Link image transmission interface (optional, such as using GM8284 to realize high-speed video data access and video data decoding for external photoelectric image sensor payloads), a low-speed RS422, a clock, and a power supply module. Internally, the FPGA interconnects with the embedded ARM CPU processor unit via a high-speed PCIe x4 bus, and externally provides one or more high-speed Camera Link interfaces. The system includes an optional Link image transmission interface circuit, a high-speed GTXRapidIO high-speed bus interconnection interface, and one or more low-speed RS422 command control interfaces. All of these FPGA external interfaces are interconnected with the onboard VPX backplane via high-speed signal pins to achieve data interconnection and interoperability with external optoelectronic payload devices (i.e., image acquisition devices) and the main control device. In this invention's system, the FPGA unit is used to complete image data parsing and real-time buffering of the DDR multi-frame image circular queue, ensuring uninterrupted image transmission. Furthermore, to achieve image target and background segmentation and adaptive extraction in complex backgrounds such as clouds and mountains, this invention designs a complex scene image segmentation and target region adaptive extraction pipeline based on the parallel characteristics of FPGA hardware. The FPGA unit is configured to perform multi-level preprocessing pipeline operations on image data, including image median filtering and denoising, adaptive switching of processing kernel parameters based on target distance, image grayscale statistical binarization segmentation, image connected component parsing, target location marking, and size parameter extraction (details will be described in the image processing section below).

[0078] It should be noted that the FPGA unit cache of this invention adopts a circular queue design to provide conflict-free data read and write services to different units. Specifically, refer to... Figure 4If the system is connected to an external optoelectronic payload via Camera Link, the differential video or image signal output from the external optoelectronic payload is converted from differential to single-ended level, synchronized in time, and decoded by a decoding chip to obtain the original image data. This data is then written in real time to the original image DDR multi-frame image circular buffer queue. The queue length of this DDR multi-frame circular buffer queue is configurable. Taking a circular buffer queue length of 3 as an example, the buffer can be divided into areas A, B, and C, forming a three-frame circular buffer architecture. The principle of caching and high-speed transmission of image data is as follows: the FPGA operates in EP (End Point) mode, and the CPU operates in RC (Root Complex) mode. During image data caching and transmission, a pipelined scheduling strategy of "FPGA writing to the current buffer area and CPU reading from the previous buffer area" is adopted. The original image data is cyclically written to buffer areas A, B, and C in sequence. Simultaneously, as a complete frame of original image is written to a buffer area by the FPGA, the FPGA generates a PCIe interrupt signal to notify the CPU that "reading the image frame" is ready. After receiving the PCIe interrupt, the CPU actively reads the currently cached frame of image data into its memory via the PCIe bus. The next frame of original image is then written by the FPGA to the next buffer area in the DDR multi-frame image circular buffer queue. This method achieves pipelined parallelism between FPGA DDR high-speed caching, CPU PCIe bus high-speed reading, and caching, avoiding conflicts caused by simultaneous writing and reading of the same buffer area, thus solving frame loss and screen tearing problems in high frame rate video transmission scenarios. Furthermore, the target binary mask image generated after the FPGA performs image grayscale statistical binarization segmentation processing on the original image is also cached and transmitted using the same method.

[0079] The NPU unit, as a dedicated inference acceleration component for deep learning convolutional neural networks, provides high-performance computing services (no less than 10 TOPS, INT8) for the forward inference computation of image object detection models deployed on it. Its hardware components include a dedicated deep learning intelligent processor chip (such as the MLU series), DDR4 video memory circuitry, PCIe interconnect interface circuitry, reference clock circuitry, etc., and it is interconnected with the embedded ARM CPU processor unit via the PCIe bus.

[0080] The power module (which can use the SM4644MB DC-DC power converter) receives a 12V, 3.3V, or 5V power input from the connection backplane. After voltage conversion by the DC-DC power converter inside the power module, it provides a stable operating voltage for the entire embedded intelligent heterogeneous computing board.

[0081] The VPX connector unit is used to interconnect the entire embedded intelligent heterogeneous computing board with the connection backplane, and provides one or more high-speed Camera Link image transmission interface circuits connected to the FPGA, one high-speed GTX RapidIO high-speed bus interconnection interface, one or more low-speed RS422 command control interfaces, and interfaces such as Gigabit Ethernet and UART serial ports for the embedded ARM CPU processor unit. A 3U VPX connector conforming to the VITA46 specification (such as JVPX-21T8aAA8-A) is preferred. This connector includes ports P0, P1, and P2. Port P0 is used for power input, port P1 is used for high-speed GTX RapidIO X4 high-speed bus transmission communication interface interconnection, and port P2 is used for interconnection of one or more high-speed Camera Link interfaces, embedded ARM CPU processor unit Gigabit Ethernet ports, UART serial ports, and other high-speed bus transmission communication interfaces for the FPGA.

[0082] (ii) RapidIO bus to fiber optic communication board.

[0083] The RapidIO bus-to-fiber optic communication board is used to provide long-distance, interference-resistant fiber optic communication, RS422 serial communication, and gigabit Ethernet communication services for embedded intelligent heterogeneous computing boards. For example... Figure 5 As shown, its hardware components include a high-speed RapidIO to fiber optic communication unit, a gigabit Ethernet communication unit, a UART to RS422 communication unit, a DC-DC power supply module, a VPX connector unit, and a high-density connector debugging interface, etc. Among them:

[0084] The high-speed RapidIO to fiber optic communication unit is used to convert the electrical signals of each RapidIO X4 high-speed bus differential transmission communication interface on the VPX connector unit into optical signals. This invention uses a three-channel (adjustable according to the number of embedded intelligent heterogeneous computing boards, each channel serving one computing board) integrated optical module to provide long-distance, interference-resistant optical communication services for the system's embedded intelligent heterogeneous computing boards. Preferably, an optical module with a transmission rate of 10Gbps or higher, such as the HTQ8501A-PH-D36XX-G, is preferred. This optical module adopts a parallel transceiver design, featuring low power consumption, high speed, high density, long-distance transmission, and interference resistance. The module's electrical interfaces mainly include the transmit channel HTQ_TX_N / P[1:4] and the receive channel HTQ_RX_N / P[1:4]. Among them, the transmitting channel HTQ_TX_N / P[1:4] is interconnected at high speed with the transmitting channel RIO_TX_N / P[1:4] of each embedded intelligent heterogeneous computing module RapidIO bus X4 communication interface introduced by the standard VPX backplane, so as to realize the conversion of the electrical signal of the transmitting channel RIO_TX_N / P[1:4] of each high-speed RapidIO bus communication interface into an optical signal for transmission; at the same time, the optical signal received by the optical module is converted into the receiving channel HTO_RX_N / P[1:4] electrical signal, and transmitted through the standard VPX backplane to the external high-speed RapidIO bus communication interface RIO_RX_N / P[1:4] of the embedded intelligent heterogeneous computing module, so as to realize the interconnection and communication between the embedded intelligent heterogeneous computing module and the external host high-speed RapidIO bus.

[0085] The Gigabit Ethernet communication unit is used to achieve electrical isolation, impedance matching, and common-mode rejection between the internal Ethernet physical layer transceiver and the external network interface. Taking the LT1606-A as an example, this Gigabit Ethernet communication unit uses three high-speed network isolation transformers. One end of each network isolation transformer module is the Gigabit Ethernet port MDI[3:0] differential signal of the CPU processor unit of the computing board, and the other end is the 1000Base-T Ethernet port communication differential signal after differential-mode coupling isolation filtering by the network isolation transformer. The Gigabit Ethernet communication unit is interconnected with the high-density connector HJ30J debugging interface to provide three 1000Base-T Ethernet port communication capabilities.

[0086] The UART-to-RS422 communication unit employs a 3-channel (adjustable according to the number of embedded intelligent heterogeneous computing boards, with each channel serving one computing board) RS422 transceiver design to achieve bidirectional data communication. One end of the chip interconnects with the UART interface signal of the ARM CPU processor unit of one embedded intelligent heterogeneous computing board on the VPX connector unit. The other end provides dual + / -5.0V EIA / TIA-422 level transceiver signals and interconnects with the high-density connector HJ30J debugging interface to provide 3-channel serial port long-distance and anti-interference communication services. Taking the SM3490 full-duplex RS-485 / 422 transceiver chip as an example, this chip converts the external low-speed UART signal interface introduced through the standard VPX backplane into differential signals conforming to the RS422 standard. Simultaneously, it converts the externally input RS422 differential signal into a single-ended UART signal and transmits it to the embedded intelligent heterogeneous computing board through the standard VPX backplane.

[0087] The power module is used to filter, purify, convert, and stabilize the input power of the standard VPX backplane. Taking the SM4644MB DC-DC power module as an example, it receives a 5V power input from the backplane, and after voltage conversion by the internal DC-DC power converter, it provides a stable operating voltage for the communication board.

[0088] The VPX connector unit is used for: interconnecting with the RapidIO X4 high-speed bus differential transmission communication interface of each embedded intelligent heterogeneous computing board; interconnecting with the Gigabit Ethernet port and UART serial port of each embedded intelligent heterogeneous computing board CPU processor unit; and providing a 5V power input. This VPX connector unit preferably uses a 3U VPX connector conforming to the VITA46 specification. Port P0 of the connector is used for power input, port P1 is used for interconnecting each RapidIO X4 high-speed bus transmission communication interface, and port P2 is used for interconnecting the Gigabit Ethernet port and UART serial port communication interface of each CPU processor unit.

[0089] The high-density connector debugging interface (such as HJ30J-100ZKW2P7) adopts a high-speed micro rectangular connector to bring out the serial port and gigabit Ethernet port of the embedded intelligent heterogeneous computing board in the connection backplane, thereby providing data interaction conditions for the external host device (host computer) to develop and debug the embedded intelligent heterogeneous computing board.

[0090] (III) Standard VPX backplate.

[0091] The preferred connection backplane for this invention is a standard VPX backplane. Each board on the computing device is mounted and fixed to the standard VPX backplane via a vertical plug-in method. The RapidIO bus Mesh star network interconnection topology circuitry of the standard VPX backplane enables: point-to-point interconnection between each embedded intelligent heterogeneous computing board and the RapidIO bus-to-fiber communication board via a RapidIO network; it provides high-speed Camera Link interface communication connectors and low-speed RS422 interface communication connectors for each embedded intelligent heterogeneous computing board, which can be further interconnected with conversion connectors on the system chassis or communicate directly with the outside world; and it provides power input to each board through a 12V / 3.3V / 5V power supply topology on the standard VPX backplane.

[0092] like Figure 6 As shown, this invention preferably uses a 3U, 4-slot standard VPX backplane to simultaneously meet the computing power and small size requirements of edge deployment. The backplane is configured with three embedded intelligent heterogeneous computing board slots (Slot1, Slot2, Slot3) and one RapidIO bus-to-fiber communication board slot (Slot4). Each slot is equipped with a ruggedized hybrid electrical connector conforming to the VITA 46 specification (e.g., JVPX-21Z8aAA8-A). Through the power routing and planes inside the standard VPX backplane, the +12Vdc, +5Vdc, and +3.3Vdc voltages are routed to predefined power pins on the P0 connector of each slot (Slot1-Slot4). At the physical level, pre-defined point-to-point differential wiring is used in the corresponding high-speed connector areas (P1, P2, etc.) between Slot4 and Slots1-Slot3 to form a direct high-speed data channel, allowing the master control device node to access each embedded intelligent heterogeneous computing board at high speed. The two types of slots are interconnected via RapidIO X4 bus through port P1, and interconnected via gigabit network port and serial port through port P2.

[0093] (iv) Main control equipment (host computer).

[0094] The main control device provides more than three physical slots for the fiber optic-RapidIO to PCIe bus communication board and runs the LINUX operating system and the dedicated integrated chip driver for the RapidIO-to-PCIe bridge chip. In addition, the main control device has high-speed image / video data acquisition and caching capabilities, enabling it to distribute the cached payload image / video data stream in real time to various embedded intelligent heterogeneous computing boards via the RapidIO bus to PCIe bus bridge chip to perform related calculations for intelligent target recognition and target region extraction. Simultaneously, the main control device also receives the target image obtained after the computing device processes the original image to synthesize target tracking video.

[0095] (v) Fiber optic-RapidIO to PCIe bus communication board.

[0096] like Figure 7 As shown, this board adopts a standard half-height, half-length PCIe daughter card design, including an optical communication to RapidIO bus communication unit, a RapidIO to PCIe bus bridge unit, a PCIe x4 gold finger interface unit, and a power module, etc. Among them:

[0097] The optical communication to RapidIO bus communication unit completes the conversion between optical and electrical signals through a single transceiver optical module (such as the HTQ8501A-PH-D36XX-G type QSFP optical module), and it supports the insertion of optical fiber physical connection lines.

[0098] The RapidIO to PCIe bus bridge chip unit serves as the core module, enabling mutual conversion between RapidIO Gen2 and PCIe Gen2 protocols. It provides a 20 Gbaud line-speed conversion capability, thus offering high-bandwidth, low-latency RapidIO network system-level data transmission services based on packet switching for PCIe bus-based, ARM / X86 processor architecture master control devices. Taking the PRB0400 as the core chip of the RapidIO to PCIe bus bridge chip unit as an example, its high-speed PCIe bus interface is directly interconnected with the PCIe X4 data transceiver channel of the gold finger interface via differential PCB traces, thereby achieving high-speed interconnection and data interaction with the master control device. The chip's high-speed RapidIO X4 interface is cross-connected with the high-speed transceiver channel electrical interface of the QSFP optical module via differential PCB traces.

[0099] The PCIe x4 gold finger interface unit adopts a standard PCIe x4 interface for high-speed PCIe x4 interconnection and slot installation between the communication board and the host device.

[0100] The power module (which can use the SM4644MB DCDC) receives 12V and 3.3V power inputs through the PCIe X4 gold fingers. After conversion by the internal DCDC converter, it outputs the board's operating power supply voltage. The voltage values ​​include, but are not limited to, +1.0V for powering the PRB0400 core, +1.8V for powering the interface logic, and +1.5V for powering the reference circuit.

[0101] It should be noted that multiple fiber-optic-RapidIO to PCIe bus communication cards can be configured on the main control device to avoid communication resource squeezing or interference between different intelligent heterogeneous computing cards, thereby improving the overall processing speed of the system.

[0102] II. Introduction to Image Processing Methods

[0103] According to one embodiment of the present invention, the present invention is based on a heterogeneous system hardware architecture and adopts a matching image processing method: the original image is preprocessed using the hardware parallel characteristics of the FPGA unit, and then the preprocessed image data is handed over to the NPU unit for subsequent processing such as target recognition and extraction. Reference is still made to... Figure 4 The preprocessing operations of the FPGA unit include median filtering for noise reduction, image grayscale statistical binarization segmentation, image connected component analysis, and target location marking and size parameter extraction.

[0104] The following sections describe each operation in the preprocessing.

[0105] (I) Median filtering for noise reduction.

[0106] The purpose of median filtering denoising is to suppress random noise introduced during image acquisition and transmission while preserving complete target edge details. Median filtering denoising overcomes the drawback of traditional linear filtering denoising, which easily blurs target edges, providing a high-quality image data source for subsequent accurate segmentation and extraction of target images. Specifically, median filtering denoising is expressed as:

[0107]

[0108] Where G(x,y) represents the pixel gray value at coordinates (x,y) in the filtered output image; I(x+i,y+j) represents the pixel gray value of the original image within an N×N filtering window centered at coordinates (x,y); i,j represent the offset of the pixel within the filtering window relative to the center pixel, with an offset range of [−(N−1) / 2,(N−1) / 2], covering the entire N×N window; Mid{⋅} represents taking the median.

[0109] like Figure 8 As shown, this formula is equivalent to the following steps: T1, setting a sliding filter window in the original image ( Figure 8 (The filter window size in this example is 3×3). T2: The sliding filter window traverses the entire original image row by row and column by column. Each slide performs the following processing: T21: Traverse all pixels within the current coverage area of ​​the filter window to obtain the gray value of each pixel, and sort the gray values ​​of all pixels within the current coverage area in ascending order; T22: Take the median value from all the ascending sorted gray values ​​and assign it to the center pixel within the current coverage area of ​​the filter window.

[0110] After median filtering, the original image retains image details while significantly suppressing discrete random noise, thereby improving the signal-to-noise ratio of the original photoelectric image. Figure 8 The left image contains a large number of randomly distributed snowflake noises, while the right image has had these removed.

[0111] (ii) Image grayscale statistical binarization segmentation.

[0112] Traditional image grayscale statistical binarization segmentation methods generally use a fixed-size kernel (i.e., grayscale statistical window) to perform filtering and segmentation. The kernel size depends on manual on-site parameter tuning, so the system performance degrades rapidly after changes in operating conditions, failing to meet the engineering requirements of adaptive parameter adaptation and struggling to simultaneously extract details from small targets at a distance and contours from large targets at close range. To adapt to the segmentation needs of targets at different distances and scales, this invention innovatively introduces target distance guidance information to adaptively switch the extraction size of the Region of Interest (ROI) and the grayscale statistical window size based on the target distance. This allows for the matching of differentiated image processing kernel parameters for targets at far, medium, and near distances, thereby overcoming the technical deficiency of traditional image grayscale statistical binarization segmentation methods in balancing the segmentation accuracy of distant and near targets.

[0113] Specifically, such as Figure 9 As shown, this invention presets two levels of distance thresholds to divide the scene into long-distance, medium-distance, and short-distance scenes, and matches them with three image processing kernels: 3×3, 5×5, and 7×7, respectively. The adaptive switching logic of the processing kernel is as follows: If the target distance is greater than or equal to the long-distance threshold, it is determined to be a long-distance small-scale target scene, and the 3×3 processing kernel is automatically activated to adapt to the fine segmentation of small and weak targets in the image and improve the detection capability of long-distance small targets; if the target distance is between the short-distance threshold and the long-distance threshold, it is determined to be a medium-distance regular target scene, and the 5×5 processing kernel is automatically switched to balance the segmentation accuracy and FPGA hardware operation latency to achieve the optimal balance between accuracy and processing efficiency; if the target distance is less than or equal to the short-distance threshold, it is determined to be a short-distance large-scale target scene, and the 7×7 processing kernel is automatically activated to expand the pixel processing coverage, fully cover large-size targets, and prevent the segmentation of short-distance targets from being missed.

[0114] Furthermore, traditional image grayscale statistical binarization segmentation methods only perform single-level local grayscale comparisons, making it difficult to distinguish large-scale background interference caused by extensive cloud cover, undulating terrain, etc. In complex scenes, oversegmentation and missed detection are prone to occur, resulting in a high false alarm rate for target recognition and tracking. To address this, this invention proposes a two-level grayscale statistical binarization segmentation method to further suppress complex backgrounds and enhance target recognition, stably outputting accurate binarized images of suspected target regions, providing reliable preceding image data for ROI dynamic extraction, target segmentation, and localization tracking.

[0115] Specifically, the image two-level grayscale statistical binarization segmentation includes: S1. Constructing a pixel operation matrix centered on the pixel to be processed, which includes a central sub-block, multiple inner sub-blocks distributed adjacent to the central sub-block, and multiple outer sub-blocks distributed adjacent to all inner sub-blocks. The size of all sub-blocks is equal to the size of the grayscale statistical window. S2. Sliding the grayscale statistical window to calculate the inner grayscale statistical values ​​corresponding to the areas occupied by all inner sub-blocks and the outer grayscale statistical values ​​corresponding to the areas occupied by all outer sub-blocks, and using the product of the inner and outer grayscale statistical values ​​as the pixel feature value of the pixel to be processed. S3. Calculating the pixel feature values ​​of all pixels in the original image to obtain a feature response map, and calculating the global mean and global standard deviation of the feature response map, using the sum of the global mean and global standard deviation as the adaptive segmentation threshold. S4. Comparing each pixel feature value with the adaptive segmentation threshold one by one to generate a target binary mask image.

[0116] To facilitate understanding of two-level grayscale statistical binarization segmentation of images, a 3×3 grayscale statistical window and sub-blocks are used as an example for illustration. (Refer to...) Figure 10First, the FPGA constructs a 15×15 pixel operation matrix using row buffers and shift register chains. Using the center pixel of the matrix as the processing reference, the 15×15 matrix is ​​divided into 25 3×3 sub-blocks. One central 3×3 sub-block constitutes the central region (white area), eight 3×3 sub-blocks constitute the inner region (green area), and sixteen 3×3 sub-blocks constitute the outer region (light gray area). Then, the average pixel grayscale value of all 3×3 sub-blocks is calculated. Next, the grayscale statistics of the inner and outer layers are calculated separately. A preferred method is as follows: For the eight sub-blocks of the inner layer, four sets of paired directional combinations are formed (A1-A5, A2-A6, A3-A7, A4-A8). The absolute grayscale difference between the central sub-block and each neighboring sub-block is calculated, and the product of the absolute grayscale differences in the same paired directional direction is taken as the contrast response in that direction. The minimum contrast response in the four directions is taken as the grayscale statistics of the inner layer. This enhances the feature difference between the central target and the near-field background. For the 16 outer layer sub-blocks, the absolute grayscale difference between each sub-block and the central sub-block is calculated, and the minimum value among the 16 results is taken as the grayscale statistics of the outer layer. This characterizes the grayscale difference between the target and the distant, large-scale background. Subsequently, the grayscale statistics of the inner and outer layers are multiplied and a scaling factor is introduced for correction (the scaling factor W is optional and is used to correct the calculation deviation of the FPGA through empirical values) to obtain the pixel feature value of the central pixel. The pixel feature values ​​of all pixels in the image are calculated in the manner described above to obtain a feature response map, thereby enhancing the response of weak targets and suppressing clutter in complex backgrounds. Next, to adapt to dynamic environmental changes, this invention introduces an adaptive threshold segmentation mechanism: the global mean and standard deviation of the feature response map are statistically calculated and multiplied by the corresponding empirical coefficients (the empirical coefficients corresponding to the global mean). The empirical coefficient corresponding to the global standard deviation is 2. The adaptive segmentation threshold is calculated by multiplying the two sub-terms by an empirical coefficient (40). This threshold is dynamically adjusted based on environmental noise and background complexity to avoid false alarms and missed detections caused by a fixed threshold. Finally, the feature values ​​of each pixel in the image are compared pixel by pixel with the adaptive segmentation threshold to distinguish between suspected target candidate regions and background regions, and a binary mask image of the target is generated. This provides high-real-time, high-quality image data input for subsequent dynamic ROI extraction, connected component labeling, and target parameter extraction.

[0117] Two-level image grayscale statistical binarization segmentation divides the pixels surrounding the pixel to be processed into two layers (inner and outer layers) to calculate the grayscale statistical value of each layer, thereby further calculating the pixel feature value. Compared with traditional binarization segmentation, which calculates pixel feature value based on single-layer grayscale statistical value, this method can more accurately distinguish large-scale background local interference such as large-scale cloud layers and mountainous backgrounds, and avoid over-segmentation, missed detection, and high false alarm rate that are prone to occur in complex scenes.

[0118] (III) Image Connectivity Analysis.

[0119] Image connected component parsing involves detecting and labeling connected components in a binary mask image of a target. It obtains independent connected regions by aggregating spatially adjacent target pixels within the image and assigns a unique identifier (ID) to each connected region to complete the counting of suspected targets and spatial location differentiation. It supports the localization of up to 32 connected regions. (See reference...) Figure 11 The specific process is as follows: First, the target binary mask image is scanned row by row and pixel by pixel to identify all bright suspected target pixels; then, the number of suspected target pixels in the image is counted, and spatially adjacent bright pixels are assigned to the same connected region by the 8-neighborhood connectivity judgment rule, and a unique identifier ID (i.e., 1 and 2 in the figure) is assigned to each independent connected region; finally, the connected regions are filtered by combining the preset minimum pixel area threshold, and noise pseudo connected regions with a pixel count lower than the threshold are removed to suppress false alarms in the background, and all valid connected regions (i.e., connected regions that have not been removed) are labeled with data.

[0120] (iv) Target location marking and size parameter extraction.

[0121] The final step of preprocessing is to parse the position coordinates and contour size parameters of all valid targets (each valid connected component is considered a valid target) based on the image connected component labeling results, and generate the target bounding rectangle parameters. (Still referring to...) Figure 11 The specific process is as follows: First, read the pixel set of all valid connected components to parse each independent target connected component one by one; then, traverse all pixels in each connected component and count the maximum and minimum values ​​of the pixel coordinates in that connected component, namely the minimum horizontal coordinate, the maximum horizontal coordinate, the minimum vertical coordinate, and the maximum vertical coordinate; then, solve the bounding rectangle parameters based on the coordinate extreme values, the difference between the maximum and minimum horizontal coordinates is the target width, and the difference between the maximum and minimum vertical coordinates is the target height; finally, complete the calculation and caching of each target parameter in sequence, extract the coordinates and size of each connected region, and package and cache the target unique ID (i.e., the unique ID assigned to the connected component), the center pixel coordinates, the bounding rectangle size, and other key parameters in a unified manner to complete the extraction of target position and size information.

[0122] In the complete preprocessing flow described above, the key points of this invention include: First, the original image is subjected to median filtering of neighboring pixels using an image median filtering denoising algorithm. This aims to suppress random noise in the image while preserving the complete target edge details, thus avoiding target edge blurring caused by linear filtering and providing a high-quality image data source for subsequent target region segmentation and accurate extraction. Second, a target distance-based adaptive switching method for processing kernel parameters is used to process binary segmentation. This allows for matching different sizes of image processing kernel parameters to targets at different distances (far, medium, and near), overcoming the limitation of fixed-size processing kernels in balancing the segmentation accuracy of distant and near targets. Third, a dual-level grayscale statistical binary segmentation method is used to achieve target enhancement and complex background suppression, thereby stably outputting an accurate target binary mask image. Finally, a target connected component labeling algorithm is used to perform connected component detection and labeling on the target binary mask image, completing the extraction of target position and size parameters.

[0123] Compared to the traditional approach that only transmits the complete original image to the backend, this invention achieves random noise suppression, target enhancement, and complex background suppression through the aforementioned complex scene image segmentation and target region adaptive extraction pipeline design. This avoids the backend NPU performing a large number of invalid calculations on the background of the original image, significantly reducing the system's invalid computing power overhead and improving the real-time processing performance and stability of the complete business chain of target recognition and target tracking under complex background interference.

[0124] According to one embodiment of the present invention, the present invention also provides an image target tracking method, the method comprising the following steps: M1, acquiring multiple original images; M2, inputting all the original images into the adaptive target tracking system based on embedded intelligent heterogeneous computing as described in the present invention, to obtain a target tracking video.

[0125] In summary, to overcome the multiple problems existing in current edge-embedded intelligent image target computing schemes, such as limited computing power, limited data transmission rate, and inaccurate front-end image dynamic segmentation and target extraction, this invention provides an adaptive target tracking system and method based on embedded intelligent heterogeneous computing. Specifically, in terms of hardware architecture: this invention first constructs a high-performance embedded intelligent heterogeneous computing board based on a VPX embedded heterogeneous intelligent computing architecture composed of an ARM processor, an FPGA field-programmable gate array, and an NPU intelligent processor, along with high-speed RapidIO to PCIe bus interconnection communication technology. Next, a computing device is composed of multiple parallel embedded intelligent heterogeneous computing boards, one RapidIO bus to optical communication board, and a standard VPX backplane. Finally, a main control device is composed of multiple fiber-optic RapidIO to PCIe bus communication boards and multiple physical fiber optic cables. The main control device and the computing device together constitute the system of this invention. In terms of image processing: This invention constructs a pipeline for complex scene image segmentation and adaptive target region extraction based on the parallel characteristics of FPGA hardware. It realizes image median filtering and noise reduction, adaptive processing kernel size and two-level image binarization segmentation, ROI target region extraction, and transmits data to the CPU and NPU in real time through high-speed PCIe bus to further complete fine-grained intelligent detection and recognition processing.

[0126] In summary, the solution of this invention, through the combination of heterogeneous hardware architecture and innovative image processing methods, effectively solves the problems of computing power bottleneck, large-scale image data transmission bottleneck, and real-time image segmentation and target extraction accuracy bottleneck in existing edge-embedded intelligent image target computing solutions. It improves the efficiency of target recognition and extraction, and provides an effective solution for edge intelligent perception, intelligent tracking, and intelligent decision-making tasks, with good prospects for practical application and promotion.

[0127] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0128] This invention can be a computer program product. A computer program product mainly refers to a software product that implements this solution through a computer program.

[0129] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An adaptive target tracking system based on embedded intelligent heterogeneous computing, used to identify and extract a target from multiple original images for tracking the target, wherein each original image carries its corresponding target distance information; characterized in that, The system includes a computing device and a main control device, wherein: The computing device includes multiple embedded intelligent heterogeneous computing boards, one communication board, and a connection backplane, wherein: Each embedded intelligent heterogeneous computing board is used to adaptively adjust the extraction range according to the distance of the target in each original image according to preset rules, and extract all pixels within that range from the original image to form an accurately segmented target image. The connection backplane is used to connect each embedded intelligent heterogeneous computing board and communication board separately; Communication boards are used to enable data transmission between computing devices and main control devices; The main control device is used to send multiple raw images to the computing device for processing, and to receive the target image corresponding to each raw image to synthesize target tracking video.

2. The adaptive target tracking system based on embedded intelligent heterogeneous computing according to claim 1, characterized in that, Each embedded intelligent heterogeneous computing board is configured to process each original image according to preset rules as follows: The target binary mask image is generated for each original image using the image grayscale statistical binarization segmentation method. The size of the grayscale statistical window is adaptively adjusted according to the distance to the target, and the size of the grayscale statistical window is smaller as the target distance increases. The target is identified and segmented from the corresponding original image based on the target binary mask image to determine the extraction range; Extract all pixels within this range from the original image to compose an accurately segmented target image.

3. The adaptive target tracking system based on embedded intelligent heterogeneous computing according to claim 2, characterized in that, Each embedded intelligent heterogeneous computing board has three preset grayscale statistical window sizes: 3×3, 5×5, and 7×7, as well as near-distance and far-distance thresholds. Each embedded intelligent heterogeneous computing board is configured to adaptively adjust the size of the grayscale statistical window by comparing the target distance with the near-distance and far-distance thresholds. If the target distance is greater than or equal to the preset long distance threshold, the size of the grayscale statistics window will be adjusted to 3×3. If the target distance is less than or equal to the preset near distance threshold, the size of the grayscale statistics window will be adjusted to 7×7. Otherwise, adjust the size of the grayscale statistics window to 5×5.

4. The adaptive target tracking system based on embedded intelligent heterogeneous computing according to claim 2, characterized in that, Each embedded intelligent heterogeneous board is equipped with a CPU unit, an FPGA unit, and an NPU unit, among which: The CPU unit is used to schedule image data across devices, boards, and units; The FPGA unit is configured to preprocess the original image to generate a target binary mask image and target location and size. The preprocessing includes median filtering for noise reduction, image grayscale statistical binarization segmentation, image connected component parsing, and target location marking and size parameter extraction. The NPU unit is equipped with an image target detection model, which is used to identify and extract targets from the original image based on the target location and size generated by the FPGA unit.

5. The adaptive target tracking system based on embedded intelligent heterogeneous computing according to claim 4, characterized in that, The FPGA unit is configured with a raw image circular buffer queue and a target binary mask image circular buffer queue, wherein: The raw image circular buffer queue consists of multiple sequentially arranged buffers, in which each raw image is sequentially written to a different buffer for the CPU to read, and the CPU reads the raw image from the last buffer that has been written to. The target binary mask image circular buffer queue includes multiple sequentially arranged buffers. Each target binary mask image generated by the FPGA unit is sequentially written to different buffers for the CPU to read, and the CPU reads the target binary mask image from the previously written buffer.

6. The adaptive target tracking system based on embedded intelligent heterogeneous computing according to claim 4, characterized in that, The FPGA unit is configured to perform median filtering denoising as follows: T1. Set a sliding filter window in the original image; T2. The sliding filter window traverses the entire original image row by row and column by column. Each slide performs the following processing: T21. Traverse all pixels within the current coverage area of ​​the filter window to obtain the grayscale value of each pixel, and sort the grayscale values ​​of all pixels within the current coverage area in ascending order. T22. Take the median value from all grayscale values ​​sorted in ascending order and assign it to the center pixel within the current coverage area of ​​the filter window.

7. The adaptive target tracking system based on embedded intelligent heterogeneous computing according to claim 4, characterized in that, The FPGA unit is configured to perform image grayscale statistical binarization segmentation based on two-level grayscale statistics, including: S1. Construct a pixel operation matrix centered on the pixel to be processed, which includes a central sub-block, multiple inner sub-blocks distributed adjacent to the central sub-block, and multiple outer sub-blocks distributed adjacent to all inner sub-blocks. The size of all sub-blocks is equal to the size of the grayscale statistics window. S2. The sliding grayscale statistics window calculates the inner grayscale statistics value corresponding to the area occupied by all inner layer sub-blocks and the outer grayscale statistics value corresponding to the area occupied by all outer layer sub-blocks respectively, and uses the product of the inner grayscale statistics value and the outer grayscale statistics value as the pixel feature value of the pixel to be processed. S3. Calculate the pixel feature values ​​of all pixels in the original image to obtain the feature response map, and calculate the global mean and global standard deviation of the feature response map, and calculate the adaptive segmentation threshold as follows: Where T represents the adaptive segmentation threshold; The global mean of the characteristic response map; The empirical coefficient corresponding to the global mean of the characteristic response map is 2; The global standard deviation representing the characteristic response map; The empirical coefficient corresponding to the global standard deviation of the characteristic response map is 40; S4. Compare each pixel feature value with the adaptive segmentation threshold one by one to generate the target binary mask image.

8. The adaptive target tracking system based on embedded intelligent heterogeneous computing according to claim 7, characterized in that, In S2, the inner layer grayscale statistics and the outer layer grayscale statistics are calculated in the following way: S21. Calculate the average gray value of each sub-block; S22. Calculate the absolute difference between the mean gray value of the central sub-block and the mean gray value of each inner sub-block, and multiply the absolute differences calculated on the same pair of paths to obtain the median value on each pair of paths, and take the minimum median value as the gray value of the inner layer; where the pair of paths refers to the line connecting the center pixel of the central sub-block and the center pixel of each inner sub-block. S23. Take the smallest average gray value among all outer layer sub-blocks as the outer layer gray value statistical value.

9. The adaptive target tracking system based on embedded intelligent heterogeneous computing according to claim 4, characterized in that, The FPGA unit is configured to perform image connected component resolution as follows: M1. Perform a full scan of the target binary mask image line by line and pixel by pixel to identify all bright pixels, where a bright pixel is defined as a pixel with a brightness value of 1. M2. The eight-neighbor connectivity determination rule is adopted to classify spatially adjacent bright pixels into the same connected domain, and a unique ID is assigned to each connected domain as the target ID corresponding to that connected domain. M3. Compare the number of pixels in each connected component with the preset minimum pixel count threshold. Retain connected components with a pixel count greater than or equal to the minimum pixel count threshold as valid connected components, and discard the remaining connected components.

10. The adaptive target tracking system based on embedded intelligent heterogeneous computing according to claim 9, characterized in that, The FPGA unit is configured to perform target location marking and size parameter extraction in the following manner: Traverse all pixels within each valid connected component to determine the center pixel coordinates of each valid connected component, and calculate the minimum horizontal coordinate, maximum horizontal coordinate, minimum vertical coordinate, and maximum vertical coordinate of each pixel within each valid connected component to solve for the bounding rectangle size of that valid connected component; where the difference between the maximum horizontal coordinate and the minimum horizontal coordinate is the target width, and the difference between the maximum vertical coordinate and the minimum vertical coordinate is the target height; The target ID, center pixel coordinates, and bounding rectangle size of each valid connected component are packaged and cached to complete the target location marking and size parameter extraction.

11. The adaptive target tracking system based on embedded intelligent heterogeneous computing according to claim 1, characterized in that, The communication board is a RapidIO bus to fiber optic communication board, which is equipped with multiple optical modules, each of which serves an embedded intelligent heterogeneous board. The main control device is equipped with multiple fiber-to-RapidIO to PCIe bus communication boards. Each fiber-to-RapidIO to PCIe bus communication board is connected to an optical module on the RapidIO bus to fiber-to-fiber communication board via a fiber optic link.

12. The adaptive target tracking system based on embedded intelligent heterogeneous computing according to claim 1, characterized in that, The connection backplane is a standard VPX backplane, which includes one RapidIO bus to fiber optic communication board slot and multiple embedded intelligent heterogeneous board slots, wherein: Each embedded intelligent heterogeneous board slot is equipped with a power interface, a RapidIO interface, a Camera Link interface, a network port, and a serial port; The RapidIO bus to fiber optic communication board slot is equipped with a power interface, a RapidIO interface, a network port, and a serial port; the RapidIO bus to fiber optic communication board slot and the embedded intelligent heterogeneous board slot are connected via a backplane circuit.

13. The adaptive target tracking system based on embedded intelligent heterogeneous computing according to claim 12, characterized in that, The standard VPX backplane is 3U in size and is equipped with one RapidIO bus to fiber optic communication board slot and three embedded intelligent heterogeneous board slots.

14. An image target tracking method, characterized in that, The method includes the following steps: M1: Acquire multiple original images; M2. Input all the original images into the adaptive target tracking system based on embedded intelligent heterogeneous computing as described in any one of claims 1-13 to obtain target tracking video.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 14.

Images