A low-delay visual detection method and system based on a Pengcheng Ascend heterogeneous platform

Abstract

The application provides a low-delay visual detection method and system based on the Kunpeng Ascending heterogeneous platform, relates to the technical field of industrial automation control and edge computing, and the method comprises the following steps: collecting original pulse signals of an encoder based on an FPGA acquisition card; obtaining standardized pulse signals; obtaining image data; writing the image data into DDR memory through a DMA engine; reading the image data in the DDR memory based on a dynamically adjusted DVPP strategy, and performing preprocessing on the image data by using a DVPP of an Ascending NPU; and inputting the preprocessed image data into a deep learning model of an AI core for inference; the application directly writes the obtained image data into DDR memory through a DMA engine, directly pulls the data by using a DVPP, and does not involve Kunpeng CPU in image data carrying, so that zero copy is realized.

Description

Technical Field

[0001] This application relates to the fields of industrial automation control and edge computing technology, and in particular to a low-latency visual inspection method and system based on the Kunpeng Ascend heterogeneous platform. Background Technology

[0002] In the field of intelligent manufacturing, online inspection based on machine vision has become a key link in ensuring product quality. Currently, most mainstream industrial vision inspection systems adopt an architecture of "x86 industrial computer + dedicated GPU (display core or vision processor)". However, this architecture has certain drawbacks: First, heterogeneous data transfer bottleneck: In the traditional "CPU (Central Processing Unit) + GPU" architecture, image data often needs to be frequently copied between system memory and graphics card memory. However, under the domestic Kunpeng (ARM) + Ascend (NPU) architecture, due to the difference in memory consistency models, if the traditional data flow is used, a large amount of CPU resources will be consumed in data preprocessing (such as decoding and scaling), resulting in idle NPU computing power and a decrease in the overall system throughput. Second, soft trigger latency jitter: Industrial production lines are extremely fast. Existing vision software based on general operating systems often has millisecond-level system jitter from receiving PLC signals to triggering the camera to take pictures. On high-speed production lines, this can lead to deviations in the picture position and cause missed detections. Third, lack of edge-side closed loop: Existing solutions mostly upload the results to the cloud for processing, lacking the ability to directly complete the "perception-decision-control" closed loop at the edge, making it difficult to meet high real-time requirements.

[0003] Therefore, there is an urgent need for an industrial vision inspection system and method that can fully leverage the advantages of domestically produced heterogeneous computing platforms to achieve efficient data flow and precise synchronous control. Summary of the Invention

[0004] To address one of the aforementioned technical shortcomings, this application provides a low-latency visual inspection method and system based on the Kunpeng Ascend heterogeneous platform.

[0005] According to the first aspect of this application, a low-latency visual inspection method based on the Kunpeng Ascend heterogeneous platform is provided, comprising the following steps: acquiring the raw pulse signal of the encoder based on a field-programmable gate array (FPGA) acquisition card and preprocessing it to obtain a standardized pulse signal; obtaining image data based on the obtained standardized pulse signal; the FPGA acquisition card writes the obtained image data into the system memory (DDR memory) of the Kunpeng computing module through a DMA engine, and the image data is perceived by the Kunpeng processor (CPU) to reach the DDR memory; the DDR memory is divided into a circular buffer, and the circular buffer is provided with a shared descriptor ring for filling metadata, the shared descriptor ring including multiple slots, each slot storing a task descriptor; the digital vision preprocessing unit (DVPP) of the Ascend AI processor (NPU) reads the image data in the DDR memory based on a dynamically adjusted DVPP strategy and preprocesses it; the preprocessed image data is input into the deep learning model of the AI ​​core for inference, and the inference result is output; the inference result includes the target bounding box coordinates, defect category, and confidence level.

[0006] According to a second aspect of this application, a low-latency visual inspection system based on the Kunpeng Ascend heterogeneous platform is provided. The system includes: an industrial camera; an encoder; and an FPGA acquisition card electrically connected to the industrial camera and encoder. Its internal logic modules include: an encoder input module for acquiring the encoder's raw pulse signal; a preprocessing module for preprocessing the raw pulse signal to obtain a standardized pulse signal; a speed calculation module for calculating the target instantaneous speed of the conveyor belt based on the obtained standardized pulse signal; a delay calculation module for calculating a compensation delay time based on the calculated target instantaneous speed of the conveyor belt and a preset physical distance parameter; a calibration signal acquisition module for obtaining a hard-trigger delay calibration signal based on the calculated compensation delay time; an effective signal acquisition module for performing hardware-level debouncing processing on the acquired external trigger signal to obtain an effective signal; a trigger output module for obtaining a hard-trigger signal based on the obtained hard-trigger delay calibration signal and the effective signal, and then sending the hard-trigger signal to an external camera to trigger the external camera's exposure; and a DMA engine for directly transferring image data to the Kunpeng computing module's DMA. The R memory contains the Kunpeng computing module, which includes a Kunpeng CPU and DDR memory. The Kunpeng CPU is divided into a control core and a scheduling core. The control core runs the operating system, manages the shared descriptor ring's state, and handles exceptions and interrupts. The scheduling core polls the shared descriptor ring, monitors the image input rate and NPU inference throughput, calculates the load index, dynamically adjusts the DVPP strategy, and executes an adaptive scheduling strategy. A physically contiguous large page memory region is allocated from the DDR memory as a zero-copy buffer, i.e., a circular buffer. This region is mapped as shared memory for FPGA writing. Image data is input for the Ascend NPU to read; the Ascend AI acceleration module, including the Ascend NPU, has internal core units including DVPP and AI core. The DVPP contains independent hardware decoders, croppers, and scalers, and has an independent DMA channel for reading image data from DDR memory; the AI ​​core is used to execute deep learning models, perform inference on the preprocessed image data, and output the inference results; a high-speed interconnect bus connects the FPGA acquisition card and the Ascend AI acceleration module to the Kunpeng computing module via PCIe.

[0007] The beneficial effects of this application are as follows:

[0008] The low-latency visual detection method based on the Kunpeng Ascend heterogeneous platform provided in this application establishes a closed-loop control flow of "speed perception triggering → physical address pass-through → rate adaptive back pressure" through a hardware system (FPGA acquisition card + Kunpeng CPU + Ascend NPU). On the one hand, the FPGA acquisition card acquires images and provides a precise time base and unambiguous data source through speed feedforward calculation. The acquired image data is directly written into the DDR memory of the Kunpeng computing module through the Direct Memory Access Engine (DMA engine), and the Ascend DVPP is used to directly pull the data. While achieving true zero copy, it also extends the DMA concept to cross-device (i.e., FPGA to NPU) physical address pass-through. Furthermore, the FPGA hardware-level speed feedforward and dynamic latency compensation achieve microsecond-level trigger accuracy and eliminate operating system soft trigger jitter. On the other hand, the Kunpeng CPU in this application only updates the task descriptor in the shared descriptor ring and senses the arrival of data. It does not participate in image data transfer. Moreover, the Kunpeng CPU returns directly after writing the task descriptor without waiting for Ascend NPU processing or responding to interrupts after Ascend NPU completion. This eliminates context switching overhead, enabling a single-core CPU to support image scheduling at tens of thousands of FPS (frames per second) and achieve zero interrupt latency. Furthermore, this application transforms the heterogeneous concept from "computation splitting" to "IO backpressure control," solving the "speed mismatch" problem by dynamically adjusting the strategy, and achieving a complete "perception-decision-control" closed loop at the edge, meeting the millisecond-level real-time quality inspection requirements of high-speed production lines.

[0009] This application divides a circular buffer in DDR memory, which is mapped as shared memory for the FPGA to write image data and for the Ascend NPU to read data. Moreover, the circular buffer does not store large amounts of image data or long logs, but task descriptors, which eliminates the need to move large amounts of data, greatly reduces latency, and achieves efficient zero-copy transmission. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0011] Figure 1 A memory layout diagram of the circular buffer provided in this application;

[0012] Figure 2 A diagram illustrating the movement process of the Head and Tail pointers provided in this application. Detailed Implementation

[0013] To make the technical solutions and advantages of the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. Unless otherwise specified, FPGA refers to FPGA acquisition card, CPU refers to Kunpeng CPU, NPU refers to Ascend NPU, and DVPP refers to DVPP unit.

[0014] To address some problems existing in the prior art, firstly, this application provides a low-latency visual detection method based on the Kunpeng Ascend heterogeneous platform, the method comprising:

[0015] The original pulse signal from the encoder is acquired using an FPGA acquisition card and preprocessed to obtain a standardized pulse signal; image data is then obtained based on the obtained standardized pulse signal.

[0016] The FPGA acquisition card writes the acquired image data into the DDR memory of the Kunpeng computing module via a DMA engine, and the Kunpeng CPU senses the arrival of the image data in the DDR memory. The DDR memory is divided into a circular buffer, and the circular buffer has a head pointer, a tail pointer, and a shared descriptor ring for filling metadata. The metadata is the control information contained in the task descriptor stored in each slot of the shared descriptor ring. The shared descriptor ring includes multiple slots, each slot stores one task descriptor, and the slot status of each slot includes idle, pending, and processing. In specific implementation, the total depth of the shared descriptor ring (i.e., the depth of the ring) is N (e.g., N=1024, which needs to be a power of 2 for modulo operation). The FPGA directly writes the image data into the pre-allocated physical address area of ​​the DDR memory through the DMA engine, realizing physical address pass-through, thereby avoiding CPU copying and gaining time margin for subsequent high-throughput scheduling.

[0017] The Ascend NPU's DVPP directly reads image data from DDR memory based on a dynamically adjusted DVPP strategy and preprocesses it. In practice, the timestamp written by the FPGA is read by the NPU here to calculate the end-to-end latency.

[0018] The preprocessed image data is input into the deep learning model at the core of AI for inference, and the inference results are output; the inference results include the target bounding box coordinates, defect category, and confidence level.

[0019] This application can determine whether the current product has defects and the specific type of defects (such as scratches, dents, stains, etc.) through reasoning results, determine the precise location of the defects on the product, and decide whether to trigger downstream actuators (such as rejection devices, alarm systems, etc.) based on confidence thresholds, and feed the detection results back to the production line control system in real time to achieve closed-loop quality control.

[0020] DVPP (Dynamic Video Processing) strategy refers to a systematic approach that utilizes the dedicated hardware acceleration units (VPC, JPEGD, etc.) built into the Ascend NPU to replace traditional CPU software processing, thereby achieving efficient and low-latency preprocessing of image / video data. DVPP is not merely a simple "image scaling" strategy, but a complete acceleration solution encompassing pipeline orchestration, zero-copy transfer, dynamic parameter adaptation, and load-adaptive adjustment. Specifically, this strategy includes: geometric transformation and size adaptation strategies, dynamic AIPP (Artificial Intelligence Preprocessing) strategies, zero-copy and memory management strategies, and load-aware adaptive DVPP strategies.Specifically, the geometric transformation and size adaptation strategy involves using a hardware interpolation algorithm within the VPC unit to uniformly scale or crop input images (input from the FPGA) of different resolutions to a fixed input size (e.g., 640x640) required by the AI ​​model (i.e., deep reinforcement learning model). This supports arbitrary scaling and region-of-interest (ROI) matting, with processing time weakly correlated with image resolution and extremely stable performance. The dynamic AIPP strategy, the highest level of the DVPP strategy, addresses the contradiction between "static model" and "dynamic scene," including dynamic parameter injection and data normalization hardware offloading. On one hand, traditional AIPP parameters (such as subtracted mean and normalization coefficients) are hardcoded during model transformation. However, in reality, camera exposure parameters differ across production lines or under different lighting conditions. Therefore, a dynamic AIPP mode (i.e., dynamic parameter injection) is enabled. The scheduling core, based on the image metadata (such as gain and exposure time) input from the FPGA, calls the input AIPP interface before each inference execution to dynamically distribute the current normalization parameters or color gamut conversion parameters. This process eliminates the need to retrain the model, allowing the AI ​​model to adapt to input data from different lighting conditions or camera models, significantly improving detection accuracy. On the other hand, the data normalization hardware is offloaded, moving the pixel value subtraction and multiplication operations to the DVPP hardware instead of calculating them on the CPU. Zero-copy and memory management strategies eliminate performance bottlenecks caused by data transfer. These strategies include device-side memory closure and asynchronous concurrency control. The device-side memory closure involves the FPGA directly writing image data to the Ascend NPU's onboard memory (DeviceDDR) via DMA, and the DVPP directly reading the DeviceDDR. DDR data undergoes preprocessing, and the processed data is directly fed into the AI ​​core for inference. The entire process avoids the Kunpeng CPU memory, thus avoiding PCIe bandwidth bottlenecks and CPU-memory copy overhead. Asynchronous concurrency control utilizes the Ascend computing language's stream mechanism (a task queue that manages task parallelism and execution order) to orchestrate DVPP tasks and AI inference tasks in the same or different streams. The execution logic is that while DVPP is decoding the previous frame, the AI ​​core is inferring the frame two frames prior. The scheduling core listens for DVPP completion signals through callback functions. Once preprocessing is complete, inference is immediately triggered, achieving ultimate pipeline parallelism. A load-aware adaptive DVPP strategy is used to handle sudden traffic surges.

[0021] In practical applications, on the one hand, the Kunpeng CPU senses the arrival of image data in DDR memory through interrupts or polling. The specific operation process of interrupt is as follows: after the FPGA is written, it sends a hardware interrupt signal to the Kunpeng CPU. The Kunpeng CPU responds to the interrupt, suspends the currently executing regular task, and enters the interrupt service routine. In the interrupt service routine, the Kunpeng CPU reads the status register of the shared descriptor ring, confirms that the image data has been written to DDR memory, fills the task descriptor in the corresponding free slot, moves the Tail pointer forward, and then exits the interrupt service routine to resume the execution of the original task. The specific operation process of polling is as follows: the Kunpeng CPU continuously checks a specific memory address (such as the status bit of the shared descriptor ring) in an infinite loop to determine whether the shared descriptor ring has been updated. On the other hand, the image data is preprocessed by the DVPP unit, including decoding, scaling, and normalization, to generate feature data that conforms to the model input format. Then, the AI ​​core performs end-to-end feature extraction and inference on the feature data through high-concurrency matrix multiplication (GEMM) operations. In addition, the DVPP unit of Ascend NPU monitors the shared descriptor ring through a hardware polling engine. Once a new descriptor is detected, DVPP directly initiates a DMA read request to pull image data from DDR memory for decoding, cropping, or format conversion (such as YUV to RGB).

[0022] Based on the above scheme, the low-latency visual detection method based on the Kunpeng and Ascend heterogeneous platforms provided in this application establishes a closed-loop control flow of "speed perception triggering → physical address pass-through → rate adaptive back pressure" through the hardware system (FPGA acquisition card + Kunpeng CPU + Ascend NPU). Furthermore, this application utilizes the hardware characteristics of Kunpeng and Ascend to achieve "non-grounded" data transmission. On the one hand, the FPGA acquires images and provides a precise time base and unambiguous data source through speed feedforward calculation. It then bypasses the CPU via a DMA engine, directly writing the massive image data (such as several MB of raw images) into a pre-defined physical address region in DDR memory. At this point, the CPU is completely unaware and does not participate in any data transfer. The data is then directly retrieved using DVPP, achieving true zero-copy while extending the DMA concept to cross-device (i.e., FPGA to NPU) physical address pass-through. Simultaneously, the physical address is directly passed through the task descriptor, allowing the DMA engine to directly access the memory region acquired by the camera, realizing a zero-copy data stream from the camera to the NPU. Furthermore, the FPGA hardware-level speed feedforward and dynamic latency compensation achieve microsecond-level trigger accuracy, eliminating operating system soft-trigger jitter. On the other hand, the CPU in this application only updates the task descriptor in the shared descriptor ring and senses the arrival of data, without participating in image data transfer. The CPU returns immediately after writing to the task descriptor, without waiting for NPU processing or responding to interrupts after NPU completion, eliminating context switching overhead. This allows a single-core CPU to support image scheduling at tens of thousands of FPS, achieving zero interrupt latency. Furthermore, this application transforms the heterogeneous concept from "computation splitting" to "IO backpressure control," solving the "speed mismatch" problem by dynamically adjusting the strategy, and achieving a complete "perception-decision-control" closed loop at the edge, meeting the millisecond-level real-time quality inspection requirements of high-speed production lines.

[0023] The encoder pulses are bursty, while the FPGA's calculation speed, table lookup, and output of hard trigger signals require a tiny amount of time. If the pulses are too dense, direct processing may not be fast enough. Therefore, this application partitions a circular buffer in DDR memory. This circular buffer is mapped as shared memory, allowing the FPGA to write image data and the Ascend NPU to read data. Furthermore, the circular buffer does not store massive amounts of image data or long log strings, but rather task descriptors, eliminating the need to move large amounts of data and significantly reducing latency.

[0024] In practical implementation, a slot status of "Idle" indicates an unused slot, meaning it can be written to by the Kunpeng CPU; a slot status of "Pending Processing" indicates a slot that has been filled, meaning data has been written and is waiting for the Ascend NPU to read it; and a slot status of "Processing" indicates a slot that is being processed, meaning the Ascend NPU has read it and is executing the DVPP strategy. Additionally, a slot status of "Done" indicates that DVPP execution has been completed and the slot is waiting for CPU reclamation. This slot status is optional and depends on whether the result needs to be written back.

[0025] Specifically, the deep learning model adopts the YOLOv8 object detection architecture based on convolutional neural networks (CNN), which is deployed on the AI ​​core of the Ascend NPU.

[0026] Furthermore, the metadata includes the physical address Phys_Addr, data size Data_Size, timestamp Timestamp, and region of interest (ROI) information ROI_Info. Phys_Addr is equivalent to the "storage location," indicating to the Ascend NPU the exact physical coordinates of this batch of image data in DDR memory. Based on this address, the Ascend NPU can directly "fetch" the image data from DDR memory via DVPP hardware. Data_Size is equivalent to the "thickness of a book," indicating the size of this batch of data for the Ascend NPU, preventing it from reading out of bounds or making mistakes. Timestamp is equivalent to the "entry time," recording the precise moment the FPGA acquired this frame of image, which is crucial for subsequent multi-camera synchronization or calculating object motion speed. ROI_Info is equivalent to a "bookmark," indicating to the Ascend NPU which area in the entire image truly needs AI recognition and which is the background. The Ascend NPU can directly crop this area during preprocessing, saving computing power.

[0027] Specifically, after the camera driver writes image data to DDR memory via PCIe DMA, the CPU driver obtains the physical address of the image in the circular buffer through the IOMMU (Input / Output Memory Management Unit) or a reserved memory pool. Passing virtual addresses is strictly prohibited because the NPU firmware runs in an independent address space or without an operating system (OS), and therefore cannot resolve CPU virtual addresses. The IOMMU is a hardware component used to manage and control direct access to system memory by physical devices. Its core functions include address translation, device isolation, and security, which are particularly crucial in virtualized environments. The reserved memory pool is existing technology and will not be discussed further here.

[0028] like Figure 1As shown in the figure, the memory layout structure of the circular buffer is mainly a circular structure composed of 8 consecutive memory blocks (i.e., data blocks). Each block is called a "task descriptor" and is numbered from D[0] to D[7]. That is, the capacity of the circular buffer is 8. It is often used in scenarios of efficient and lock-free data exchange between hardware and software, such as FPGA and host communication, DMA transfer, network packet processing, etc. The circular buffer is used to cache encoder pulse events or trigger instruction queues to support high-speed, low-latency real-time control. In addition, the circular buffer is essentially a logically connected continuous memory area. It realizes the orderly writing and reading of data by moving two pointers - the Head pointer and the Tail pointer, without physically moving the data, thereby greatly improving efficiency.

[0029] Specifically, the starting address of the entire structure of the circular buffer is marked as the "Base Address", which is the starting point for the Kunpeng CPU or DMA engine to access the circular buffer.

[0030] The Head (consumer) represents the "read" position (i.e., the consumer's read pointer position). In the triggering system, inside the FPGA acquisition card, the "consumer" is the triggering logic or DMA engine inside the FPGA, responsible for retrieving the trigger events or speed data to be processed from the circular buffer. That is, it is responsible for retrieving pulse / speed data from the circular buffer for calculation, or retrieving the image data descriptor to initiate DMA transfer to DDR memory. At the shared descriptor ring level of DDR memory, the "consumer" is the DVPP unit of the Ascend NPU, responsible for reading the task descriptor from the shared descriptor ring according to the Head pointer and Cons_Idx, and then initiating DMA to read image data according to Phys_Addr.

[0031] Tail (producer) represents the "write" position (i.e., the producer's read pointer position). In the triggering system, inside the FPGA acquisition card, the "producer" is usually the encoder input module or the host computer, which is responsible for putting newly captured pulse events into the queue, that is, writing the newly captured pulse timestamp, speed value or trigger request into the circular buffer. At the shared descriptor ring level of DDR memory, the "producer" is the Kunpeng CPU, which is responsible for writing task descriptors in the shared descriptor ring and updating the Tail pointer and Prod_Idx.

[0032] The Head pointer (red arrow) points to the location of the task descriptor that the current "consumer" is about to read (D[0] in the figure), and Head = Cons_Idx % N, that is, the Head pointer points to the (Cons_Idx % N)th slot at the physical level, and Cons_Idx is counted at the logical level.

[0033] Tail pointer (blue arrow): points to the position of the task descriptor that the current “producer” is about to write (D[4] in the figure), and Tail=Prod_Idx % N, that is, the Tail pointer points to the (Prod_Idx % N)th slot at the physical level, and Prod_Idx is counted at the logical level.

[0034] The circular characteristic of this circular buffer is that when the Tail pointer reaches D[7], the next write will automatically return to D[0]; similarly, the Head pointer will also return to D[0] after reading D[7]. This design effectively avoids memory relocation and achieves efficient circular use.

[0035] Furthermore, the shared descriptor ring also includes a control block located at the head of the shared descriptor ring. This block includes two monotonically increasing atomic counters updated by different cores: a producer index, Prod_Idx, and a consumer index, Cons_Idx. Both are mapped to physical slots in the circular buffer by taking the modulo of N. Prod_Idx records the total number of task descriptors written to the shared descriptor ring by the Kunpeng CPU, representing the number of tasks that have been written. It is updated or maintained by the Kunpeng CPU and points to the next writable free slot. Cons_Idx records the total number of task descriptors processed by the Ascend NPU, representing the number of tasks that have been processed. It is updated or maintained by the Ascend NPU firmware and points to the next unprocessed slot.

[0036] In practical applications, the descriptor structure defines the basic control information of the task descriptor, input image information (i.e., raw material address), output image information (i.e., finished product storage location), preprocessing operator parameters, and status and verification (completion receipt). The basic control information is used to confirm identity and task type, including the magic number, task ID, and operation type. The input image information instructs the NPU where to find the original image, including the physical starting address, size, and format. The output image information instructs the NPU where to place the processed image, including the output physical address and target specifications. The preprocessing operator parameters contain specific algorithm execution details, including cropping and padding parameters. The status and verification are used to provide feedback on task execution, including processing status and checksum.

[0037] Specifically, the Kunpeng CPU fills the task descriptor structure into Ring[Prod_Idx % N], where Ring[Prod_Idx % N] is the core addressing formula of the circular buffer. Its function is to map an infinitely growing linear index to a finite-sized physical memory array, thereby achieving the effect of circular writing. In the formula, Ring represents a pre-allocated circular buffer array (i.e., a shared memory region), assuming its length (i.e., the depth of the ring) is fixed at N; Prod_Idx is the producer index, i.e., the producer pointer, representing the total number of tasks written by the Kunpeng CPU, which will keep increasing (0, 1, 2, ... 100, 101...) and will never be reset to 0; N represents the depth of the ring, i.e., the total number of pre-allocated slots in the circular buffer (such as 8, 16, or 32), which is a constant; % is the modulo operator, used to map a monotonically increasing linear index to a fixed-size circular buffer array, and is the key mathematical tool for achieving circular writing; and through Prod_Idx % N can calculate the physical location of the next writable free slot (i.e., the location pointed to by the Tail pointer).

[0038] More specifically, the magic number is used to verify data validity and prevent the reading of incorrect data; the task ID is a globally unique number for easy log tracking and troubleshooting; the operation type clearly indicates the specific operation performed by the NPU, such as scaling, cropping, color conversion, or filling; the physical start address is the most critical field, pointing to the actual physical location of the image data in DDR memory and bypassing the virtual memory management unit; the size and format include width, height, pixel format (such as YUV420SP, RGB888, etc.), and row span, ensuring that the NPU can correctly parse memory blocks; the output physical address is a pre-allocated memory address used to store the results after DVPP processing; the target specification specifies the width, height, and format requirements of the processed image data; the cropping parameters define the coordinates of the top-left corner and the width and height of the crop; the filling parameters define the number of pixels to be filled in the top, bottom, left, and right directions; the processing status is used to mark whether the current task is waiting, successful, or has a hardware error, and the checksum usually uses cyclic redundancy check (CRC check) to prevent the NPU from reading incorrect addresses or parameters due to memory bit flipping.

[0039] Specifically, the internal structure of each task descriptor includes core fields, metadata fields, and extended fields. Among them, the core fields (taking D[0] and D[1] as examples) include: Buf Addr, Length / Flags, and Status. Buf Addr indicates the physical memory address where the actual data is stored (such as the buffer for storing speed values, timestamps, or trigger parameters). Length / Flags indicates the length of the data written this time or the status flag (such as "data valid", "processed", "retransmission required"). Status indicates the status bit of the task descriptor itself, used to synchronize producers and consumers (such as "idle", "filled", "processing"). The metadata fields include Phys_Addr, Data_Size, Timestamp, and ROI_Info. The extended fields include basic control information, input image information, output image information, preprocessing operator parameters, status, and verification.

[0040] In practical applications, the task descriptor is a lightweight control data structure, independent of the actual image payload storage. It encapsulates the physical addressing information, data boundaries, temporal attributes, and preprocessing control parameters of the image data. By completely decoupling the massive image data stream from the lightweight control instruction stream in physical memory, the CPU only needs to perform atomic-level write operations to the task descriptor to drive the DVPP hardware to initiate independent DMA reads based on the physical addresses within the task descriptor. This mechanism fundamentally eliminates the bus congestion and wasted CPU computing power caused by cross-device memory copying in traditional architectures.

[0041] Specifically, when the encoder captures a pulse, the task descriptor not only records the time but also calculates the speed. This complex data packet is stored at the location pointed to by Buf Addr, while D[x] is simply an "index card" pointing to this data packet.

[0042] Furthermore, the ring space of the shared descriptor ring needs to be checked (non-blocking), specifically including:

[0043] The Kunpeng CPU reads Cons_Idx; based on Pending_Num = (Prod_Idx - Cons_Idx) % N, it calculates the number of unprocessed tasks in the current ring; where Pending_Num represents the number of tasks to be processed, N represents the ring depth, and represents the maximum number of concurrent tasks allowed by the system, which mainly depends on the DDR memory size;

[0044] If Pending_Num < N, it means there is a free slot, so continue the write operation. That is, the Kunpeng CPU fills the task descriptor corresponding to the current frame into the Ring[Prod_Idx % N] slot, and then performs the atomic addition operation Atomic_Add(&Prod_Idx, 1) to move Prod_Idx forward.

[0045] If Pending_Num == N, it indicates that the ring is full (NPU processing is too slow), and a frame dropping strategy is executed to ensure real-time performance. In practice, the frame dropping strategy directly releases the DDR physical memory area (i.e., the image buffer) occupied by the current frame image data, updates Prod_Idx to point to the next frame, and does not write the task descriptor corresponding to the frame into the shared descriptor ring.

[0046] Based on the above scheme, this application naturally forms flow control by judging the full / empty state of the circular buffer. When the NPU cannot handle the load, the CPU automatically drops frames to prevent system memory overflow or latency accumulation.

[0047] like Figure 2 As shown, this diagram illustrates the movement of the Head and Tail pointers. Orange blocks represent slots awaiting processing, i.e., task descriptors written by the "producer" and awaiting processing by the "consumer"; gray blocks represent idle slots; and the gray blocks within the dashed boxes represent released / idled slots, i.e., task descriptors processed by the "consumer." This diagram demonstrates the changes in the pointers over time, illustrating how data flows. The specific process is as follows:

[0048] Phase 1 (Initial Accumulation): Head=0, Tail=4, indicating that the "producer" is moving quickly, writing 4 task descriptors (D[0]~D[3]), while the "consumer" has not yet moved and is stuck at the starting point D[0]. In this phase, the production line has just started and is moving quickly, generating multiple triggering demands instantly, which are piling up in the queue waiting to be processed.

[0049] Phase 2 (Consumption and Release): Head=1, Tail=4. At this time, the "consumer" has finished processing D[0]. The Head pointer moves to the right to 1, and D[0] becomes a released / idle slot, indicating that the FPGA has completed the response to the first pulse (such as the camera exposure has been triggered). The slot is marked as "available" and is ready to be recycled.

[0050] Phase 3 (Continuous Production): Head=1, Tail=5. At this time, the "producer" continues to work and writes new data to D[4]. The Tail pointer moves to the right to D[5]. D[4] becomes a new slot to be processed, indicating that a new pulse has arrived. The system puts it into the next free slot. At this time, there are 4 data to be processed in the queue (D[1], D[2], D[3], D[4]).

[0051] In practical applications, to prevent program crashes or data overwriting, this application defines the system's security rules as follows: (1) Empty ring: Head == Tail. This indicates that the read and write pointers coincide (i.e., the Head pointer and Tail pointer coincide), meaning that the "producer" has not written new data, or the "consumer" has read all the data. At this time, the FPGA is in an idle waiting state; (2) Full ring: (Tail + 1) % N == Head, where % represents the modulo operator and N represents the ring depth. This is the most critical protection mechanism because the coincidence of the Head pointer and Tail pointer can be either empty or full, so usually a storage unit is sacrificed to distinguish them. If the queue is full (e.g., all 7 slots are full), the "producer" must stop writing, otherwise the new data will overwrite the old data that the Head has not yet read, causing a missed clock.

[0052] Based on the above scheme, the circular buffer in this application is equivalent to a flexible reservoir. The encoder only needs to throw pulses into the shared descriptor ring of the circular buffer without considering whether the FPGA is working. The "producer" only needs to throw pulses into D[x], and the "consumer" takes data from the Head for calculation according to its rhythm. As long as the circular buffer is not full, no data will be lost. If the production line suddenly accelerates and the encoder pulses surge, causing the ring to be full (i.e., the Tail catches up with the Head), the hardware logic can immediately feed back to the encoder interface: "pause counting" or "discard the current pulse", thereby protecting the system from logical chaos. Moreover, D[x] only stores an address pointer. The actual speed data or trigger parameters are stored in the Kunpeng DDR memory. After the FPGA finishes processing D[0], it only needs to move the Head pointer and does not need to move a large amount of data, which greatly reduces the latency.

[0053] Furthermore, the method also includes dynamically adjusting the DVPP strategy based on the acquired image input rate and NPU inference throughput, specifically including:

[0054] The Kunpeng CPU's scheduling core polls the shared descriptor ring to obtain the image input rate and NPU inference throughput. The image input rate is determined by the frame rate reported by the FPGA, which is affected by the conveyor belt speed. The NPU inference throughput is the frame rate that the Ascend NPU is currently actually completing inference at.

[0055] According to the formula The difference in monitoring indicators is calculated; where, For image input rate, For NPU inference throughput, The difference in monitoring indicators;

[0056] Based on the calculated difference in monitoring indicators The DVPP strategy is dynamically adjusted as follows:

[0057] like ,but The DVPP strategy is switched to frame skipping keep-alive mode;

[0058] like ,but The DVPP strategy is switched to deep preprocessing mode.

[0059] This application monitors the two-dimensional rate in real time (i.e.) and This enables the switching of the DVPP strategy, and the adjusted strategy determines how DVPP reads image data from DDR memory (such as batch reading / single frame reading) and how it processes the data after reading (such as whether to perform additional preprocessing).

[0060] In practical applications, congestion control scenarios will be encountered (i.e.) In scenarios where data collection is faster than processing and computational mining is used (i.e., data collection is faster than processing), data mining is used. (Processing is faster than acquisition). In congestion control scenarios, the inference time increases due to conveyor belt acceleration or the detection of complex defects. At this time, the DVPP strategy switches to frame skipping keep-alive mode. That is, the FPGA acquisition card directly discards non-critical frames at the hardware layer based on the timestamp (e.g., discarding 1 frame every N frames). Or, when the Kunpeng CPU is updating the shared descriptor ring and detects that the shared descriptor ring is close to full (i.e., Pending_Num is close to N), it must discard some frames to prevent overflow. In this case, it notifies the Ascend NPU to skip the inference of the frame and only retain the critical frames for quality inspection. In computing power mining scenarios, the conveyor belt slows down or the products are sparse. At this time, the DVPP strategy switches to deep preprocessing mode. That is, the Kunpeng scheduling core issues instructions to enable idle DVPP operators in the Ascend NPU (such as denoising, sharpening, contrast enhancement) or increase the number of inference iterations of the NPU (such as multi-scale inference). Among them, the frame skipping keep-alive mode can prevent DDR memory overflow, ensure that the system does not crash, and prioritize the processing of the latest data; the deep preprocessing mode can utilize the idle computing power of Ascend NPU to improve detection accuracy and image quality, achieving "slow and meticulous work".

[0061] Based on the above scheme, in order to cope with sudden traffic surges, this application proposes a load-aware adaptive DVPP strategy (combined with a scheduling core), when R...in Much larger than R proc When the buffer is about to overflow, the DVPP strategy automatically switches to frame skipping keep-alive mode, i.e., batch processing mode. Utilizing the DVPP batch processing interface, multiple images are decoded at once, significantly reducing hardware configuration overhead and improving throughput. When the system is under high load, the scheduler (i.e., the scheduling core) can dynamically adjust the batch size sent to the AI ​​model, maximizing the Ascend NPU computing power while ensuring detection accuracy, meeting the millisecond-level real-time quality inspection needs of industrial production lines. When the load is extremely high, a degradation and frame skipping strategy is implemented. The scheduling core instructs DVPP to skip complex preprocessing of non-critical frames (such as skipping detailed ROI matting and replacing it with full-image scaling), or directly discard some task descriptors, prioritizing low-latency inference of core frames. In summary, the DVPP strategy is a "zero-copy data stream processing mechanism based on hardware offloading and dynamic parameter adaptation." It not only solves the problem of slow CPU image processing but also perfectly adapts to the complex and ever-changing detection needs of industrial sites through dynamic AIPP and asynchronous pipelines.

[0062] Optionally, the Kunpeng CPU's scheduling core polls the shared descriptor ring to obtain the image input rate and NPU inference throughput. Specifically, the Kunpeng CPU's scheduling core reads the movement frequency of the tail pointer in the shared descriptor ring to obtain the image input rate. For example, if the tail pointer advances 30 times in 1 second, it means that the FPGA has written 30 images to the DDR memory, then the image input rate is... 𝑖𝑛 =30FPS; The Kunpeng CPU's scheduling core reads the movement frequency of the head pointer in the shared descriptor ring or the internal hardware registers of the Ascend NPU (such as the AI ​​core completion interrupt counter) to obtain the NPU inference throughput. For example, if the head pointer advances 28 times in 1 second, it means that the Ascend NPU has processed 28 images, then FPS is 30. 𝑝𝑟𝑜𝑐 =28FPS.

[0063] Based on the above scheme, in a high-performance heterogeneous computing architecture, and The acquisition is achieved by reading the "status / metadata" in DDR memory, rather than by processing massive "image pixel data".

[0064] During operation, the FPGA acquisition card writes image data into DDR memory. The Kunpeng CPU fills in metadata (Phys_Addr, etc.) in the shared descriptor ring and moves the Tail pointer forward. At this point, the task descriptor status becomes "Ready". The scheduling core then polls the shared descriptor ring and finds that the Tail pointer advances (i.e., writes) quickly, while the Head pointer advances (i.e., reads) slowly. Calculations are then performed... Then, the scheduling core determines that the system is congested and switches the DVPP strategy (such as changing the original single-frame processing instruction to a batch processing instruction with Batch Size = 8). Finally, the DVPP unit of Ascend NPU reads the new strategy issued by the scheduling core and, based on the Phys_Addr pointed to by the Head pointer in the shared descriptor ring, pulls image data from DDR memory for decoding, cropping, or format conversion.

[0065] In practical applications, the images captured by the camera are of various sizes (e.g., 1920x1080) and are in YUV or Bayer Raw format. However, the backend AI models (such as YOLO) usually only recognize fixed-size (e.g., 640x640) RGB format. In order to adapt to the input requirements (tensor alignment) of the AI ​​model, DVPP needs to use hardware interpolation algorithms to directly complete the size scaling and color space conversion when pulling data from DDR memory, so as to ensure that the data sent to the AI ​​core is "plug and play". Meanwhile, if the raw data or compressed bitstream (such as JPEG) from industrial cameras is decoded and scaled by the CPU, it will instantly fill up the CPU cores, making it unable to perform monitoring scheduling and post-processing tasks. DVPP, however, is an independent hardware accelerator within the NPU that can complete these heavy pixel-level operations with almost zero CPU utilization, thus effectively freeing up the computing power of the Kunpeng CPU (hardware offloading). In addition, without DVPP, the data flow is: DDR → CPU memory → NPU video memory, requiring two memory copies. With DVPP, the data flow is: DDR → DVPP direct reading → AI core (zero-copy pipeline). The data flows entirely on the device side without passing through the Kunpeng CPU, completely eliminating the bandwidth bottleneck of the PCIe bus. This is the physical basis for achieving "low latency" in the system.

[0066] Furthermore, the method also includes: calculating the load index of Kunpeng CPU and Ascend NPU in real time, and adaptively scheduling strategies based on the calculated load index, specifically including:

[0067] According to the formula The load index of the Kunpeng CPU is calculated; where, The Kunpeng CPU load index represents the load of the central processing unit, that is, the utilization rate of the general computing unit (i.e., the Kunpeng CPU core, which is a general-purpose processor core based on the ARM architecture). In AI systems, the CPU is usually responsible for non-AI computing tasks such as task scheduling, data preprocessing, and post-processing. This load index not only reflects computing power, but also focuses on reflecting IO throughput pressure. Core utilization rate represents the CPU utilization rate of the core where the scheduler is located and adjacent cores (normalized from 0 to 1). This represents the memory bandwidth pressure, indicating the proportion of current DDR memory read / write bandwidth to its theoretical peak. Since image transmission is extremely memory bandwidth-intensive, this value is relatively low. Set it to the highest level (e.g., 0.5); The shared descriptor ring stacking degree represents the proportion of unprocessed slots in the shared descriptor ring. If the ring is almost full, it indicates that the CPU is producing too fast or the NPU is consuming too slowly, and this value will surge. , and Each of these represents a weighting coefficient, and the sum of the three is 1; in practice, =0.2, =0.5, =0.3;

[0068] According to the formula The Ascend NPU load index is calculated; where, The Ascend NPU Load Index represents the overall load of the neural network processing unit and is used to reflect the computing power saturation of AI inference and preprocessing, i.e. how busy the NPU is when performing AI inference tasks. AI core utilization rate, representing the utilization rate of matrix calculation units obtained by reading the NPU's underlying registers; DVPP utilization rate indicates the busy status of the digital vision preprocessing subsystem (responsible for decoding, scaling, and cropping); and Both represent weighting coefficients, where, As a weighting factor for AI computing power, it determines the impact of AI core utilization on the total load; The weighting factor for preprocessing / auxiliary computing power determines the impact of DVPP utilization on the total load; in practice, it is dynamically adjusted according to the task type. and If it is a pure reasoning task, then Conversely, for heavy preprocessing tasks, the opposite applies; pure inference tasks refer to application scenarios where the DVPP preprocessing workload is minimal (e.g., the image is already in the format required by the model), and the NPU's computing resources are almost entirely used for AI inference (matrix multiplication / convolution operations). In this case, the AI ​​core utilization is high, while the DVPP utilization is low, therefore the weighting factor... It should approach 1. Approaching 0), a typical scenario is when the image has already undergone external preprocessing, and the NPU only needs to perform inference; heavy preprocessing tasks refer to application scenarios where the DVPP preprocessing workload is large (such as requiring complex decoding of Raw images, multi-scale scaling, color space conversion, etc.), the DVPP unit is busy, and AI inference waits for the preprocessing results. In this case, the DVPP utilization is high, while the AI ​​core utilization is relatively low, therefore the weight factor... It should be increased. The corresponding reduction is typical of scenarios where the input is a high-resolution Raw image, requiring a large number of DVPP operations to convert it into the model input format.

[0069] According to the formula The load index difference was calculated. In the formula, This represents the difference in load index.

[0070] Based on the calculated load index difference and the preset lag threshold, mode switching is performed, specifically including: when When the time comes, switch the mode to NPU bottleneck mode; where, The preset hysteresis threshold; when When, switch the mode to CPU bottleneck mode; when When the time is right, switch to balanced mode. Among them, NPU bottleneck mode means that the CPU is idle, but the NPU computing power is saturated, and there is a risk of backlog in the shared descriptor ring. CPU bottleneck mode means that the NPU is waiting for data, while the CPU is busy copying data or processing post-processing, resulting in IO blocking.

[0071] Based on the switched mode, an adaptive scheduling strategy is adopted; the strategy includes an intelligent batching strategy, an operator unloading and zero-copy strategy, and a single-frame pipeline mode strategy that maintains the standard.

[0072] In practical applications, the scheduler maintains a finite state machine (FSM) that switches modes based on the load index difference and a preset lag threshold. The purpose of introducing the lag threshold is to prevent frequent mode jitter. When the load index fluctuates slightly around the threshold, without lag, the system would repeatedly switch between modes (ping-pong effect), leading to frequent changes in the scheduling strategy and reducing system efficiency. With the lag threshold, mode switching only occurs when the load index difference clearly exceeds the threshold, ensuring system stability.

[0073] Optionally, an adaptive scheduling strategy is adopted based on the switched mode, specifically including:

[0074] When switching to NPU bottleneck mode, an intelligent batching strategy is executed. Specifically, when this strategy is implemented, the CPU no longer sends each image as it arrives; instead, it temporarily stores the image pointer (i.e., the value of the Phys_Addr field) in a local queue. Once a batch is accumulated, it is written to multiple task descriptors at once. This strategy is triggered when the queue length reaches the preset batch processing threshold N{batch} or the waiting time exceeds the maximum latency T. max_latrncy At intervals of 5ms or less, a batch submission is triggered to ensure real-time performance. At this time, the NPU's "multi-frame inference interface" is called to submit the N... batchThe strategy of stitching images together into a tensor and sending it to the NPU effectively utilizes the NPU's large matrix computation advantage, significantly reducing kernel startup overhead and improving throughput. Here, N{batch} is a preset batch processing threshold, representing a "threshold value" or "target number." The CPU continuously collects images and adds them to a queue until the number of images in the queue reaches N{batch}, at which point the "shipment" action is triggered. max_latrncy N refers to the maximum time a task is allowed to wait in the queue. batch This represents the actual amount of data during execution. It is the target batch size dynamically calculated based on the available computing power of the current NPU. When the triggering condition is met, the number of images processed when the system performs the stitching operation is the preset batch processing threshold.

[0075] When switching to CPU bottleneck mode, an operator offloading and zero-copy strategy is implemented. In the traditional process, the NPU returns the result after inference, and the CPU then performs NMS (non-maximum suppression) or coordinate restoration. However, this strategy dynamically loads a custom operator library on the NPU side through the scheduler, and directly sends the NMS task to the NPU for execution. The CPU only receives the final structured text result (tens of bytes), which greatly reduces CPU interrupts and computational burden. If the hardware supports it, the scheduler is configured with an IOMMU, which allows the image data acquired by the camera to be transferred to the FPGA first through the Camera Link / GigE interface. Then, the DMA engine on the FPGA directly writes the data into the NPU's HBM (high bandwidth memory) or dedicated SRAM (static random access memory), completely bypassing the CPU's system memory copying process. If the CPU load is extremely high and a loss of accuracy is acceptable, a downsampling strategy is adopted. The scheduler instructs the FPGA or DVPP to perform 2x2 or 4x4 hardware downsampling directly at the input, reducing the amount of data from the source.

[0076] When switching to balanced mode, a single-frame pipeline mode strategy that maintains the standard is executed to keep the lowest latency.

[0077] Based on the above scheme, the DVPP strategy and the adaptive scheduling strategy are two complementary strategy layers. The DVPP strategy, as the first layer (fine-grained / local optimization), is a fast and lightweight adjustment of the DVPP unit's own operating mode, with a response time in the microsecond range. The adaptive scheduling strategy, as the second layer (coarse-grained / global optimization), is a system-level, more macroscopic resource scheduling decision, with a response time in the millisecond range. The two can run simultaneously and work collaboratively. For example, if the system is currently in NPU bottleneck mode (determined by the adaptive scheduling strategy), an intelligent batching strategy is executed; simultaneously, the DVPP strategy can adjust the scheduling based on R... in and R procThe specific difference determines whether DVPP uses frame skipping or deep processing. In short, the adaptive scheduling strategy determines the "general direction," while the DVPP strategy determines "how DVPP hardware should be implemented."

[0078] Furthermore, the Kunpeng CPU also needs to execute a data memory barrier instruction to ensure that the task descriptor has been completely written into DDR memory, preventing the NPU from reading dirty data due to out-of-order execution by the CPU. The data memory barrier is an assembly instruction under the ARM architecture, whose core function is to enforce the execution order of memory accesses, preventing data inconsistency caused by "out-of-order execution" performed by the CPU to improve performance. This operation requires the CPU to complete all memory write operations before the data memory barrier instruction (i.e., the data is actually written to DDR memory) before it can continue executing any memory access instructions after the data memory barrier instruction. It does not guarantee that the data will be "seen" by other processors, only that the order and completion of the "write operations" are guaranteed.

[0079] Furthermore, the Kunpeng CPU also needs to perform atomic updates, which is equivalent to "opening the door" to notify the Ascend NPU that there is a new task. Specifically, it performs the atomic addition operation Atomic_Add(&Prod_Idx, 1), which is to perform an uninterruptible increment operation on the variable Prod_Idx to ensure that the value of the variable will not be interfered with by other processors or threads in a multi-core or concurrent environment, thereby ensuring data consistency.

[0080] Furthermore, the Ascend NPU runs a lightweight real-time firmware (RTOS or Bare-metal) responsible for polling and hardware scheduling (i.e., calling the DVPP hardware engine).

[0081] Specifically, the Ascend NPU performs lightweight polling, including the following steps:

[0082] The Ascend NPU firmware continuously reads Prod_Idx and Cons_Idx in the main loop:

[0083] If Prod_Idx == Cons_Idx, it indicates that the loop is empty, and WFE or short-term sleep (such as usleep(10)) is executed to reduce power consumption; where WFE is a wait event, which is a low-level hardware instruction under the ARM architecture, mainly used to reduce power consumption; if Prod_Idx != Cons_Idx, it indicates that there is a new task;

[0084] Read the task descriptor in Ring[Cons_Idx % N] to obtain a new task; specifically, the Ascend NPU firmware calculates the modulo value of the current Cons_Idx (i.e., the consumption pointer) with respect to the ring buffer depth N (Cons_Idx % N), locates the (Cons_Idx % N)th slot in the shared descriptor ring array Ring, and reads the task descriptor stored in that slot;

[0085] Newly acquired tasks are validated using Checksum and Magic Number checks to prevent memory data corruption. If a check fails, the status is marked as HW_Error and the process is skipped. Checksum is a checksum / CRC check, and Magic Number is a magic number check. Status = HW_Error indicates a hardware-level error. When Ascend NPU detects a check failure (incorrect Magic Number or Checksum mismatch), it knows that the current image data or task parameters are unreliable. In this case, Ascend NPU will mark HW_Error on the status bit of the descriptor. After marking this, Ascend NPU will skip the processing of this frame and will not send incomplete data to the AI ​​model to prevent the model from calculating incorrect results or even crashing. Instead, it will increment Cons_Idx by 1 and continue to the next frame.

[0086] More specifically, the checksum verification process includes: Before the Kunpeng CPU writes the task descriptor into the shared descriptor ring, it performs mathematical operations (such as CRC32 or simple XOR summation) on the key data (such as address and size) in the task descriptor, and fills the result into the checksum field; after the Ascend NPU reads the data, it recalculates using the same algorithm: if the calculated result is equal to the value in the field, the verification is considered successful, indicating that no bit flips occurred during data transmission; if the calculated result is not equal to the value in the field (even if it differs by only 1 bit), the verification is considered to have failed.

[0087] More specifically, the Magic Number verification process includes: When the Kunpeng CPU writes to the task descriptor, it writes a pre-defined special value (such as 0xDEADBEEF) at a fixed location; when the Ascend NPU reads the memory, it checks whether the value at this location is the pre-defined special value: if it is, the verification is considered successful, indicating that the memory has most likely not been accidentally overwritten; if it is not, the verification is considered to have failed, indicating that the memory data has been corrupted or the NPU has read the wrong address.

[0088] Specifically, the Ascend NPU performs hardware scheduling, including the following steps:

[0089] The configuration includes DVPP units in AIC or AIV. Ascend NPU does not directly perform pixel calculations. Among them, AIC stands for AICore, which is the AI ​​core, and AIV stands for AI Vector, which is the AI ​​vector.

[0090] Configure DMA channels: Write src_phys_addr (source physical address) from the task descriptor to the source address register of DVPP, and write dst_phys_addr (destination physical address) to the destination address register;

[0091] Configure the operator register: Write parameters such as crop_x / y / w / h and resize_ratio into the DVPP control register; where crop_x / y / w / h and resize_ratio are specific control parameters for the cropping operator, x and y represent the coordinates of the top left corner of the cropping region, indicating the position in the image from which to start cropping; w and h represent the width and height of the cropping region, i.e., the size of the cropped image; resize_ratio represents the scaling ratio;

[0092] Send a start command to DVPP;

[0093] Asynchronous execution and write-back, specifically:

[0094] DVPP hardware independently performs image scaling, cropping, and format conversion (such as YUV420SP→RGB888).

[0095] During execution, the Ascend NPU core can continue to check the next task descriptor (if out-of-order or double buffering is supported), or wait for DVPP to complete the interrupt;

[0096] After the task is completed, the firmware updates the status=1 in the task descriptor and executes the atomic addition operation Atomic_Add(&Cons_Idx, 1). Specifically, it performs an uninterruptible increment operation on Cons_Idx. After updating the status, the Ascend NPU must inform the system that "this frame has been processed and the next frame can be moved on". Here, Atomic_Add is a hardware-level atomic instruction that ensures that Cons_Idx will not be interrupted or interrupted by other threads during the increment process. status = 1 indicates that the task status is set to "completed". After the DVPP hardware completes scaling, cropping, and format conversion, the NPU firmware will change the status field in the current task descriptor from "processing" to 1 (representing Done / Success).

[0097] Furthermore, the raw pulse signal from the encoder is acquired using an FPGA acquisition card and preprocessed to obtain a standardized pulse signal, specifically including:

[0098] The original pulse signal of the encoder is read through the external interface of the FPGA acquisition card. In specific implementation, the external interface of the FPGA acquisition card is a high-speed GPIO interface or a dedicated quadrature decoding interface; the original pulse signal of the encoder is the A / B phase quadrature pulse signal.

[0099] The read raw pulse signal is preprocessed to obtain a standardized pulse signal. Specifically, the preprocessing includes level conversion and noise reduction and shaping. The standardized pulse signal is a standard TTL / CMOS level signal. In practical applications, the raw pulse signal needs to be converted into a standard TTL / CMOS level signal that can be recognized by the internal logic of the FPGA acquisition card.

[0100] In practical applications, a sliding window filter is built inside the FPGA to perform high-frequency acquisition of the original pulse signal input by the encoder (such as a sampling rate of 10MHz), and calculate the target instantaneous speed of the conveyor belt by counting the number of pulses in the window, thereby filtering out high-frequency glitches caused by mechanical vibration and avoiding false triggering.

[0101] Furthermore, based on the obtained standardized pulse signal, image data is acquired, specifically including:

[0102] The target instantaneous speed of the conveyor belt is calculated based on the obtained standardized pulse signal;

[0103] Based on the calculated target instantaneous speed of the conveyor belt and the preset physical distance parameters, the compensation delay time is calculated, specifically including: according to the formula The compensation delay time is calculated, which is the time interval between the next trigger; where, To compensate for the delay time, that is, the delay time from the detection of the object arriving at the trigger point to the camera should expose; L is the preset physical distance parameter, that is, the actual distance from the trigger point to the shooting point (determined by process requirements), such as the fixed physical distance from the camera optical center to the actuator (such as the spray valve or robotic arm), in mm. This parameter is written into the FPGA acquisition card as a system register configuration. The system fixed delay, i.e., the fixed circuit response delay of the system (obtained through calibration), includes image acquisition time, algorithm processing time, I / O port response time, etc., and is introduced... A compensation mechanism is in place to correct system errors.

[0104] Based on the calculated compensation delay time, the hard trigger delay calibration signal is obtained, specifically including: according to the formula The calculated compensation delay time is converted into the number of clock cycles within the FPGA acquisition card to obtain the hard-triggered delay calibration signal; that is, the number of clock cycles is the numerical representation of the hard-triggered delay calibration signal; where, To compensate for the delay, This serves as the reference clock cycle for the operation of the FPGA's internal logic. This refers to the number of clock cycles inside the FPGA acquisition card when it is detected. When the value approaches 0, i.e., when the conveyor belt stops, the logic circuit forces the value to... Set to the maximum value or maintain the previous state to prevent division-by-zero overflow errors. Additionally, convert the calculated compensation delay time into the number of clock cycles within the FPGA for subsequent counting.

[0105] The acquired external trigger signal is subjected to hardware-level debouncing to obtain a valid signal; in specific implementation, the valid signal is the valid trigger flag bit; the external trigger signal is a raw signal input from outside the FPGA (such as a trigger command from a PLC, photoelectric sensor or host computer), which has not been processed and may contain electromagnetic interference or mechanical jitter; the rule for hardware-level debouncing is "continuous sampling confirmation method";

[0106] The hard trigger signal is obtained based on the hard trigger delay calibration signal and the valid signal; the hard trigger delay calibration signal provides a delay reference, and the valid signal provides confirmation of the trigger timing.

[0107] A hard trigger signal is sent to an external camera to trigger the camera to expose and obtain image data. This image data, which contains a precise timestamp and physical address pointer, is directly written into a circular buffer in DDR memory.

[0108] Based on the above scheme, by calculating the compensation delay time, microsecond-level "velocity feedforward compensation" can be achieved. The trigger timing is dynamically adjusted according to the current speed to ensure a fixed imaging position. Even if the conveyor belt starts or stops or the load changes causing vibration, the imaging position can still be guaranteed to be absolutely accurate. Furthermore, the calculated compensation delay time is shorter when the object moves at a high speed and longer when the speed is slow, thus ensuring that regardless of how the production line accelerates or decelerates, the camera always exposes when the object reaches the designated position.

[0109] Optionally, the target instantaneous speed of the conveyor belt is calculated based on the obtained standardized pulse signal. Specifically, this includes: calculating the instantaneous speed of the conveyor belt based on the obtained standardized pulse signal; optimizing the calculated instantaneous speed of the conveyor belt to obtain the target instantaneous speed of the conveyor belt; in specific implementation, an optimization algorithm (i.e., a first-order low-pass filtering algorithm) is used to optimize the calculated instantaneous speed of the conveyor belt, effectively reducing the jitter of the instantaneous speed of the conveyor belt.

[0110] Optionally, the instantaneous speed of the conveyor belt is calculated based on the obtained standardized pulse signal, including:

[0111] According to the formula The instantaneous speed of the conveyor belt is calculated; where, T represents the instantaneous speed of the conveyor belt. sample The sampling period is a fixed time window (e.g., 1 ms); N pulse The number of pulses captured within the sampling period is obtained by counting the rising edges of the normalized pulse signal within each window. This represents the physical displacement depicted by a single encoder pulse, i.e., the physical distance the conveyor belt moves for each encoder output pulse, measured in mm / pulse. It represents the pulse equivalent. C is the circumference of the roller, representing the distance the roller travels per revolution, in mm; R is the number of encoder lines, which is the number of pulses output by the encoder per revolution, in pulses / revolution.

[0112] Or according to the formula The instantaneous speed of the conveyor belt is calculated; where, For high-frequency reference clock frequency (e.g., 100MHz, i.e., period T) clk =10ns); M is the number of clock cycles between the rising edges of two adjacent encoder pulses, which can be measured using the high-frequency master clock inside the FPGA (such as 100MHz); K is the encoder resolution constant.

[0113] Based on the above scheme, a hybrid frequency and period measurement method (M / T method) or a high-frequency clock counting method is used to perform real-time calculations on the obtained standardized pulse signals, with the high-frequency clock counting method applied to high-speed pipeline scenarios. Furthermore, when using the high-frequency clock counting method for real-time calculation, boundary protection is required, i.e., a division-by-zero protection logic is added. When M is detected to be too large (i.e., the speed approaches 0), the current state is automatically locked or a standby mode is entered to prevent calculation overflow.

[0114] Specifically, the calculated instantaneous speed of the conveyor belt is optimized to obtain the target instantaneous speed of the conveyor belt, which specifically includes: according to the formula The target instantaneous speed of the conveyor belt is calculated; where, The target instantaneous speed of the conveyor belt; The target instantaneous speed of the conveyor belt at the previous moment; It is the smoothing coefficient, and =0.1~0.3.

[0115] In practical applications, a hardware divider IP core is instantiated within the FPGA to calculate the time difference between signal transmission and mechanical motion in real time, i.e., to compensate for the delay time. The divider IP core uses the DividerGenerator IP of Xilinx / Intel FPGA, or is implemented through a pipelined divider. In addition, since the formula for calculating the compensation delay time includes division, hardware triggers need to be configured. However, in many cases, the calculation of the target instantaneous speed of the conveyor belt already includes division, so multiple division / multiplication units may be needed within the FPGA.

[0116] Optionally, the acquired external trigger signal is subjected to hardware-level debouncing to obtain an effective signal, specifically including: according to the formula The clock cycle threshold is obtained, where, The clock cycle threshold. The preset debouncing threshold time (e.g., 1ms); when a high level is detected in the acquired external trigger signal, a counter set inside the FPGA acquisition card... Accumulate; Counter During the accumulation process, if the external trigger signal goes low, it is determined to be glitches or interference. Reset immediately; when When this happens, output the internal valid trigger flag. That is, a valid signal is obtained; among which, It is a status flag generated after logic processing inside the FPGA.

[0117] In practical applications, a high-precision programmable delay generator (PLD) is used as the execution end. When set, the current bit is locked. The value is used to freeze the current speed calculation result to ensure consistency in a single trigger. Then, a decrementing counter is started, with an initial value that is locked. The counter decrements by 1 every clock cycle, and flips when it reaches 0. The signal outputs a high-level pulse. The pulse width is controlled by another register (e.g., set to 50μs), and automatically pulls low after the pulse ends, waiting for the next trigger. This indicates the camera trigger output signal, which is the final physical output of the entire FPGA logic processing flow and is directly connected to the hardware interface of the industrial camera. When... When the voltage level is low (0), the camera is in standby mode and does not take pictures; when... When the flip is high (1), the camera detects the rising edge (or high level) and immediately performs an exposure.

[0118] Specifically, a digital filter is connected in series in the trigger path to address potential electromagnetic interference or mechanical jitter in the external trigger signal.

[0119] Based on the above scheme, this application uses a countdown execution and hardware-level de-jitter processing method to obtain the hard trigger signal. On the one hand, after receiving the hard trigger delay calibration signal instruction, the high-precision counter inside the FPGA is started to count down. On the other hand, digital filtering logic is added before the output hard trigger signal. Only when the trigger condition meets the requirement that the duration exceeds a preset threshold (e.g., filtering out electromagnetic interference glitches less than 1μs) is it determined to be a valid signal. When the countdown ends and the verification passes, the output pin is pulled high / low to generate a standard hard trigger signal (such as an optocoupler isolation signal or a TTL level signal) to drive the external camera to expose.

[0120] Furthermore, before sending a hard trigger signal to the external camera to trigger its exposure and obtain image data, the process also includes:

[0121] Real-time detection of the conveyor belt speed change rate; in specific implementation, the speed change rate is acceleration.

[0122] Based on the rate of change of speed, the preset system sensitivity constant, and the instantaneous speed of the target on the conveyor belt, the target exposure time value is obtained, specifically:

[0123] when At that time, according to the formula The target exposure time value is calculated, where, This is the rate of change of velocity, i.e., acceleration; This is a preset system sensitivity constant, which is a preset system sensitivity constant for the start and stop phases of the conveyor belt and is used to define the stability boundary of the conveyor belt motion. The target exposure time value is a new, dynamic exposure time value. The baseline exposure time value represents the standard exposure time (e.g., 5000μs) when the conveyor belt moves at a constant speed and the lighting is stable, and is used to ensure normal image brightness. The maximum allowable motion blur pixel equivalent is a dimensionless value in pixels, representing the system's tolerance for motion blur (values ​​range from 1, 2, 3, ...). To prevent blurring, the maximum allowable exposure time is defined as the amount of pixel trailing within the exposure time corresponding to every 1mm movement of the object. In practice, the more drastic the speed change, the shorter the exposure time. If the target's instantaneous speed is very high, the target exposure time value will become very small (e.g., 1000μs). That is, in order to avoid image blurring, the exposure time must be shortened to below this value.

[0124] Based on the above scheme, this application obtains image data through implicit triggering actions (the starting point of data), explicit data output (the entity of data), and data confirmation and flow (the landing of data). The implicit triggering action involves dynamically calculating the compensation delay time to obtain a hard trigger delay calibration signal. Then, based on the hard trigger delay calibration signal and the valid signal, a hard trigger signal is obtained and sent to the camera. The camera begins exposure the instant this signal is emitted. This implicit triggering action is the "starting gun" for image data generation; without the triggering of the compensation delay time calculated based on the instantaneous speed of the conveyor belt, there would be no subsequent image data. The explicit data output is the exposure compensation based on acceleration sensing, which yields the target exposure time value. Based on this value, the FPGA automatically adjusts the camera's exposure time register. 𝑒𝑥𝑝 And when the camera receives the trigger, it will execute the FPGA settings as configured. 𝑒𝑥𝑝 The exposure is time-sensitive. At the instant the exposure ends, the analog light signal is converted into a digital electrical signal, which is the physical moment of image data generation. All calculations before obtaining the image data are to ensure that the data generated at this moment is clear and accurately positioned. The confirmation and transfer of data, that is, the final image data, carries a precise timestamp and physical address pointer, and is directly written into the circular buffer of DDR memory. This is the most direct written confirmation of the obtained image data. The timestamp indicates that the data has been processed by the FPGA (i.e., tagged), and the writing to DDR indicates that the data has been transferred from the camera to the system memory, becoming a valuable asset that can be used by subsequent AI models. The FPGA ensures the quality and timing accuracy of the input data through high-frequency speed sampling, hardware de-shake, dynamic trigger delay calculation, and synchronous exposure compensation based on acceleration sensing.

[0125] In summary, this application can achieve the following effects: (1) Elimination of dynamic errors: Traditional methods use fixed delays, which can cause the trigger position to shift when the speed of the conveyor belt fluctuates (such as the acceleration and deceleration of the motor). However, this invention ensures that the camera triggering time always accurately corresponds to the physical position of the object, regardless of the speed change, by real-time calculation of the compensation delay time; (2) Hardware-level anti-interference: The parallel logic of the FPGA is used to realize microsecond-level digital de-jittering, which can filter high-frequency noise more accurately and prevent false triggering compared to software de-jittering (ms level); (3) Zero CPU occupation: The entire speed acquisition, calculation, de-jittering, and delay triggering process is completed in a closed loop in the FPGA hardware logic without the need for upper computer CPU intervention, which greatly reduces system latency and jitter.

[0126] Secondly, this application provides a low-latency visual inspection system based on the Kunpeng Ascend heterogeneous platform, the system comprising:

[0127] Industrial cameras connect to the back-end FPGA acquisition card via Camera Link or GigE interfaces to transmit Raw format image data. Camera Link and GigE are two commonly used physical interface standards for industrial cameras. GigE is based on Gigabit Ethernet and has a long transmission distance, while Camera Link is an interface specifically designed for high-speed machine vision with higher bandwidth.

[0128] The encoder is installed on the end of the conveyor belt roller shaft. Its signal line is directly connected to the differential input port of the FPGA acquisition card, and it outputs A / B phase quadrature pulse signals that are proportional to the speed of the conveyor belt in real time.

[0129] The FPGA data acquisition card's internal logic modules include: an encoder input module for acquiring the encoder's raw pulse signals (i.e., A / B phase quadrature pulse signals) and improving resolution through frequency quadrature technology; a preprocessing module for preprocessing the acquired raw pulse signals to obtain standardized pulse signals; a speed calculation module (e.g., an instantaneous speed calculator) for calculating the target instantaneous speed of the conveyor belt based on the obtained standardized pulse signals; a delay calculation module (e.g., a dynamic delay generator) for calculating the compensation delay time based on the calculated target instantaneous speed of the conveyor belt and preset physical distance parameters; and a calibration signal acquisition module. The system comprises several modules: a hard trigger delay calibration module for obtaining a hard trigger delay calibration signal based on the calculated compensation delay time; an effective signal acquisition module for performing hardware-level debouncing on the acquired external trigger signal to obtain an effective signal; a trigger output module for obtaining a hard trigger signal based on the acquired hard trigger delay calibration signal and the effective signal, and then sending the hard trigger signal to the external camera (i.e., the industrial camera) to trigger exposure; a DMA engine for directly transferring image data to the DDR memory of the Kunpeng computing module; and an orthogonal decoder module built inside the FPGA acquisition card for performing quadrature frequency counting on the encoder signal to improve speed measurement resolution.

[0130] The Kunpeng computing module includes a Kunpeng CPU and DDR memory. The Kunpeng CPU is a multi-core processor based on the ARMv8 instruction set (such as the Kunpeng 920), supporting a multi-channel DDR4 memory controller. It is divided into control cores (Core 0-3) and scheduling cores (Core 4-7). The control cores run the operating system (such as EulerOS), manage the state of the shared descriptor ring, and handle exceptions and interrupts. The scheduling cores poll the shared descriptor ring, monitor the image input rate and NPU inference throughput, calculate the load index, dynamically adjust the DVPP policy, and execute adaptive scheduling policies. A physically contiguous large page memory region is allocated from the DDR memory as a "zero-copy buffer," i.e., a circular buffer. This region is mapped as shared memory for the FPGA to write image data and for the Ascend NPU to read data. Furthermore, dividing the Kunpeng CPU into control and scheduling cores aims to avoid scheduling tasks and system management tasks competing for CPU resources, ensuring the real-time nature of scheduling decisions.

[0131] The Ascend AI acceleration module, including the Ascend NPU (such as Ascend 310 / 910), has internal core units including DVPP and AI core (Cube / Vector unit). The DVPP contains independent hardware decoders, croppers, and scalers, and has an independent DMA channel for reading image data from DDR memory. The AI ​​core is used to perform matrix operations (i.e., matrix multiplication or convolution operations, which are existing technologies and will not be elaborated here) of deep learning models, perform inference on preprocessed image data, and output the inference results.

[0132] The high-speed interconnect bus (PCIe) connects the FPGA acquisition card and the Ascend AI acceleration module to the Kunpeng computing module via PCIe. In practice, the FPGA acquisition card and the Ascend AI acceleration module are connected to the Kunpeng CPU's PCIe root complex via a PCIe switch chip.

[0133] Specifically, the industrial camera uses a global shutter area array or linear array industrial camera; the encoder is a rotary encoder.

[0134] In practical implementation, the modules within the FPGA's internal logic module employ a linear, unidirectional data flow, achieving closed-loop control throughout the entire process from physical motion signal acquisition to precise triggering. Specifically, the encoder input module, serving as the system's sensing entry point, physically connects to an external rotary encoder or linear encoder to acquire and standardize motion feedback signals in real time—that is, to capture raw pulse signals reflecting the object's motion state. The preprocessing module preprocesses the acquired raw pulse signals into standardized pulse signals and transmits them in real time to the speed calculation module, serving as the fundamental data source for speed measurement. Upon receiving the standardized pulse signals, the speed calculation module utilizes the FPGA's internal high-frequency clock to perform real-time calculations, converting discrete pulse frequencies into continuous physical velocity quantities (i.e., the target instantaneous velocity), and then displays the calculated target instantaneous velocity in real time. The data is passed to the delay calculation module, providing a precise basis for subsequent dynamic delay. This module dynamically calculates the required compensation delay time to offset system response lag. It also implements "velocity feedforward compensation," dynamically adjusting the trigger timing based on the current velocity to ensure a fixed imaging position. The calibration signal acquisition module obtains the hard trigger delay calibration signal. The effective signal acquisition module performs hardware-level de-jitter processing on the acquired external trigger signal to prevent false triggering. The trigger output module, acting as the execution terminal, receives the delay-calibrated signal and the hardware-level de-jitter processed effective signal, and sends the final hard trigger signal to the external camera or detection device. The entire process is executed in parallel within the hardware logic (i.e., FPGA), without CPU intervention, ensuring microsecond-level low-latency response characteristics.

[0135] In practical applications, to more clearly describe the relationships between the components, the signal flow during system operation is divided into three main lines: hard-triggered control flow (microsecond-level response), zero-copy data flow (high-bandwidth transmission), and heterogeneous collaborative scheduling flow (millisecond-level decision-making). The hard-triggered control flow path includes conveyor belt movement, encoder pulses, target instantaneous speed calculation by the FPGA acquisition card, dynamic delay calculation, hard-triggered signal acquisition, hard-triggered signal output, and industrial camera exposure. This closed loop is entirely completed within the FPGA hardware logic, without passing through the Kunpeng CPU, completely eliminating triggering errors caused by operating system jitter. The zero-copy data flow path includes the industrial camera, the FPGA acquisition card's DMA engine, the Kunpeng DDR memory's circular buffer (physical address pass-through), Ascend DVPP (hardware DMA fetch), and the AI ​​core. In this signal stream, data is copied from the industrial camera to the Ascend NPU without the participation of the Kunpeng CPU. The Kunpeng CPU only notifies the Ascend NPU by updating the task descriptor in the shared descriptor ring, thus achieving "separation of control and data". The heterogeneous collaborative scheduling flow path is: FPGA reporting acquisition rate, Kunpeng scheduling core (calculation and difference), generating configuration instructions, Ascend CANN driver, adjusting DVPP preprocessing operators (such as enabling / disabling noise reduction) or NPU inference batches. This signal stream is based on dynamic feedback that is aware of IO rate to prevent memory overflow or idle computing power.

[0136] Based on the above scheme, the industrial camera, encoder, and FPGA acquisition card constitute the image acquisition and trigger control subsystem of the system, namely the front-end perception layer. This layer is responsible for physical signal capture and hard trigger signal generation, which is key to solving "motion blur" and "timing inaccuracies." The FPGA acquisition card, as the core hardware, serves as a bridge connecting the physical world and the digital system, undertaking "hardware de-jitter" and "speed feedforward computing" tasks. The Kunpeng computing module is the system's general-purpose computing layer. Based on the ARM architecture, this module is responsible for IO management, protocol parsing, and system scheduling, and is crucial for resolving "interrupt storms" and "resource contention." The Ascend AI acceleration module is the system's dedicated computing layer. Based on the Da Vinci architecture, this module is responsible for high-throughput image preprocessing and AI inference, and is crucial for solving the "computing power bottleneck." The high-speed PCIe interconnect bus ensures that data written to DDR memory by the FPGA acquisition card can be directly read by the Ascend NPU through the PCIe channel (Peer-to-Peer communication or via memory transfer), achieving separation of data flow and control flow.

[0137] Thirdly, this application provides a device, which can be any device capable of implementing a low-latency visual inspection method based on the Kunpeng Ascend heterogeneous platform. This device can be various terminal devices, such as desktop computers, laptops, tablets, handheld devices, etc., and can be implemented through software and / or hardware. Exemplarily, the device includes: a memory; a processor; and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the low-latency visual inspection method based on the Kunpeng Ascend heterogeneous platform as described above.

[0138] Fourthly, this application provides a computer-readable storage medium, which may be a ROM, RAM, disk, or optical disk, etc. Exemplarily, the computer-readable storage medium stores a computer program; the computer program is executed by a processor to implement the low-latency visual detection method based on the Kunpeng Ascend heterogeneous platform as described above.

[0139] Those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A low-latency visual inspection method based on the Kunpeng Ascend heterogeneous platform, characterized in that: Includes the following steps: The original pulse signal from the encoder is acquired using an FPGA acquisition card and preprocessed to obtain a standardized pulse signal. Image data is obtained based on the standardized pulse signal. The FPGA acquisition card writes the acquired image data into the DDR memory of the Kunpeng computing module through the DMA engine, and the Kunpeng CPU senses that the image data has reached the DDR memory. The DDR memory is divided into a circular buffer, and the circular buffer is equipped with a shared descriptor ring for filling metadata. The shared descriptor ring includes multiple slots, and each slot stores a task descriptor. The Ascend NPU's DVPP reads image data from DDR memory based on a dynamically adjusted DVPP strategy and preprocesses it. The preprocessed image data is input into the deep learning model of the AI ​​core for inference, and the inference results are output; the inference results include the target bounding box coordinates, defect category and confidence level.

2. The low-latency visual detection method based on the Kunpeng Ascend heterogeneous platform according to claim 1, characterized in that: The metadata includes physical address, data size, timestamp, and region of interest information; The circular buffer also includes a head pointer and a tail pointer. The circular buffer uses the movement of the head pointer and the tail pointer to achieve ordered writing and reading of data. The slot status of each slot includes idle, pending, and being processed; The shared descriptor ring also includes a control block located at the head of the shared descriptor ring, which includes a producer index and a consumer index; wherein, the producer index is updated by the Kunpeng CPU and points to the next writable free slot; the consumer index is updated by the Ascend NPU firmware and points to the next unprocessed slot; The task descriptor includes Buf Addr, Length / Flags, and Status. Buf Addr points to the physical memory address where the actual data is stored, Length / Flags records the length of the data written or a status flag, and Status represents the status bits of the task descriptor itself.

3. The low-latency visual detection method based on the Kunpeng Ascend heterogeneous platform according to claim 2, characterized in that: Also includes: The DVPP strategy is dynamically adjusted based on the acquired image input rate and NPU inference throughput, specifically including: The Kunpeng CPU's scheduling core polls the shared descriptor ring to obtain the image input rate and NPU inference throughput, specifically: The Kunpeng CPU's scheduling core reads the movement frequency of the tail pointer in the shared descriptor ring to obtain the image input rate; The Kunpeng CPU's scheduling core reads the movement frequency of the head pointer in the shared descriptor ring or the internal hardware registers of the Ascend NPU to obtain the NPU inference throughput. According to the formula The difference in monitoring indicators is calculated; where, For image input rate, For NPU inference throughput, The difference in monitoring indicators; Based on the calculated difference in monitoring indicators The DVPP strategy is dynamically adjusted as follows: like ,but The DVPP strategy is switched to frame skipping keep-alive mode; like ,but The DVPP strategy is switched to deep preprocessing mode.

4. The low-latency visual detection method based on the Kunpeng Ascend heterogeneous platform according to claim 2, characterized in that: Also includes: The system calculates the load index of Kunpeng CPU and Ascend NPU in real time and adopts an adaptive scheduling strategy based on the calculated load index, specifically including: According to the formula The load index of the Kunpeng CPU is calculated; where, The Kunpeng CPU load index not only reflects computing power, but also focuses on reflecting I / O throughput pressure. Core utilization rate represents the CPU utilization of the core where the scheduler is located and adjacent cores; Memory bandwidth pressure represents the proportion of current DDR memory read / write bandwidth to its theoretical peak. The shared descriptor ring stacking degree represents the proportion of unprocessed slots in the shared descriptor ring. , and Each of these represents a weighting coefficient, and the sum of the three is 1. According to the formula The Ascend NPU load index is calculated; where, The Ascend NPU load index represents the overall load of the neural network processing unit and is used to reflect the computing power saturation of AI inference and preprocessing. AI core utilization rate, representing the utilization rate of matrix calculation units obtained by reading the NPU's underlying registers; DVPP utilization rate indicates the busy status of the digital vision preprocessing subsystem; and All represent weighting coefficients, where, As a weighting factor for AI computing power, it determines the impact of AI core utilization on the total load; The weighting factor for preprocessing / auxiliary computing power determines the impact of DVPP utilization on the total load; According to the formula The load index difference is calculated; where, This represents the difference in load index. Based on the calculated load index difference and the preset lag threshold, mode switching is performed; the modes include NPU bottleneck mode, CPU bottleneck mode and balanced mode. Based on the switched mode, an adaptive scheduling strategy is adopted; the strategy includes an intelligent batching strategy, an operator unloading and zero-copy strategy, and a single-frame pipeline mode strategy that maintains the standard.

5. The low-latency visual detection method based on the Kunpeng Ascend heterogeneous platform according to claim 4, characterized in that: The mode switching based on the calculated load index difference and the preset hysteresis threshold specifically includes: when When this happens, the mode will be switched to NPU bottleneck mode; where, The preset hysteresis threshold; when When necessary, switch the mode to CPU bottleneck mode; when When needed, switch the mode to Balanced mode; The adaptive scheduling strategy based on the switched mode specifically includes: When switching to NPU bottleneck mode, an intelligent batch processing strategy is executed; When switching to CPU bottleneck mode, operator unloading and zero-copy strategy are executed; When switching to balanced mode, the single-frame pipeline mode strategy that maintains the standard is executed.

6. The low-latency visual detection method based on the Kunpeng Ascend heterogeneous platform according to claim 1, characterized in that: The process of obtaining image data based on the obtained standardized pulse signal specifically includes: The target instantaneous speed of the conveyor belt is calculated based on the obtained standardized pulse signal; Based on the calculated target instantaneous speed of the conveyor belt and the preset physical distance parameters, the compensation delay time is calculated. Based on the calculated compensation delay time, the hard trigger delay calibration signal is obtained; The acquired external trigger signal is subjected to hardware-level debouncing to obtain a valid signal; The hard trigger signal is obtained based on the hard trigger delay calibration signal and the valid signal. A hard trigger signal is sent to an external camera to trigger the camera to expose and obtain image data.

7. The low-latency visual detection method based on the Kunpeng Ascend heterogeneous platform according to claim 6, characterized in that: The calculation of the target instantaneous speed of the conveyor belt based on the obtained standardized pulse signal specifically includes: The instantaneous speed of the conveyor belt is calculated based on the obtained standardized pulse signal. The calculated instantaneous speed of the conveyor belt is optimized to obtain the target instantaneous speed of the conveyor belt; The calculation of the compensation delay time based on the calculated target instantaneous speed of the conveyor belt and preset physical distance parameters specifically includes: According to the formula The compensation delay time is calculated; where, To compensate for the delay time; L is a preset physical distance parameter, that is, the actual distance from the trigger point to the shooting point. This parameter is written into the FPGA acquisition card as a system register configuration. The target instantaneous speed of the conveyor belt; Set a fixed delay for the system; The step of obtaining the hard-triggered delay calibration signal based on the calculated compensation delay time specifically includes: According to the formula The calculated compensation delay time is converted into the number of clock cycles inside the FPGA acquisition card to obtain the hard trigger delay calibration signal; where, This serves as the reference clock cycle for the operation of the FPGA's internal logic. This refers to the number of clock cycles inside the FPGA acquisition card. The hardware-level debouncing process for the acquired external trigger signal to obtain a valid signal specifically includes: According to the formula The clock cycle threshold is obtained; where, The clock cycle threshold. The preset de-jitter threshold time; When a high-level external trigger signal is detected, the counter set inside the FPGA acquisition card... Accumulate; counter During the accumulation process, if the external trigger signal goes low, it is determined to be glitches or interference. debounce Reset immediately; when When the internal valid trigger flag is output, a valid signal is obtained.

8. The low-latency visual detection method based on the Kunpeng Ascend heterogeneous platform according to claim 7, characterized in that: The calculation of the instantaneous speed of the conveyor belt based on the obtained standardized pulse signal specifically includes: According to the formula The instantaneous speed of the conveyor belt is calculated; where, T represents the instantaneous speed of the conveyor belt. sample The sampling period is a fixed time window; N pulse This represents the number of pulses captured within the sampling period. This represents the physical displacement represented by a single encoder pulse, in mm / pulse, and represents the pulse equivalent. C is the circumference of the roller in mm, and R is the encoder lines in pulses per revolution. Or according to the formula The instantaneous speed of the conveyor belt is calculated; where, M is the high-frequency reference clock frequency; K is the number of clock cycles between the rising edges of two adjacent encoder pulses; M is the encoder resolution constant. The optimization of the calculated instantaneous speed of the conveyor belt to obtain the target instantaneous speed of the conveyor belt specifically includes: According to the formula The target instantaneous speed of the conveyor belt is calculated; where, The target instantaneous speed of the conveyor belt; The target instantaneous speed of the conveyor belt at the previous moment; This is the smoothing coefficient.

9. The low-latency visual detection method based on the Kunpeng Ascend heterogeneous platform according to claim 6, characterized in that: Before sending a hard trigger signal to an external camera to trigger its exposure and obtain image data, the process also includes: Real-time detection of the conveyor belt speed change rate; The target exposure time value is obtained based on the rate of change of speed, the preset system sensitivity constant, and the instantaneous speed of the target on the conveyor belt, specifically including: when At that time, according to the formula The target exposure time value is calculated, where, The rate of change of velocity; This is a preset system sensitivity constant, which is a preset system sensitivity constant for the start and stop phases of the conveyor belt and is used to define the stability boundary of the conveyor belt motion; The target exposure time value; The baseline exposure time value represents the standard exposure time under conditions of uniform conveyor belt movement and stable illumination. The maximum allowable pixel equivalent of motion blur represents the system's tolerance for motion blur. To prevent blurring, the limit time indicates the maximum permissible exposure time.

10. A low-latency visual inspection system based on the Kunpeng Ascend heterogeneous platform, comprising: Industrial cameras and encoders, characterized in that: they further include: The FPGA acquisition card, electrically connected to industrial cameras and encoders, includes the following internal logic modules: an encoder input module for acquiring the raw pulse signals from the encoder; a preprocessing module for preprocessing the raw pulse signals to obtain standardized pulse signals; a speed calculation module for calculating the target instantaneous speed of the conveyor belt based on the obtained standardized pulse signals; a delay calculation module for calculating the compensation delay time based on the calculated target instantaneous speed of the conveyor belt and preset physical distance parameters; a calibration signal acquisition module for obtaining a hard trigger delay calibration signal based on the calculated compensation delay time; a valid signal acquisition module for performing hardware-level de-jitter processing on the acquired external trigger signals to obtain valid signals; a trigger output module for obtaining a hard trigger signal based on the obtained hard trigger delay calibration signal and valid signals, and then sending the hard trigger signal to the external camera to trigger the external camera's exposure; and a DMA engine for directly transferring image data to the DDR memory of the Kunpeng computing module. The Kunpeng computing module includes a Kunpeng CPU and DDR memory. The Kunpeng CPU is divided into a control core and a scheduling core. The control core is used to run the operating system, manage the state of the shared descriptor ring, and handle exceptions and interrupts. The scheduling core is used to poll the shared descriptor ring, monitor the image input rate and NPU inference throughput, calculate the load index, dynamically adjust the DVPP policy, and execute the adaptive scheduling policy. A large physical contiguous page memory region is allocated in the DDR memory as a zero-copy buffer, i.e., a circular buffer. This region is mapped as shared memory for the FPGA to write image data and for the Ascend NPU to read data. The Ascend AI acceleration module includes the Ascend NPU, whose internal core units include DVPP and AI core. DVPP contains independent hardware decoder, cropper and scaler, and DVPP has independent DMA channel for reading image data in DDR memory; AI core is used to execute deep learning models, perform inference on preprocessed image data, and output the inference results. The high-speed interconnect bus connects the FPGA acquisition card and the Ascend AI acceleration module to the Kunpeng computing module.

Images