A sensor-computer integrated visual localization method and system for robot target localization

Abstract

This invention discloses a sensor-computer integrated visual positioning method and system for robot target localization. Image data is acquired by a sensor board equipped with sensors and sent to a core processor. Image processing and target edge extraction are performed at the programmable logic unit (PL), and the data is transmitted to the processing system unit (PS). The PS reads edge points, performs arc segment extraction to obtain elliptical arc segments, and writes them to the PL. Based on this, the PL combines this with a symmetric matrix to optimize the pulsation array and achieve ellipse fitting to complete target localization. An interface board microcontroller controls the power supply timing and manages external interfaces. The core processor (PS) and peripheral interfaces are responsible for overall system control and Ethernet communication. Integrating image sensing and edge computing into a single hardware component, through collaborative processing, hardware acceleration of key algorithms, optimized data transmission, and integrated management, this method achieves low latency, high accuracy, and high robustness in robot target localization, and possesses advantages such as compact structure, low power consumption, high adaptability, and rapid deployment.

Description

Technical Field

[0001] This invention belongs to the field of robot vision perception and embedded image processing technology, and in particular relates to a sensing and computing integrated vision localization method and system for robot target localization. Background Technology

[0002] Against the backdrop of rapid development in intelligent manufacturing and industrial automation, robot target localization has become one of the important supporting technologies for achieving automated operations. Traditional target localization methods typically rely on the collaborative work of laser rangefinders, industrial cameras, and computer image processing systems. While these systems possess a certain level of accuracy, they still have many limitations in terms of structural integration, real-time response capabilities, and environmental adaptability.

[0003] Existing visual inspection systems typically employ a separate "acquisition-transmission-processing" architecture, where a camera captures images and transmits them to a backend PC for processing. This architecture results in significant image transmission latency and low system response speed, making it difficult to meet the demands of robot dynamic localization and real-time guidance. Furthermore, the overall system size is bulky, hindering flexible deployment on mobile platforms. In addition, changes in ambient lighting, occlusion, and reflection interference can severely impact the detection stability of traditional systems.

[0004] To address the above problems, this invention proposes a sensor-computer integrated visual positioning method and system for robot target localization. Summary of the Invention

[0005] To address the above technical problems, this invention provides a sensor-computer integrated visual positioning method and system for robot target localization.

[0006] The technical solution adopted by this invention to solve its technical problem is:

[0007] A sensor-computer integrated vision localization method for robot target localization, the method comprising the following steps:

[0008] S100: Acquires image data of the target scene through a sensor board equipped with a CMOS sensor and sends it to the core processor;

[0009] S200: On the programmable logic side (PL) of the core processor, based on the FPGA architecture, the acquired image data is processed, the target edge is extracted from the processed image data, and the data is transmitted from the programmable logic side (PL) to the processing system side (PS) of the core processor in a zero-copy manner via the AXI4 bus.

[0010] S300: On the processor running on the processing system side PS of the core processor, the edge coordinate points are read and the arc segment extraction algorithm is executed to obtain the elliptical arc segment. The coordinates of the arc segment points are written to the BRAM memory buffer on the PL side through direct memory access DMA.

[0011] S400: The PL end uses the arc point coordinates combined with the symmetric matrix optimized pulse array to achieve ellipse fitting and complete target localization;

[0012] S500: Controls the power timing of the system and manages external interfaces through the microcontroller on the interface board;

[0013] S600: Through the processing system PS integrated in the core processor and peripheral interfaces, it performs overall system control and Ethernet communication.

[0014] Preferably, the processing of the acquired image data in step S200 includes:

[0015] S210: A dynamically adjustable delay chain is constructed using the input delay unit IDELAY and the input delay control unit IDELAY_CTRL primitive to phase-shift the accompanying clock, so that the sampling clock edge is aligned with the midpoint of the data.

[0016] S220: A 1:10 deserialization operation is achieved through a deserializer, and a Bitslip control unit is cascaded after the deserializer to match the synchronization header through cyclic shifting; channel alignment of 4 data channels is completed through register delay.

[0017] S230: The 12-bit pixel Bayer format data output from the front-end image acquisition is filled with 4-bit check codes every 5 pixels to form a 64-bit data packet, which is written to the DDR4 storage control module through the burst transmission mode of the AXI4 bus. At the same time, an address mapping controller is designed to store the unaligned pixel data continuously across memory banks in row priority order.

[0018] Preferably, step S200 involves target edge extraction from the processed image, including:

[0019] S240: Reads Bayer data stored in DDR4, performs de-mosaicing and grayscale conversion, and forms a 3×3 capture window using a row buffer and a shift register. Unaligned pixel data flows into the row buffer continuously according to the clock cycle, and performs median filtering on all pixels of a frame of image; among them, for parallel pipelined processing, a three-level comparison tree is used to obtain the median.

[0020] S250: For the median-filtered image, a 3×3 capture window is constructed to acquire data and perform multiplication and addition operations on the data in the window; where the initial weights of the Sobel operator are stored in a register, multiplication is implemented using a signed booth multiplier, and accumulation is implemented through a parallel addition tree;

[0021] S260: The two rows of pixel data are stored in a FIFO buffer and, together with the current row of data, are used to construct a 3×3 window structure through a D flip-flop delay. The magnitude of the neighboring pixels of the center pixel in the gradient direction is compared. If the magnitude of the center pixel is less than the magnitude of its neighboring pixels in the gradient direction, the pixel is judged as a non-edge point and suppressed. The gradient magnitude is approximated by the sum of the absolute values ​​of Gx and Gy instead of the Euclidean norm, and the gradient direction is simplified and estimated based on the sign and absolute value relationship of Gx and Gy. Here, Gx is the gradient component of the image in the horizontal direction, and Gy is the gradient component of the image in the vertical direction.

[0022] S270: The current pixel gradient magnitude is compared with the preset high threshold and low threshold by two parallel comparators, and the results are marked as strong edge, weak edge or non-edge respectively; a 3×3 neighborhood window is constructed by using a dual-row buffer and register array, and the edge connection module is controlled by a finite state machine to perform parallel scanning of the 8 neighborhoods of each weak edge pixel. If there is a strong edge point in the neighborhood, the current weak edge pixel is remarked as a strong edge point, otherwise it is discarded, and finally a coherent and complete binary edge image is formed, which is then transferred to the PS memory by DMA zero copy through the AXI4 bus.

[0023] Preferably, S300 includes:

[0024] S310: Design a DMA module based on the AXI4-FULL protocol, which writes the edge coordinates of the buffer on the PL side into the PS DDR memory, and the PS side reads directly through the DMA mapping area; at the same time, configure dual-port RAM and interrupt control logic on the PL side, and trigger an interrupt to notify the PS side when the data is ready;

[0025] S320: Represent the edge coordinates read from the PS terminal as a point set. , the starting point and the finish line Connect the points to form a straight line and calculate the maximum perpendicular distance between this line segment and all intermediate points. The formula for the distance from a point to a line segment is:

[0026] ;

[0027] like If the error exceeds the error threshold ε, then the point is used as the new split point, and the point set is divided into two subsets for recursive processing. Otherwise, the point set is considered to be divisible by a straight line. Approximate representation: The ARM processor uses recursion to implement this logic, combined with a dynamic threshold strategy to adaptively adjust the accuracy, and finally outputs a sequence of polygon vertex coordinates to complete the polygon approximation of the edge;

[0028] S330: An algorithm based on curvature and convexity constraints is used to divide the contour segments of the polygon approximation to obtain elliptical arc segments.

[0029] Preferably, S330 specifically comprises:

[0030] Let the approximated contour segment be denoted as the point set. And define the vector between adjacent vertices as ; any adjacent vector and The included angle between them is expressed as ,in Indicates from Rotate to The angle formed Indicates the index of the vertex sequence. This represents the total number of vertices contained in the approximated contour, and j represents the index of the included angle;

[0031] To ensure that the contour segments form a continuous, smooth, and monotonous arc structure, the following judgment criterion is designed: within a continuous sub-segment, the signs of all angles must remain consistent, i.e., the convexity condition must be satisfied:

[0032] ;

[0033] Meanwhile, to exclude abrupt curvature or acute-angle broken line segments, curvature thresholds and scale constraints are set:

[0034] ;

[0035] in, The maximum allowable angle change. These are scale constraint parameters used to prevent abrupt changes in line segment length. and Represent two adjacent vectors respectively and Length;

[0036] Extract continuous fields that meet the above conditions as elliptical arc segments.

[0037] Preferably, S400 includes:

[0038] S410: The general equation of an ellipse at any position in a plane is expressed as:

[0039] ;

[0040] in, Let A be the coordinates of any point in the image plane coordinate system, and let A, B, C, D, and E be five coefficients describing the shape, position, and orientation of the ellipse. A controls the rotation of the ellipse, B determines the eccentricity and scaling of the ellipse, and C and D control the center of the ellipse at... and Translation of direction For constant terms;

[0041] S420: Based on the least squares principle, the ellipse fitting approximates N data points in a plane. For these N data points, an objective function is established to minimize the sum of squared residuals, specifically:

[0042] ;

[0043] in, For the sum of squared residuals, Here, N is the index of the data point, and N is the total number of data points used for ellipse fitting.

[0044] make The normal equation can be obtained as follows:

[0045] ;

[0046] Further expressed as:

[0047] ;

[0048] Further results were obtained:

[0049] ;

[0050] in, Indicates left division;

[0051] S430: Designing a matrix inversion circuit structure to obtain... The inverse matrix is ​​used to design a matrix multiplication circuit structure for calculation. Finally, the values ​​of the five parameters A, B, C, D, and E that define the ellipse are calculated. Based on the relationship expression between the coefficients of the elliptic curve equation and the coordinates of its center point, the coordinates of the center point of the ellipse are obtained, thus achieving ellipse fitting and completing target localization.

[0052] Preferably, the matrix inversion circuit structure is designed in S430 to obtain... The inverse matrix includes:

[0053] S431: The input matrix is ​​decomposed using the Cholesky decomposition module. It can be decomposed into a lower triangular matrix L and its transpose, i.e. This module employs a specially designed pipeline structure to solve each element of L sequentially. Its computational unit includes a CORDIC-based square root operation module, a signed divider, a multiply-adder, and an accumulator.

[0054] S432: Subsequently, in the matrix inversion module, the lower triangular matrix L is inverted. This operation uses the forward substitution method to solve for the inverse matrix column by column. An array of solver units consisting of multipliers and adders was constructed, employing a row recursive structure and using FIFO to achieve parallel processing of multiple columns of data. This module outputs a matrix. ,because and Since it is a symmetric matrix, the output can be obtained by rounding. ;

[0055] S433: Finally, a static pulsating array was designed to implement the above matrix multiplication operation. The array uses a 5×5 multiply-accumulate unit, supports on-chip dual-channel read mechanism and result accumulation write-back function. Each element in the array is data scheduled by shared BRAM and managed by on-chip controller, thereby ensuring high-speed pipeline operation and automatic construction of matrix transpose structure.

[0056] Preferably, the matrix multiplication circuit structure is designed and calculated in S430. Finally, the values ​​of the ellipse parameters A, B, C, D, and E are obtained, including:

[0057] Using time multiplexing, the static pulsating array designed with the results of the inversion process is used for computation. Finally, the values ​​of the ellipse parameters A, B, C, D, and E are obtained.

[0058] Preferably, the relationship between the elliptic coefficients and the coordinates of their center point in S430 is as follows:

[0059] ;

[0060] The coordinates of the center point are obtained by sorting. , .

[0061] A sensor-computer integrated vision localization system for robot target localization includes:

[0062] The core board uses the Xilinx ZYNQ Ultrascale+ MPSoC FPGA ZU19EG as the core processor and integrates a power management module, a memory module, an Ethernet communication module, and peripheral interfaces.

[0063] The interface board is equipped with a microcontroller for power timing control and external interface management;

[0064] The sensor board, equipped with a CMOS sensor, provides image acquisition and communicates with the core board via an LVDS interface;

[0065] The ZU19EG integrates a programmable logic unit (PL) and a processing system unit (PS). On the PL, based on an FPGA architecture, the acquired image data is processed, target edges are extracted, and the data is transferred to the PS via the AXI4 bus in a zero-copy manner. On the PS, the edge coordinates are read, and an arc segment extraction algorithm is executed to obtain elliptical arc segments. The arc segment coordinates are written to the PL's BRAM buffer via DMA. The PL then uses the arc segment coordinates combined with a symmetric matrix-optimized pulsating array to perform ellipse fitting, thus completing target localization.

[0066] The aforementioned sensor-computing integrated visual positioning method and system for robot target localization integrates image sensing and edge computing functions into a single hardware device. Through collaborative processing on the sensor-computing integrated hardware platform, hardware acceleration of key algorithms, optimization of data transmission paths, and integrated system management, it achieves low latency, high precision, and high robustness in robot target localization, and also possesses the advantages of compact structure, low power consumption, high adaptability, and rapid deployment. Attached Figure Description

[0067] Figure 1 This is a flowchart of a sensor-computer integrated visual positioning method for robot target localization according to an embodiment of the present invention;

[0068] Figure 2 This is a schematic diagram of the hardware processor of a single camera in one embodiment of the present invention;

[0069] Figure 3 This is an estimation diagram of the gradient direction in one embodiment of the present invention. Detailed Implementation

[0070] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0071] In one embodiment, such as Figure 1 As shown, a sensor-computer integrated vision localization method for robot target localization includes the following steps:

[0072] S100: Acquires image data of the target scene through a sensor board equipped with a CMOS (Complementary Metal-Oxide-Semiconductor) sensor and sends it to the core processor;

[0073] S200: On the programmable logic (PL) side of the core processor, based on the FPGA (Field-Programmable Gate Array) architecture, the acquired image data is processed, the target edge is extracted from the processed image data, and the data is transmitted from the programmable logic (PL) side to the processing system (PS) side of the core processor in a zero-copy manner via the AXI4 bus.

[0074] S300: On the processor running on the processing system side PS of the core processor, the edge coordinate points are read and the arc segment extraction algorithm is executed to obtain the elliptical arc segment. The coordinates of the arc segment points are written to the BRAM (Block Random Access Memory) buffer on the PL side through DMA (Direct Memory Access).

[0075] S400: The PL end uses the arc point coordinates combined with the symmetric matrix optimized pulse array to achieve ellipse fitting and complete target localization;

[0076] S500: Controls the power timing of the system and manages external interfaces through the microcontroller unit (MCU) on the interface board;

[0077] S600: Through the processing system PS integrated in the core processor and peripheral interfaces, it performs overall system control and Ethernet communication.

[0078] Specifically, the hardware processor of a single camera, such as Figure 2As shown, the system consists of a core board and a sensor board. The core board uses a Xilinx ZYNQ Ultrascale+ MPSoC (Multi-Processor System on Chip) FPGA as its core processor, integrating a power management module, memory (including PS / PL DDR (Double Data Rate), QSPI (Quad Serial Peripheral Interface), eMMC (embedded MultiMediaCard) memory), an Ethernet (ETH) chip, and related peripheral interfaces. The interface board is equipped with an STM32F103RBT6 microcontroller for power timing control and management of external interfaces (ETH, UART (Universal Asynchronous Receiver / Transmitter), SD (Secure Digital), etc.). The CMOS board is equipped with a Sony IMX273LLR-C ​​CMOS sensor, which provides high-precision image acquisition and communicates with the core board through the LVDS (Low-Voltage Differential Signaling) interface.

[0079] This invention also proposes a heterogeneous acceleration framework for robot target localization, based on the Zynq UltraScale+ MPSoC heterogeneous chip, achieving algorithm-level hardware acceleration through the collaborative division of labor between the programmable logic unit (PL) and the processing system unit (PS). This framework decouples the target localization algorithm into three parts: image edge extraction employs an improved edge-preserving Canny (Canny Edge Detector) operator, achieving low-latency processing through the FPGA's parallel convolution engine; arc detection first uses the RDP (Ramer-Douglas-Peucker algorithm) to approximate polygon edges, then extracts arc segments by segmenting the edges using curvature and convexity; this part is deployed on an ARM (Advanced RISC Machines) Cortex-A53 processor for recursive optimization; ellipse fitting is accelerated through a custom least-squares matrix operation array on the FPGA, supporting real-time matrix equation solving. The PL end integrates a high-throughput CMOS image acquisition module and dense convolution and matrix calculation units, while the PS end relies on multi-level cache and branch prediction units to efficiently handle irregular data access. The two achieve zero-copy data interaction through the AXI high-speed bus, constructing an end-to-end low-latency processing link from image acquisition to target localization.

[0080] In one embodiment, CMOS image data flows into the PL terminal via a 4-channel LVDS protocol through the high-speed pins of the ZU19EG HP Bank (High Performance Bank). The S200 processes the acquired image data, including:

[0081] S210: A dynamically adjustable delay chain is constructed using the input delay unit IDELAY and the input delay control unit IDELAY_CTRL primitive to phase-shift the accompanying clock, so that the sampling clock edge is aligned with the midpoint of the data.

[0082] S220: A 1:10 deserialization operation is achieved through a deserializer, and a Bitslip (BitSlip) control unit is cascaded after the deserializer to match SONY's unique synchronization header through cyclic shifting; channel alignment of 4 data channels is completed through register delay.

[0083] The S230 DDR4 (Double Data Rate 4) storage control module uses the Xilinx MIG (Memory Interface Generator) IP (Intellectual Property core) to achieve efficient caching of image data. For the 12-bit pixel Bayer format data output from the front-end image acquisition, a 64-bit data packet is formed by filling 4-bit check codes every 5-pixel cycle (12bit × 5 = 60bit). This data is then written to the DDR4 storage control module via the AXI4 bus burst transfer mode (Burst Length = 8). At the same time, an address mapping controller is designed to store unaligned pixel data continuously across banks (Memory Banks) in row-major order to reduce the probability of access conflicts.

[0084] In one embodiment, step S200 involves extracting the target edge from the processed image, including:

[0085] S240: Image edge-preserving filtering is performed using a median filter: Bayer data stored in DDR4 is read, and after de-mosaicing and grayscale conversion, a 3×3 capture window is formed using a row buffer and a shift register. Unaligned pixel data flows into the row buffer continuously according to the clock cycle, and median filtering is performed on all pixels of a frame of image; among them, for parallel pipelined processing, a three-level comparison tree is used to obtain the median;

[0086] S250: The image gradient is calculated using the Sobel (Sobel Operator) operators Gx and Gy: For the median-filtered image, a 3×3 capture window is constructed to acquire data and perform multiplication and addition operations on the data in the window; the initial weights of the Sobel operator are stored in a register, the multiplication is implemented using a signed booth multiplier, and the accumulation is implemented using a parallel addition tree;

[0087] S260: To accurately locate edge points, non-maximum suppression is used to refine the gradient map: Two rows of pixel data are buffered and stored using a FIFO (First In, First Out) queue. Together with the current row of data, a 3×3 window structure is constructed using a D flip-flop delay. The magnitudes of neighboring pixels along the gradient direction of the center pixel are compared. If the magnitude of the center pixel is less than the magnitude of its neighboring pixels along the gradient direction, the pixel is judged as a non-edge point and suppressed. The gradient magnitude is approximated by the sum of the absolute values ​​of Gx and Gy, replacing the Euclidean norm. The gradient direction is simplified and estimated based on the signs and absolute values ​​of Gx and Gy. Figure 3 As shown; where Gx is the gradient component of the image in the horizontal direction, and Gy is the gradient component of the image in the vertical direction.

[0088] S270: The current pixel gradient magnitude is compared with the preset high threshold and low threshold by two parallel comparators, and the results are marked as strong edge, weak edge or non-edge respectively; a 3×3 neighborhood window is constructed by using a dual-row buffer and register array, and the edge connection module is controlled by a finite state machine to perform parallel scanning of the 8 neighborhoods of each weak edge pixel. If there is a strong edge point in the neighborhood, the current weak edge pixel is remarked as a strong edge point, otherwise it is discarded, and finally a coherent and complete binary edge image is formed, which is then transferred to the PS memory by DMA zero copy through the AXI4 bus.

[0089] In one embodiment, S300 includes:

[0090] S310: The arc segment extraction algorithm involves complex iterative calculations such as polygon approximation and curve segment fitting, which are difficult to implement efficiently using FPGA logic resources. Therefore, this algorithm is executed by the ARM processor on the PS side. To achieve efficient data transmission between the PL and PS sides, a DMA module based on the AXI4-FULL protocol is designed. The edge coordinates cached on the PL side are written to the PS DDR memory, and the PS side reads directly through the DMA mapping area, achieving high-speed, low-latency data transfer. At the same time, a dual-port RAM (Random Access Memory) and interrupt control logic are configured on the PL side. When the data is ready, an interrupt is triggered to notify the PS side, ensuring that the algorithm execution and data transmission are synchronized, thereby balancing system processing efficiency and resource utilization while ensuring algorithm accuracy.

[0091] S320: The edge polygon approximation is achieved through the RDP algorithm. In the specific implementation, the edge coordinate points read from the PS end, i.e., the obtained continuous pixel sequence, are represented as a point set. , the starting point and the finish line Connect the points to form a straight line and calculate the maximum perpendicular distance between this line segment and all intermediate points. The formula for the distance from a point to a line segment is:

[0092] ;

[0093] like If the error exceeds the error threshold ε, then the point is used as the new split point, and the point set is divided into two subsets for recursive processing. Otherwise, the point set is considered to be divisible by a straight line. Approximate representation: The ARM processor uses recursion to implement this logic, combined with a dynamic threshold strategy to adaptively adjust the accuracy, and finally outputs a sequence of polygon vertex coordinates to complete the polygon approximation of the edge, which is then used for subsequent arc fitting.

[0094] S330: An algorithm based on curvature and convexity constraints is used to divide the contour segments of the polygon approximation to obtain elliptical arc segments.

[0095] In one embodiment, S330 specifically refers to:

[0096] Let the approximated contour segment be denoted as the point set. And define the vector between adjacent vertices as ; any adjacent vector and The included angle between them is expressed as ,in Indicates from Rotate to The angle formed Indicates the index of the vertex sequence. This represents the total number of vertices contained in the approximated contour, and j represents the index of the included angle;

[0097] To ensure that the contour segments form a continuous, smooth, and monotonous arc structure, the following judgment criterion is designed: within a continuous sub-segment, the signs of all angles must remain consistent, i.e., the convexity condition must be satisfied:

[0098] ;

[0099] Meanwhile, to exclude abrupt curvature or acute-angle broken line segments, curvature thresholds and scale constraints are set:

[0100] ;

[0101] in, The maximum allowable angle change. These are scale constraint parameters used to prevent abrupt changes in line segment length. and Represent two adjacent vectors respectively and Length;

[0102] Extract continuous fields that meet the above conditions as elliptical arc segments.

[0103] Specifically, by using the formulas described above, continuous arc segments in the contour can be effectively extracted while satisfying local geometric consistency. This greatly simplifies the complex process of traditional gradient change or boundary feature point analysis and provides high-quality initial contour segments for subsequent ellipse parameter fitting.

[0104] In one embodiment, S400 includes:

[0105] S410: The general equation of an ellipse at any position in a plane is expressed as:

[0106] ;

[0107] in, Let A be the coordinates of any point in the image plane coordinate system, and let A, B, C, D, and E be five coefficients describing the shape, position, and orientation of the ellipse. A controls the rotation of the ellipse, B determines the eccentricity and scaling of the ellipse, and C and D control the center of the ellipse at... and Translation of direction For constant terms;

[0108] S420: Based on the least squares principle, the ellipse fitting approximates N data points in a plane. For these N data points, an objective function is established to minimize the sum of squared residuals, specifically:

[0109] ;

[0110] in, For the sum of squared residuals, Here, N is the index of the data point, and N is the total number of data points used for ellipse fitting.

[0111] make The normal equation can be obtained as follows:

[0112] ;

[0113] Further expressed as:

[0114] ;

[0115] Further results were obtained:

[0116] ;

[0117] in, Indicates left division;

[0118] S430: Designing a matrix inversion circuit structure to obtain... The inverse matrix is ​​used to design a matrix multiplication circuit structure for calculation. Finally, the values ​​of the five parameters A, B, C, D, and E that define the ellipse are calculated. Based on the relationship expression between the coefficients of the elliptic curve equation and the coordinates of its center point, the coordinates of the center point of the ellipse are obtained, thus achieving ellipse fitting and completing target localization.

[0119] In one embodiment, the matrix inversion circuit structure designed in S430 is obtained. The inverse matrix includes:

[0120] S431: The input matrix is ​​decomposed using the Cholesky decomposition module. It can be decomposed into a lower triangular matrix L and its transpose, i.e. This module employs a specially designed pipeline structure to solve each element of L sequentially. Its computational unit includes a CORDIC-based square root operation module, a signed divider, a multiply-adder, and an accumulator.

[0121] S432: Subsequently, in the matrix inversion module, the lower triangular matrix L is inverted. This operation uses the forward substitution method to solve for the inverse matrix column by column. ( (Assuming the data is a unit vector), a solver array consisting of multipliers and adders is constructed. A row recursive structure is used, and parallel processing of multiple columns of data is achieved through FIFO. This module outputs a matrix. ,because and Since it is a symmetric matrix, the output can be obtained by rounding. ;

[0122] S433: Finally, a static pulsating array was designed to implement the above matrix multiplication operation. The array uses a 5×5 multiply-accumulate unit, supports on-chip dual-channel read mechanism and result accumulation write-back function. Each element in the array is data scheduled by shared BRAM and managed by on-chip controller, thereby ensuring high-speed pipeline operation and automatic construction of matrix transpose structure.

[0123] In one embodiment, the matrix multiplication circuit structure is designed and calculated in S430. Finally, the values ​​of the ellipse parameters A, B, C, D, and E are obtained, including:

[0124] Using time multiplexing, the static pulsating array designed with the results of the inversion process is used for computation. Finally, the values ​​of the ellipse parameters A, B, C, D, and E are obtained.

[0125] In one embodiment, the relationship between the elliptic coefficients and the coordinates of their center point in S430 is specifically expressed as follows:

[0126] ;

[0127] The coordinates of the center point are obtained by sorting. , .

[0128] A sensor-computer integrated vision localization system for robot target localization includes:

[0129] The core board uses the Xilinx ZYNQ Ultrascale+ MPSoC FPGA ZU19EG as the core processor and integrates a power management module, a memory module, an Ethernet communication module, and peripheral interfaces.

[0130] The interface board is equipped with a microcontroller for power timing control and external interface management;

[0131] The sensor board, equipped with a CMOS sensor, provides image acquisition and communicates with the core board via an LVDS interface;

[0132] The ZU19EG integrates a programmable logic unit (PL) and a processing system unit (PS). On the PL, based on an FPGA architecture, the acquired image data is processed, target edges are extracted, and the data is transferred to the PS via the AXI4 bus in a zero-copy manner. On the PS, the edge coordinates are read, and an arc segment extraction algorithm is executed to obtain elliptical arc segments. The arc segment coordinates are written to the PL's BRAM buffer via DMA. The PL then uses the arc segment coordinates combined with a symmetric matrix-optimized pulsating array to perform ellipse fitting, thus completing target localization.

[0133] Specific limitations regarding the sensor-computer integrated vision positioning system for robot target localization can be found in the limitations of the sensor-computer integrated vision positioning method for robot target localization described above, and will not be repeated here. Each module in the aforementioned sensor-computer integrated vision positioning system for robot target localization can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0134] The aforementioned integrated visual localization method and system for robot target localization integrates image sensing and edge computing functions into a single hardware component. Image acquisition, preprocessing, feature extraction, and preliminary localization calculations are performed at the camera end, significantly reducing system latency and improving detection accuracy and robustness. The system has a compact structure, making it suitable for integration into mobile platforms under complex working conditions. It offers advantages such as rapid deployment, low power consumption, and high adaptability, providing an efficient and reliable solution for robot target recognition and localization in complex environments.

[0135] The foregoing has provided a detailed description of a sensor-computer integrated visual positioning method and system for robot target localization. Specific examples have been used to illustrate the principles and implementation methods of the invention. The descriptions of these embodiments are merely for the purpose of helping to understand the core ideas of the invention. It should be noted that those skilled in the art can make various improvements and modifications to the invention without departing from its principles, and these improvements and modifications also fall within the scope of protection of the claims of this invention.

Claims

1. A method for robot-target positioning, characterized in that, The method comprises the following steps: S100: collecting image data of a target scene by a sensor board carrying a CMOS sensor and sending to a core processor; S200: processing the collected image data based on an FPGA architecture at a programmable logic end PL of the core processor, extracting a target edge from the processed image data, and transmitting from the programmable logic end PL to a processing system end PS of the core processor in a zero-copy manner through an AXI4 bus; the processing of the collected image data in S200 comprises: S210: using an input delay unit IDELAY and an input delay control unit IDELAY_CTRL primitive to construct a dynamically adjustable delay chain to phase-shift a clock, so that a sampling clock edge is aligned with a data midpoint; S220: implementing a 1:10 deserializing operation on the data through a deserializer, and cascading a Bitslip control unit after the deserializer to match a synchronization header through cyclic shift; completing channel alignment of 4-way data through register delay; S230: filling 4-bit check codes into 64-bit data packets according to a 5-pixel period of the 12-bit pixel Bayer format data output by the front-end image collection, writing the data packets into a DDR4 storage control module through an AXI4 bus burst transmission mode, and designing an address mapping controller to store the misaligned pixel data continuously across memory banks in a row priority order; S200: extracting a target edge from the processed image, comprising: S240: reading the stored Bayer data from the DDR4, performing demosaicing and grayscale conversion, using a row buffer and a shift register to form a 3x3 capture window, and continuously flowing misaligned pixel data into the row buffer according to a clock period to perform median filtering on all pixels of an image; wherein, for parallel pipelining processing, a three-level comparison tree is used to obtain the median value; S250: for the image after median filtering, a 3x3 capture window is constructed to obtain data, and multiplication and addition operations are performed on the data in the window; wherein, the initial weights of the Sobel operator are stored in a register, signed booth multipliers are used for multiplication, and parallel addition trees are used for accumulation; S260: two rows of pixel data are stored through a FIFO buffer, together with the current row data to construct a 3x3 window structure through D flip-flop delay, the amplitude of the neighborhood pixels in the gradient direction of the center pixel point is compared, and if the amplitude of the center pixel point is smaller than that of the neighboring pixels in the gradient direction, the pixel point is judged as a non-edge point and suppressed; wherein, the gradient amplitude is approximated by the sum of the absolute values of Gx and Gy instead of the Euclidean norm, and the gradient direction is estimated according to the sign and absolute value size relationship of Gx and Gy; wherein, Gx is the gradient component of the image in the horizontal direction, and Gy is the gradient component of the image in the vertical direction. S270: Comparing the current pixel gradient amplitude with the preset high threshold and low threshold through two parallel comparators, the results are marked as strong edge, weak edge or non-edge respectively; a 3x3 neighborhood window is constructed by using double row buffer and register array, and a finite state machine is used to control the edge connection module to perform parallel scanning on the 8-neighborhood of each weak edge pixel, if there is a determined strong edge point in the neighborhood, the current weak edge pixel is re-labeled as a strong edge point, otherwise discarded, finally forming a complete edge binary image, and the image is transmitted to the PS end memory through AXI4 bus DMA zero copy; S300: In the processing system end PS of the core processor, read the edge coordinate points, and execute the arc segment extraction algorithm to obtain the elliptical arc segment, and write the arc segment point coordinates into the PL end BRAM memory buffer through direct memory access DMA; S400: The PL end realizes elliptical fitting based on the pulsating array optimized by the symmetry matrix, and completes target positioning; S500: Through the single-chip microcomputer on the interface board, the power sequence of the system is controlled and the external interface is managed; S600: Through the processing system end PS integrated with the core processor and the peripheral interface, the overall system control and Ethernet communication are performed.

2. The method of claim 1, wherein, S300 includes: S310: A DMA module based on AXI4-FULL protocol is designed to write the edge coordinate points cached in the PL end into the PS DDR memory, and the PS end directly reads through the DMA mapping area; at the same time, a dual-port RAM and an interrupt control logic are configured in the PL end, and when the data is ready, an interrupt is triggered to notify the PS end; S320: Represent the edge coordinate points read by the PS end as a point set , the starting point and the ending point are connected into a straight line segment, and the maximum vertical distance between the line segment and all intermediate points is calculated , where the distance formula from a point to a line segment is: ； If If the distance between the two points is greater than the error threshold ε, the point is taken as a new split point, and the point set is divided into two subsets, which are processed recursively. Otherwise, the point set is approximated by a straight line. The ARM processor uses recursive implementation to realize this logic, combined with a dynamic threshold strategy to adaptively adjust the precision, and finally outputs the polygon vertex coordinate sequence to complete the polygon approximation of the edge. S330: An algorithm based on curvature and convexity constraints is used to divide the polygon-approximated contour segment to obtain the elliptical arc segment.

3. The method of claim 2, wherein, S330 is specifically: Let the approximated contour segment be denoted as point set , and define the vector between adjacent vertices as ; the angle between any two adjacent vectors and is denoted as , where represents the angle formed by rotating to , represents the index of the vertex sequence, represents the total number of vertices contained in the approximated contour, and j represents the index of the angle. To ensure that the contour segment constitutes a continuous, smooth and monotonous arc structure, the following judgment criteria are designed: in a continuous sub-segment, the signs of all angles need to be consistent, that is, the convexity condition is met: ； At the same time, in order to exclude sudden curvature or sharp angle polyline segment, a curvature threshold and a scale constraint condition are set: ； wherein, is the allowed maximum angular variation, is a scale constraint parameter used to prevent abrupt changes in line segment length, and denote the length of two adjacent vectors and respectively. The continuous field meeting the above conditions is extracted as an elliptical arc segment.

4. The method of claim 3, wherein, S400 includes: S410: The general equation of an ellipse at any position in a plane is expressed as: ； wherein, A, B, C, D, E are five coefficients to be solved to describe the shape, position and direction of the ellipse, A controls the rotation of the ellipse, B determines the eccentricity and scaling of the ellipse, C and D control the translation of the center of the ellipse in and direction respectively, is a constant term; S420: According to the least square principle, the elliptical fitting is approximated as N data points in a plane, and a residual sum of squares minimum objective function is established for N data points, which is specifically: ； wherein, is the sum of squared residuals, is the index of the data point, and N is the total number of data points used for the ellipse fit. Let The normal equation is obtained: ； Further expressed as: ； Further obtained: ； wherein denotes left division; S430: obtaining the inverse matrix of the design matrix , calculating the product of the design matrix , finally calculating the values of the five parameters A, B, C, D, and E defining the ellipse, obtaining the center point coordinates of the ellipse based on the relationship expression between the coefficients of the elliptic curve equation and the center point coordinates, realizing ellipse fitting, and completing target positioning.

5. The method of claim 4, wherein, The matrix inversion circuit structure was designed in S430 to obtain The inverse matrix includes: S431: decompose the input matrix A into a lower triangular matrix L and its transpose by a Cholesky decomposition module , i.e. ; the module adopts a specially designed pipeline structure to sequentially solve each element of L, and its calculation unit contains a square root operation module based on CORDIC, a signed divider, a multiplier-adder, and an accumulator; S432: Then, in the matrix inversion module, the lower triangular matrix L is inverted, which is solved column by column using the forward substitution method , a solution unit array composed of multipliers and adders is constructed, a row recursive structure is adopted, and parallel processing of multi-column data is realized through FIFO, and the matrix , since and is a symmetric matrix, reshaping the output can obtain ; S433: Finally, a design of a static systolic array is presented to implement the matrix multiplication operation The array adopts 5x5 structure of multiply-add units, supports in-chip double-channel read mechanism and result accumulation write-back function, each element in the array is scheduled by shared BRAM and managed by on-chip controller, so as to guarantee high-speed pipeline operation and automatic construction of matrix transpose structure.

6. The method of claim 5, wherein, In S430, the matrix multiplication circuit structure is designed to calculate Finally, the values of the elliptic parameters A, B, C, D, and E are obtained, including: A systolic array computation designed using the results of the inversion process in a time-multiplexed fashion The values of the elliptic parameters A, B, C, D, E are finally obtained.

7. The method of claim 6, wherein, The expression of the relationship between the elliptical coefficients and the center point coordinates in S430 is specifically: ； The center point coordinates are obtained , .

8. A sensorimotor integrated visual positioning system for robot target positioning, which performs the method according to any one of claims 1 to 7, characterized by It includes: The core board uses Xilinx ZYNQ Ultrascale+ MPSoC FPGA ZU19EG as the core processor, and integrates a power management module, a memory module, an Ethernet communication module and a peripheral interface; The interface board is equipped with a single-chip microcomputer for power sequence control and external interface management; The sensor board is equipped with a CMOS sensor to provide image acquisition and communicate with the core board through an LVDS interface; Wherein, ZU19EG internal integration programmable logic end PL and processing system end PS, in the programmable logic end PL of the core processor, based on FPGA architecture, the collected image data is processed, the target edge of the processed image data is extracted, and the AXI4 bus is transmitted to the processing system end PS of the core processor in the zero copy mode; on the processor running in the processing system end PS of the core processor, the edge coordinate points are read, and the arc segment extraction algorithm is executed to obtain the elliptical arc segment, and the arc segment point coordinates are written into the PL end BRAM buffer area through DMA; The PL end realizes elliptical fitting according to the arc segment point coordinates combined with the pulsating array optimized by the symmetric matrix, and completes the target positioning.

Images