An unmanned agricultural machine all-around view panoramic system accelerated by CUDA under a WSL environment and an implementation method thereof

Abstract

This invention pertains to the fields of unmanned agricultural machinery and computer vision, and discloses a CUDA-accelerated panoramic surround-view system for unmanned agricultural machinery under a WSL environment and its implementation method. The method includes: acquiring calibration images from four fisheye cameras; after standardizing and calibrating the calibration images from the four fisheye cameras, storing the generated camera intrinsic parameter matrix and distortion coefficients in YAML format files respectively; acquiring the YAML format file, calculating the projection matrix by interactively selecting four marker points, and appending the projection matrix to the same YAML format file to form a complete camera parameter file; based on the complete camera parameter file, performing pixel geometric mapping calculations between the bird's-eye view and the original image, and generating a binary lookup table (LUT); synchronously receiving real-time operational images from the four fisheye cameras and the binary LUT file, performing distortion correction and panoramic stitching, and generating a panoramic bird's-eye view for panoramic visualization output.

Description

Technical Field

[0001] This invention belongs to the fields of unmanned agricultural machinery and computer vision, specifically relating to a CUDA-accelerated panoramic surround view system for unmanned agricultural machinery under WSL environment and its implementation method. Background Technology

[0002] With the advancement of smart agriculture, the widespread use of large agricultural machinery (such as tractors towing extra-long implements or agricultural drones) has become mainstream. However, while these behemoths bring great convenience to agricultural workers, they also create serious blind spots for drivers: not only is it difficult to observe the working status of the implements behind, but it is also impossible to perceive low obstacles or people around the vehicle, posing significant safety hazards and risks of operational losses.

[0003] The core value of unmanned panoramic surround view technology is to eliminate these blind spots, allowing drivers to remotely control machinery to complete operations. It uses multiple cameras (usually fisheye cameras) distributed around the front, rear, left, and right of the vehicle to collect video streams, and then uses image processing algorithms to synthesize a 360° bird's-eye view panoramic image of the surroundings of the agricultural machinery. This allows the driver to have a real-time view of the surrounding environment from inside the cab, effectively avoiding obstacles, crops, and people.

[0004] However, existing technical solutions have the following drawbacks: 1. Traditional solutions perform complex distortion correction and perspective transformation calculations during the online phase, involving a large number of floating-point operations, resulting in high processing latency and making it difficult to meet the millisecond-level response requirements of unmanned agricultural machinery.

[0005] 2. Embedded vehicle computing units have limited computing power. Real-time geometric transformation and fusion processing of multiple high-definition videos consumes a large amount of CPU / GPU resources, affecting system stability.

[0006] 3. Traditional solutions require rerunning complex global calibration algorithms after replacing cameras or adjusting installation positions, which is not conducive to rapid on-site deployment and maintenance.

[0007] 4. Most existing surround view systems are developed based on pure Linux embedded devices or dedicated industrial control computers, which requires front-line workers familiar with the agricultural field to additionally learn Linux system operation and environment configuration. The environment setup process is complex and the operation and maintenance are difficult, making it difficult to quickly implement in agricultural production scenarios.

[0008] 5. Real-time performance and accuracy are difficult to achieve simultaneously. Traditional solutions either sacrifice stitching quality to reduce latency or sacrifice real-time performance to improve quality, making it difficult to achieve both.

[0009] Searches revealed that existing surround-view panoramic technologies are primarily applied to passenger vehicles, and most of them improve image stitching speed by modifying online algorithms, use 28+ bytes / pixel floating-point data storage, provide only a single calibration process and fixed visualization layout, and are developed based on the Linux operating system. No unmanned agricultural machinery surround-view panoramic systems have been found that simultaneously possess several key technical features, including a dual-stage architecture of offline pre-computation + online CUDA acceleration, a 12-byte / pixel compact LUT (Look-Up Table, the core data structure connecting offline pre-computation and online real-time stitching), a dual-mode calibration strategy for agricultural machinery scenarios, a dual-mode visualization layout, and native compatibility with the WSL environment.

[0010] There are two main technical approaches to implementing LUTs in current surround-view technologies: The first is the access strategy optimization approach, which prefetches large LUT blocks into on-chip memory to reduce access overhead. While this method improves access efficiency, the LUT itself remains large, with a single pixel storage space exceeding 28 bytes, resulting in continued pressure on video memory bandwidth. The second approach is the compression algorithm approach, which uses techniques such as decomposition, self-similarity, and multi-level compression to perform lossless compression of the LUT, achieving a compression rate of up to 60%. However, this method requires decoding circuitry and additional latency, making it unsuitable for agricultural machinery surround-view scenarios with stringent real-time requirements.

[0011] In addition, existing GPU acceleration solutions for surround view systems mainly focus on parallelizing computing tasks, without addressing the timing overlap between data transmission and GPU computing. Performance fluctuations in the stitching process are the core issue hindering the practical application of current solutions.

[0012] To address the aforementioned technical bottlenecks, this invention proposes a systematic solution that involves collaborative design across five dimensions: architecture, data structure, calibration method, parallel computing, and development environment, resulting in a complete technical solution that differs from existing technologies. Summary of the Invention

[0013] To address the problems of insufficient real-time performance, high resource consumption, poor stitching quality, high calibration complexity, and complex operation and maintenance environment in existing technologies, this invention provides a CUDA-accelerated surround-view panoramic system for unmanned agricultural machinery under a WSL environment and its implementation method. Complex calculations are moved out of the real-time loop (offline pre-calculation), and the vehicle-mounted terminal retains only lightweight table lookups. A compact LUT data structure is designed to minimize storage bandwidth requirements. A dual-mode calibration strategy is adopted to balance accuracy and efficiency, achieving highly reliable, low-latency surround-view panoramic perception for unmanned agricultural machinery in complex field environments, allowing the driver to operate the machine independently.

[0014] To achieve the above objectives, the present invention provides the following solution: A CUDA-accelerated panoramic system for unmanned agricultural machinery in a WSL environment, the system comprising: a fisheye camera image acquisition module, a fisheye camera calibration module, a bird's-eye projection module, an offline mapping table generation module, a CUDA-accelerated image stitching module, and a panoramic visualization output module. The fisheye camera image acquisition module is used to transmit the acquired calibration images from the four fisheye cameras to the fisheye camera calibration module. The fisheye camera calibration module is used to standardize the calibration images of the four fisheye cameras and then store the generated camera intrinsic parameter matrix and distortion coefficients in YAML format files respectively. The bird's-eye view projection module is used to acquire a YAML format file, calculate the projection matrix by interactively selecting four marker points, and append the projection matrix to the same YAML format file to form a complete camera parameter file. Finally, the YAML format file containing the intrinsic parameter matrix, distortion coefficients and projection matrix is ​​transmitted to the offline mapping table generation module. The offline mapping table generation module is used to complete the pixel geometry mapping calculation between the bird's-eye view and the original image, and then transmit the generated binary lookup table (LUT) to the CUDA-accelerated image stitching module. The CUDA-accelerated image stitching module is used to simultaneously receive real-time working images and binary LUT files from four fisheye cameras, and after completing distortion correction and panoramic stitching, transmit the generated panoramic bird's-eye view to the panoramic visualization output module. The panoramic visualization output module is used to visualize the panoramic bird's-eye view for remote control by the driver.

[0015] Preferably, the fisheye camera calibration module uses a standard calibration grid of 12×10 inner corner points for agricultural machinery working distances of 0-4 meters, with a width of 20cm for each grid.

[0016] Preferably, the process of calculating the projection matrix by interactively selecting four marker points and appending the projection matrix to the same YAML format file to form a complete camera parameter file includes: Read the generated YAML file and run the run_get_projection_maps.py script to perform projection calibration in the four directions of front, back, left, and right. After the script displays the fisheye image, click the four markers in the order of top left, top right, bottom left, and bottom right. If you click the wrong marker, press the d key to delete it. The -scale xy parameter controls the horizontal and vertical stretching of the projection, with a value range of 0.3-0.6. The -shift ab parameter controls the horizontal and vertical translation distance of the projection, with a value range of -500 to 500 pixels. Based on the coordinates of the four selected marker points and the preset ideal coordinates, combined with the camera intrinsic parameter matrix and distortion coefficients read from the YAML format file, a 3×3 projection matrix is ​​calculated and appended to the corresponding YAML file to form a complete camera parameter file containing the intrinsic parameter matrix, distortion coefficients, and projection matrix.

[0017] Preferably, the generation process of the binary lookup table (LUT) includes: using the calibrated camera parameters, pre-calculating and packaging a binary lookup table containing geometric mapping relationships and fusion weights, specifically: The script 1.py reads the YAML parameter files generated by the fisheye camera calibration module and the bird's-eye projection module, and reads the 4-channel weight map weights.png, where the four channels correspond to the four overlapping regions of left rear, right front, left front, and right rear, respectively. Using the fisheye module in OpenCV and the grid_sample function in PyTorch, the coordinate mapping relationship of each pixel in the bird's-eye image to the original fisheye image is calculated, and the fusion weights are assigned to the overlapping regions according to the 4-channel weight map. All calculation results are packaged in a compact format of 12 bytes / pixel to generate a binary lookup table file surround_view.binary, which is the specific form of the binary lookup table LUT.

[0018] Preferably, the process of simultaneously receiving real-time images and binary LUT files from four fisheye cameras, completing distortion correction and panoramic stitching, and generating a panoramic bird's-eye view includes: The initialization function is called to load the binary lookup table (LUT) output by the offline mapping table generation module into the CPU memory and upload it to the GPU memory at the same time, while allocating the video memory space for the input and output images. After acquiring the operational images from the four fisheye cameras in real time, the image data is uploaded to the GPU global memory via cudaMemcpy. Asynchronous transmission and double buffering mechanisms are used to overlap the data transmission with the computation task. The CUDA kernel function is started, and a GPU thread is allocated for each pixel of the output bird's-eye view. Each thread performs the following operations: reads the primary camera coordinates (u1, v1), secondary camera coordinates (u2, v2), camera index (id1, id2), and fusion weight w from the lookup table; if the id1 field is valid, obtain the pixel value P1 from the corresponding camera image through bilinear interpolation; if the id2 field is valid, obtain the pixel value P2 from the secondary camera image through bilinear interpolation, and execute the weighted fusion formula P_out=(P1×w+P2×(255-w))>>8; if the id2 field is invalid, then P_out=P1; if the id1 field is invalid, the pixel is set to the background color; writes the fused pixel value to the corresponding position in the output image; and downloads the stitched bird's-eye view back to CPU memory for visualization output using cudaMemcpy.

[0019] The preferred asynchronous transmission mechanism: Through the cudaMemcpyAsync asynchronous transmission function and CUDA streaming technology, data transmission and GPU kernel function execution are parallelized. Specifically: a CUDA stream is created, cudaMemcpyAsync is called to asynchronously upload the image to the GPU memory, the function returns immediately without waiting for the transmission to complete, the kernel function is started in the same stream, the GPU hardware guarantees that the kernel function will execute automatically after the transmission is completed, and the CPU thread continues to acquire the next frame of image, realizing complete parallelism between data transmission and GPU computation, eliminating waiting time.

[0020] Preferably, a dual-buffering mechanism is used: two independent buffers are allocated in the GPU memory, namely Buffer A and Buffer B, which are used in a loop by switching pointers, so that while the image in the current buffer is being processed by the CUDA Kernel, the next frame image is being transmitted through the PCIe bus in the other buffer.

[0021] Preferably, the process of visualizing and outputting panoramic bird's-eye views includes: When the agricultural machinery is stationary, the turn signal is on, or the driver selects the "implementation attachment / field turn" mode, a "four-window + bird's-eye view" layout scheme is adopted, which simultaneously displays the distortion-free real-time images in the front, back, left, and right directions and the central bird's-eye view. When the agricultural machinery travels at a speed exceeding 5 km / h, without steering operation, or when the driver selects the "field travel / road travel" mode, a "front window + bird's-eye view + panoramic view" layout scheme is adopted to prioritize the forward visibility and overall environmental perception. To address blind spots in agricultural machinery operations, icons for agricultural machinery are overlaid at the bottom of the machinery in the bird's-eye view to indicate their location.

[0022] This invention also provides a method for implementing a CUDA-accelerated surround-view panoramic system for unmanned agricultural machinery in a WSL environment. The method is implemented using the aforementioned system and includes: Acquire calibration images from four fisheye cameras; After standardizing the calibration images of the four fisheye cameras, the generated camera intrinsic parameter matrix and distortion coefficients are stored separately in YAML format files. Obtain a YAML file, calculate the projection matrix by interactively selecting four marker points, and append the projection matrix to the same YAML file to form a complete camera parameter file. Based on the complete camera parameter file, after completing the pixel geometric mapping calculation between the bird's-eye view and the original image, a binary lookup table (LUT) is generated. It simultaneously receives real-time operational images and binary LUT files from four fisheye cameras, completes distortion correction and panoramic stitching, and generates a panoramic bird's-eye view for panoramic visualization output.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention employs a two-stage core architecture of offline pre-computation load reduction and online CUDA parallel acceleration. Based on the WSL environment, it achieves seamless compatibility with the native CUDA acceleration program of the Linux system under the Windows operating system. It fully covers the entire process of camera calibration, geometric mapping calculation, real-time image stitching, and panoramic visualization output. While reducing the threshold for system deployment and use, it achieves millisecond-level panoramic bird's-eye view stitching output.

[0024] This invention is not a simple combination of existing technologies, but rather a problem-driven systematic approach that constructs a two-stage architecture of offline pre-computation and online CUDA acceleration as the technical foundation. Around this foundation, it conducts collaborative design in five dimensions: data structure, calibration method, parallel computing mechanism, visualization layout, and development environment. This enables the various technical features to form a systemic relationship of mutual support, causal coupling, and positive feedback, resulting in technical advantages that exceed the sum of the individual effects of each feature. Attached Figure Description

[0025] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the system structure in an embodiment of the present invention; Figure 2 This is a schematic diagram of the core data interaction route of the panoramic system in this embodiment of the invention; Figure 3 This is a schematic diagram of the fisheye camera installation position in an embodiment of the present invention; Figure 4 This is a schematic diagram of the memory layout of the binary lookup table in an embodiment of the present invention; Figure 5 This is a schematic diagram of the dual-mode visual layout in an embodiment of the present invention; Figure 6 This is a schematic diagram of the dual-mode visualization layout of the global version and the simplified version in an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] Example 1 This invention discloses a CUDA-accelerated panoramic system for unmanned agricultural machinery in a WSL environment. Specifically, it involves a panoramic system for unmanned agricultural machinery accelerated by CUDA (Compute Unified Device Architecture) technology, based on the WSL (Windows Subsystem for Linux, a lightweight Linux native runtime environment developed by Microsoft and deeply integrated into the Windows operating system) environment. The system focuses on the specific scenarios of unmanned agricultural machinery field operations, and is specifically optimized for these scenarios. Unlike general-purpose panoramic systems or drone panoramic systems, it addresses the specific needs of static, precise operations such as 0-4 meter working distances, implement attachment, and field turning, as well as dynamic scenarios such as field travel and road travel. It provides a systematic solution in terms of calibration accuracy, real-time response, and visualization layout. Specifically: The technical solution of this panoramic system mainly consists of six modules: fisheye camera image acquisition module, fisheye camera calibration module, bird's-eye projection module, offline mapping table generation module, CUDA-accelerated image stitching module, and panoramic visualization output module. Its structure diagram is shown below. Figure 1 As shown.

[0030] The core data interaction route of the panoramic system is as follows: Figure 2As shown: The fisheye camera image acquisition module transmits the calibration images from the four fisheye cameras to the fisheye camera calibration module. After completing the standardization calibration, the fisheye camera calibration module stores the generated camera intrinsic parameter matrix and distortion coefficients in YAML format files (corresponding to the front, rear, left, and right cameras respectively). Subsequently, the bird's-eye view projection module reads these YAML format files, calculates the projection matrix by interactively selecting four marker points, and appends this matrix to the same YAML format file to form a complete camera parameter file. Finally, the YAML format file containing the intrinsic parameter matrix, distortion coefficients, and projection matrix is ​​transmitted to the offline mapping table generation module for calculating the geometric mapping relationship for panoramic stitching. After completing the pixel geometric mapping calculation between the bird's-eye view and the original image, the offline mapping table generation module transmits the generated binary lookup table (LUT) to the CUDA accelerated image stitching module. The CUDA accelerated image stitching module synchronously receives the real-time working images from the four fisheye cameras and the binary LUT file. After completing distortion correction and panoramic stitching, the generated panoramic bird's-eye view is transmitted to the panoramic visualization output module for remote control by the driver. The WSL environment provides a unified operating environment, CUDA acceleration dependencies, and adaptation support for image processing libraries and AI computing libraries for the above six modules, ensuring that all modules run stably on the Windows system. The fisheye camera calibration module and the offline mapping table generation module only need to run once when the camera installation position changes or the parameters change. The CUDA-accelerated image stitching module runs continuously on the vehicle end, achieving millisecond-level real-time stitching.

[0031] In this embodiment, the WSL environment is used to build a Linux development environment on the Windows operating system that is consistent with the target in-vehicle environment. The core task is to achieve seamless compatibility between the Windows host and the native Linux CUDA program through Windows Subsystem for Linux 2 technology, providing a development platform that is close to the in-vehicle deployment environment for algorithm prototype verification and performance optimization.

[0032] Setting up the WSL environment requires running a PowerShell process as administrator on the Windows host and executing the command `wsl --install` to install WSL2 and the default Linux distribution in one click. After installation, add the NVIDIA CUDA repository in the WSL terminal and install the CUDA Toolkit, along with the OpenCV development library and Python dependencies (PyTorch, NumPy, etc.). Once the environment is configured, verify its correctness by running `nvcc --version` and compiling the CUDA test program.

[0033] The WSL environment utilizes NVIDIA's official WSL CUDA support technology. Testing shows that, under normal circumstances, GPU computing efficiency is roughly equivalent to that of the native Linux environment, and at its worst, it reaches over 90% of the native Linux environment, significantly reducing the difference between the development environment and the final deployment environment (embedded Linux). Developers and operations personnel can use the Linux toolchain for code writing, debugging, and performance optimization within the familiar Windows interface. Once the algorithm is verified, the program can be directly ported to the native Linux system of the in-vehicle central control unit, significantly reducing the difficulty of operation.

[0034] In this embodiment, the fisheye camera image acquisition module is the basic module at the front end of the system, providing raw image data for all subsequent calibration and stitching processes.

[0035] This module installs a fisheye camera with a field of view of 180°~200° in each of the four directions (front, rear, left, and right) of the agricultural machinery to meet the requirements of panoramic coverage. The installation positions and angles of each camera are configured according to the size of the agricultural machinery and the operating scenario, as shown in Table 1. Table 1. Fisheye Camera Installation Requirements and Coverage Targets like Figure 3 As shown, the installation requirements ensure that each camera can completely cover its corresponding calibration area and that there is sufficient overlap between the fields of view of adjacent cameras to provide conditions for subsequent image stitching. The system uses PoE (Power over Ethernet) cameras (which transmit data signals and provide DC power simultaneously via a standard Ethernet cable) and connects to the vehicle's central control unit via a router to achieve real-time transmission of video data; the central control unit receives four video streams via network cable to ensure the real-time performance and stability of the acquisition.

[0036] In this embodiment, the fisheye camera calibration module is an offline calibration module located after the image acquisition module and before the bird's-eye view projection module. Its core task is to use the acquired checkerboard calibration image to solve for the intrinsic parameter matrix and distortion coefficient of each camera, providing basic parameters for subsequent image correction and projection transformation.

[0037] The fisheye camera calibration module uses a standard calibration checkerboard with 12×10 interior corner points (corresponding to 13×11 squares). Each checkerboard square is 20cm wide. This checkerboard size ensures the camera can see sufficient corner point distribution for close-range calibration (0-4 meters). This specification is specifically optimized for the single-camera coverage range (0-4 meters) of agricultural machinery bird's-eye view cameras, ensuring that corner points cover the center and edges of the image within the effective working distance, providing sufficient data support for subsequent distortion model fitting. The 12×10 design, totaling 120 interior corner points, provides a sufficient number of constraints while maintaining a moderate calibration board size, enabling nonlinear optimization algorithms to solve for camera parameters more accurately.

[0038] The fisheye camera calibration module uses the camera calibration tool in the OpenCV computer vision library to process multiple acquired checkerboard calibration images. The acquisition of calibration images follows this strategy: ensuring the checkerboard is covered at both the center and edges of the image, and encompassing multiple sets of viewpoints at different angles and distances to provide sufficient constraints for fitting the distortion model. Fisheye camera distortion is much greater at the edges than at the center. The center corner points are used to constrain camera intrinsic parameters, while the edge corner points provide necessary constraint information for higher-order distortion coefficients (k1, k2, k3, k4), thus accurately modeling the distortion variation from the center to the edges. Images from different angles help stabilize the solution of camera intrinsic parameters; images at different distances ensure the distortion model has universality within the camera's working range (0-4 meters), avoiding the model being accurate only at a single distance. This module acquires a sufficient number of calibration images (50-65 images), covering different positions and poses, to ensure the stability of the parameter solution.

[0039] Using OpenCV's built-in fisheye camera distortion model, corner detection and parameter optimization are performed on the images to solve for the intrinsic parameter matrix and distortion coefficients of each camera, i.e., the calibration results. Specifically, the `findChessboardCorners()` function (corner detection function) of OpenCV is first called to extract 12×10 internal corner points for each calibration image, and the `cornerSubPix()` function (subpixel optimization function) is used to improve the corner accuracy to the subpixel level. Then, the `fisheye::calibrate()` function, a dedicated calibration function for fisheye cameras, is called. Inputting 3D world coordinates (known coordinates of the calibration board corner points), 2D image corner point coordinates, and image size, a nonlinear optimization algorithm minimizes the reprojection error to solve for the camera's intrinsic parameter matrix and distortion coefficients (the OpenCV fisheye camera model uses the following formula to describe radial distortion: Where θ is the angle of incidence of the light ray, θ d(where k1-k4 are the four radial distortion coefficients, representing the distorted incident angle.) The solution results are stored in YAML format files (corresponding to the front, rear, left, and right cameras) for use by the subsequent offline mapping table generation module.

[0040] Existing fisheye camera calibration schemes do not optimize calibration board specifications and image acquisition strategies for close-range agricultural machinery operations. This invention, however, employs an isometric projection model from the OpenCV fisheye module and specifically designs a 12×10 inner corner point, 20cm grid calibration board for agricultural machinery operating at distances of 0-4 meters. Combined with a strategy of acquiring 50-65 multi-angle, multi-distance images, it significantly improves calibration accuracy and stability.

[0041] In this embodiment, the bird's-eye projection module is an independent intermediate module located after the fisheye camera calibration module and before the offline mapping table generation module. Its core task is to calculate the projection matrix that maps the fisheye image to the bird's-eye coordinate system, determine the projection area of ​​the bird's-eye image through interactive point selection, and append the calculation results to the projection matrix in the calibrated YAML format file.

[0042] The core script for this module is `run_get_projection_maps.py`, which needs to perform line projection calibration on the fisheye cameras in the front, back, left, and right directions respectively. Running `run_get_projection_maps.py` displays the fisheye image of the current camera. The user clicks on the four markers in sequence: top left, top right, bottom left, and bottom right. If an incorrect marker is selected, press the 'd' key to delete it. After selecting, press Enter to confirm. The script displays the projection effect in real time for preview and evaluation. By modifying four parameters—horizontal and vertical stretching, and horizontal and vertical translation distance—camera installation errors can be quickly compensated for on-site.

[0043] The script generates a 3×3 projection matrix based on the coordinates of the four selected marker points and the preset ideal coordinates, combined with the camera intrinsic parameter matrix and distortion coefficients read from a YAML file. The projection matrix H serves as the core transformation parameter connecting the fisheye image and the bird's-eye view; it is solved using a four-point interactive calibration method and calibrated quickly on-site via command-line parameters. Its mathematical structure is a 3×3 homography matrix with 8 degrees of freedom, controlling scaling, rotation, translation, and perspective distortion, with the bottom right element fixed at 1. The specific format is H= , where h 00 h 11 It is the scaling factor, which controls the horizontal and vertical scaling of the bird's-eye view; h 01 h 10 It is the rotation / shearing factor, which controls the rotation and tilt correction of the image; h 02 h 12It is the translation factor, which controls the horizontal and vertical offset of the bird's-eye view; h 20 h 21 This is the perspective factor, which controls the degree of perspective distortion and corrects trapezoids to rectangles. H is incorporated into the offline pre-calculation chain, eliminating the need for perspective transformation on the vehicle-mounted device (retaining only lightweight table lookup operations). This matrix is ​​appended to the corresponding YAML file, forming a complete camera parameter YAML file along with the intrinsic parameter matrix and distortion coefficients, for use by the subsequent offline mapping table generation module.

[0044] In existing technologies, the bird's-eye view projection module is directly used for online stitching after completing the projection matrix calculation. Each frame requires a warpPerspective operation, resulting in a high computational load. Furthermore, adjusting projection parameters requires modifying code or configuration files, leading to low deployment efficiency. This invention, however, incorporates the bird's-eye view projection module into an offline pre-calculation chain. Combined with a real-time command-line parameter adjustment mechanism and a compact LUT data structure, it achieves lightweight table lookup and interpolation on the vehicle-mounted end, significantly reducing real-time computational load and storage bandwidth requirements.

[0045] Existing solutions require code or configuration file modifications and recompilation to adjust the field of view (shift parameter) or stretching (scale parameter) of a bird's-eye view, resulting in low deployment efficiency. This invention, using the -scale xy and -shift ab command-line parameters, allows for real-time adjustment of the projection effect without code modification. Furthermore, it clearly provides recommended parameter ranges (stretching 0.3-0.6, translation distance -500~500 pixels), offering actionable guidance for rapid on-site calibration.

[0046] In this embodiment, the offline mapping table generation module is an offline preprocessing module located after the bird's-eye view projection module and before the CUDA-accelerated image stitching module. Its core task is to pre-calculate and package a binary lookup table (LUT) containing geometric mapping relationships and fusion weights using the calibrated camera parameters for direct use in the online real-time processing stage. The data structure of the binary lookup table (LUT) is shown in Table 2, and the memory layout (the way data is stored and arranged in memory) is as follows: Figure 4 As shown, this structure packages dual-camera coordinates, fusion weights, and camera indices into 12 bytes per pixel, reducing storage space by 58% compared to the traditional floating-point scheme (28 bytes per pixel), and eliminating the need for decoding overhead.

[0047] Table 2. Data structure of each pixel in a binary lookup table (LUT) file. Specifically: The binary lookup table (LUT) is stored in a compact format of 12 bytes per pixel, and the mapping data for each pixel contains the following fields: 1. Main camera coordinates (u1, v1): Occupies 4 bytes, stored as a short type, with a value range of 0-4000, representing the X and Y coordinates of the output pixel in the main camera image. When this field is valid, the system obtains the pixel value from the main camera image.

[0048] 2. Secondary camera coordinates (u2, v2): Occupies 4 bytes, stored as a short type, with a value range of 0-4000, representing the X and Y coordinates of the output pixel in the secondary camera image. This field is only valid in overlapping areas and is used for fusion calculations.

[0049] 3. Fusion Weight (w): Occupies 1 byte, stored as a uchar type, with a value range of 0-255, representing the weight of the primary camera image pixels in the fusion result. The weight of the secondary camera image is 255 minus this value, achieving weighted fusion.

[0050] 4. Camera Index (id1, id2): Occupies 2 bytes, stored as a byte type, and identifies the main camera and secondary camera numbers respectively (0-3 correspond to the front, rear, left, and right cameras respectively). When the index value is -1, it indicates that there is no valid camera image in that direction.

[0051] 5. Byte alignment padding: Occupies 1 byte and is used to ensure that data structures are aligned to 4-byte boundaries, improving memory access efficiency.

[0052] Each pixel is a total of 12 bytes. Calculated at an output resolution of 1200×1600, the total size of the LUT file is approximately 23MB, which can be loaded into the GPU memory at once for real-time stitching.

[0053] The script corresponding to this module is 1.py. This script reads the YAML parameter file (containing intrinsic parameter matrix, distortion coefficients, and projection matrix) generated by the fisheye camera calibration module and the bird's-eye projection module. Using the fisheye module in OpenCV and the grid_sample function in PyTorch, it calculates the coordinate mapping relationship between each pixel of the bird's-eye image and the original fisheye image, and assigns fusion weights (0-255) to the overlapping areas based on the 4-channel weight map. All calculation results are packaged in a compact 12-byte / pixel format to generate a binary lookup table file surrounding_view.binary (the specific form of the binary lookup table LUT), for subsequent use by the CUDA-accelerated image stitching module. Specifically: 1. Mathematical expression of mapping relationship calculation: Assuming the output bird's-eye view resolution is W×H, for any pixel coordinate (u, v), the corresponding original fisheye image coordinates are calculated using a composite mapping function. The mathematical expression is as follows: .in: Define the camera's intrinsic parameter matrix, specifying the camera's focal length and principal point position. ={k1, k2, k3, k4}, which are the distortion coefficients of the fisheye camera; It is a 3×3 bird's-eye view projection matrix.

[0054] 2. Offline calculation of fusion weights: For overlapping areas of adjacent cameras, the fusion weights are calculated using a 4-channel weight map. distribute: The four channels correspond to the four overlapping regions: left front, right front, left rear, and right rear; weight values ∈[0, 255] linear gradient based on the distance from the pixel to the boundary of the overlapping region: This design allows the pixel values ​​of the overlapping area to be weighted and synthesized from the images of the two adjacent cameras, eliminating the stitching seams.

[0055] This module adopts an offline pre-computation architecture (which means that when the system is deployed or the camera parameters change, all complex geometric mapping calculations are performed in advance, and the calculation results are packaged into a compact binary lookup table (LUT) for direct lookup in the online real-time stitching stage. This is the sum of the fisheye camera calibration module, the bird's-eye projection module, and the offline mapping table generation module). When the camera parameters change, only the 1.py script needs to be re-run to update the LUT file (surround_view.binary), without modifying or recompiling the online CUDA program. By moving complex geometric calculations out of the real-time loop, the vehicle-mounted end only performs lightweight table lookup and interpolation operations. The measured single-frame processing latency is as low as 4.23ms. While ensuring real-time stitching performance, it significantly reduces the computing power requirements of the vehicle-mounted embedded platform and improves the maintainability and on-site deployment flexibility of the system.

[0056] The binary lookup table (LUT) design of this invention differs from the two main paths in the prior art: 1. Differentiating from access strategy optimization paths: Existing technologies often prefetch large LUTs into blocks to on-chip memory to reduce access overhead, but the LUT itself is still relatively large (28+ bytes / pixel). This invention starts from the source compression path, directly designing a compact LUT through accuracy requirement analysis, making the LUT itself smaller, and can be used in conjunction with block prefetching strategies.

[0057] 2. Differentiating from compression algorithms: Existing technologies often employ decomposition, self-similarity, and multi-level compression techniques to achieve lossless compression of LUTs, reaching a compression rate of 60%, but requiring decoding circuits and additional latency. This invention adopts a source design approach, precisely matching short / uchar / byte data types by analyzing the coordinate range (0-4000 pixels) and weight precision (0-255 levels) of the agricultural machinery surround view scene, reducing storage space by 58% without decompression.

[0058] In this embodiment, the CUDA-accelerated image stitching module is an online real-time processing core module located after the offline mapping table generation module and before the panoramic visualization output module. Its core task is to utilize the parallel computing power of the GPU to read the binary lookup table (LUT) to perform distortion correction and panoramic stitching on the real-time images from the four fisheye cameras, generating a visualization output. The corresponding program for this module is CUDA (an executable file compiled from cuda.cu), and it is a core component of the system's online phase.

[0059] When this module starts, it calls the initialization function (an initialization function is a one-time configuration function executed when the CUDA-accelerated image stitching module starts, used to establish the GPU computing environment and allocate all necessary resources, preparing for the subsequent real-time stitching pipeline). This function loads the binary lookup table (LUT) output by the offline mapping table generation module into CPU memory and uploads it to GPU memory in one go. Simultaneously, it allocates GPU memory space for the input and output images (assuming an input resolution of W). in ×H in Output resolution W out ×H out If the number of color channels C=3 and the number of cameras N=4, then: the total system video memory usage M total For M total = M input + M output +M lut +M overhead The input buffer occupies space of M input =2×W in ×H in ×C×N, the output buffer occupies space of... M output =2×W out ×H out ×C, the space occupied by the lookup table LUT is M lut =W out ×H out ×12, CUDA runtime overhead is M overhead(Approximately tens of MB). After acquiring images from four fisheye cameras in real time, the image data is uploaded to the GPU's global memory via cudaMemcpy. Asynchronous transmission and double buffering mechanisms are used to overlap data transmission with computation tasks, thereby saving processing time per frame. After starting the CUDA kernel function, a GPU thread is allocated for each pixel of the output image. Each thread performs the following operations: reads the primary and secondary camera coordinates, camera index, and fusion weights from the binary lookup table (LUT); obtains the pixel value through bilinear interpolation; executes the weighted fusion formula to calculate the pixels in the overlapping area; and writes the fused pixel value to the corresponding position in the output image. The stitched bird's-eye view is finally downloaded back to CPU memory via cudaMemcpy for use by the panoramic visualization output module.

[0060] Existing research on GPU acceleration for surround-view systems mainly focuses on parallelizing computational tasks, such as GPU acceleration for pixel-level operations like distortion correction, perspective transformation, and image fusion. However, it fails to address the timing overlap between data transmission and GPU computation. Performance fluctuations in the stitching stage are a core issue hindering the practical application of current solutions, with latency drastically ranging from 16ms to 100ms.

[0061] Existing CUDA acceleration solutions mainly adopt a single-stage online processing architecture, placing complex calculations such as feature extraction, registration, and fusion in the online stage. They do not use compact LUT data structures, or only use them as simple mapping tables. Transmission and calculation are serial, resulting in performance fluctuations in the splicing stage (16-100ms). The online calculation of the fusion algorithm increases latency overhead.

[0062] This invention is the first to systematically apply a dual-stage architecture of offline pre-computation and online CUDA acceleration in a surround-view system. It reduces storage space by 58% through a compact 12-byte / pixel LUT (storing dual-camera coordinates, fusion weights, and camera indexes); it employs asynchronous transmission and a double-buffering mechanism to achieve complete temporal overlap between data transmission and GPU computation; and it pre-computes fusion weights, performing only table lookups and interpolation online. These synergistic technical features stabilize single-frame processing latency at 4.23ms (fluctuation <0.5ms), achieving an equivalent frame rate exceeding 200FPS, fundamentally solving the core problem of "performance fluctuations in the stitching stage" in existing solutions.

[0063] This invention is the first to systematically apply asynchronous transmission and double buffering mechanisms in an unmanned agricultural machinery surround-view panoramic system, achieving complete temporal overlap between data transmission and GPU computation: 1. Asynchronous transmission mechanism: Through the cudaMemcpyAsync asynchronous transmission function and CUDA streaming technology, data transmission and GPU kernel function execution are made parallel (a CUDA stream is created, cudaMemcpyAsync is called to asynchronously upload the image to the GPU memory, the function returns immediately without waiting for the transmission to complete, the kernel function is started in the same stream, the GPU hardware guarantees that the kernel function will be executed automatically after the transmission is completed, and the CPU thread continues to acquire the next frame of image, so as to achieve complete parallelism between data transmission and GPU computing and eliminate waiting time).

[0064] 2. Double buffering mechanism: Two independent buffers (Buffer A and Buffer B) are allocated in the GPU memory and used in a loop by switching between them using pointers. This allows the image in the current buffer to be processed by the CUDA Kernel while the next frame of the image is being transmitted through the PCIe (Peripheral Component Interconnect Express) bus in the other buffer.

[0065] 3. Synergy with Compact LUTs: Compact LUTs (12 bytes / pixel) reduce GPU kernel execution time by approximately 18%, creating greater timing overlap space for asynchronous transmission; asynchronous transmission allows the low-bandwidth advantage of compact LUTs to be fully utilized. The synergy between the two enables complete overlap between data transmission and GPU computation, with single-frame processing latency stabilized at 4.23ms and fluctuations less than 0.5ms, fundamentally solving the performance instability problem of existing solutions.

[0066] In this embodiment, the panoramic visualization output module is the final presentation module following the CUDA accelerated image stitching module. Its core task is to display the stitched 360° bird's-eye view to the remote control driver in an intuitive and user-friendly manner, and to provide switchable visualization solutions according to different agricultural machinery operating scenarios. The software implementation of this module is the human-machine interface of the vehicle-mounted display screen, and it is a component of the terminal output during the system's online phase.

[0067] This module receives panoramic bird's-eye view data (1200×1600 pixels resolution, frame rate ≥200FPS) output from the CUDA-accelerated image stitching module and transmits it to the remote control terminal via the vehicle network interface. To adapt to the needs of different agricultural machinery operating scenarios, this module provides two switchable visualization schemes, such as... Figure 5As shown: Scheme 1 is a "four-window + bird's-eye view" layout: the display resolution is W×H (width×height). The "four-window + bird's-eye view" layout divides the display area into two equal-height areas (top and bottom). The upper half is further divided into two equal-height areas (top and bottom) and two equal-width areas (left and right), forming a 2×2 grid layout, displaying the distortion-free real-time images from the front, back, left, and right directions respectively. The lower half is a full-width area, displaying the stitched bird's-eye view. Let I... front For the distortion-corrected image of the forward-looking camera, I rear For the distortion-corrected image of the rear-view camera, I left For the distortion-corrected image from the left-view camera, I right For the right-view camera's distortion-corrected image, I bev For bird's-eye view, I output For the final output image, W cam With H cam Given the resolution of the camera image, the output image I... output The pixel value is determined by the following piecewise function: The first scheme simultaneously displays real-time, distortion-free images from the front, back, left, and right directions, along with a central bird's-eye view, suitable for work scenarios requiring observation of details in specific directions. The second scheme is a "front view + bird's-eye view + panoramic view" layout: with a display resolution of W×H, this layout divides the display area into a left and right region. The left region is 2W / 3 wide and H high, further divided into two equal-height sub-regions. The right region is W / 3 wide and H high, displaying the bird's-eye view. The front view occupies the upper left half, the panoramic view occupies the lower left half, and the bird's-eye view occupies the full height of the right side. Let I... front For the distortion-corrected image of the forward-looking camera, I surround For a 360° panoramic view unfolded diagram, I bev For panoramic bird's-eye view, I output For the final output image, W cam With H cam For camera image resolution, W sur With H sur To achieve the resolution of the panoramic view, output image I... output The pixel value is determined by the following piecewise function: Emphasizing forward visibility and overall surround view, this invention is suitable for both driving and control scenarios. Compared to traditional single visualization solutions, this invention employs two conditionally variable visualization schemes: when the agricultural machinery is stationary, turn signals are on, or the driver selects the "implementation attachment / field turning" mode, a "four-window + bird's-eye view" layout is used, facilitating the driver's observation of surrounding obstacles and attachment point locations; when the agricultural machinery's speed exceeds 5 km / h, there is no steering operation, or the driver selects the "field driving / road driving" mode, a "forward window + bird's-eye view + panoramic view" layout is used, prioritizing forward visibility and overall environmental perception. The module also optimizes icons for blind spots, overlaying highly transparent agricultural machinery icons on the bottom area of ​​the agricultural machinery in the bird's-eye view (invalid areas marked with id=-1 in the LUT), clearly indicating the machinery's location.

[0068] In summary, 1. This invention effectively reduces the latency of single-frame image processing through a two-stage architecture of offline pre-computation and online CUDA parallel acceleration. Traditional vehicle surround-view systems typically have an end-to-end overall latency of around 80-100ms from camera acquisition to image display. The latency of sensor exposure, data transmission, and display refresh is determined by hardware, and the latency of similar hardware is not significantly different. However, the measured single-frame processing latency of this invention is only 4.23ms, which is only 1 / 20 to 1 / 25 of the traditional solution, with an equivalent frame rate exceeding 200FPS and an end-to-end overall latency of approximately 30-60ms. This provides smooth, lag-free visual feedback for remote control of unmanned agricultural machinery, ensuring that the driver can perceive obstacles and operational status around the machinery in real time in complex field environments, significantly improving operational safety and the user experience.

[0069] 2. The adoption of an offline pre-computation architecture significantly reduces maintenance costs and effectively saves GPU computing resources. When camera parameters change, only the offline script needs to be rerun to update the lookup table, without modifying or recompiling the online CUDA program. The vehicle-mounted device only performs table lookup, interpolation, and fusion operations, transferring most of the geometric computation load to the development environment. This allows the system to run stably on the vehicle-mounted embedded GPU platform, greatly reducing the online computing power requirements of the vehicle-mounted embedded platform. This ensures that the upper-layer perception algorithms have sufficient GPU resources, guaranteeing real-time performance and stability during multi-task parallelism.

[0070] 3. Employing a compact LUT data structure of 12 bytes / pixel effectively reduces storage space and transmission time. Based on an output resolution of 1200×1600, the generated surround_view.binary file is approximately 22MB in size and can be loaded into GPU memory in one go. Compared to traditional floating-point storage solutions (28 bytes / pixel, approximately 53MB), this significantly reduces storage space while maintaining functional parity, significantly alleviating storage bandwidth pressure on embedded platforms and avoiding repeated reads.

[0071] 4. Two visualization layouts have been introduced for different agricultural machinery operation scenarios, significantly improving the intuitiveness and ease of use of remote control. Option 1, "Four Views + Bird's-eye View," is suitable for operation scenarios requiring observation of specific directional details (such as implement attachment and field turning). Option 2, "Front View + Bird's-eye View + Panoramic View," is suitable for movement and control scenarios. Simultaneously, icon optimization has been implemented for blind spots on agricultural machinery. High-transparency agricultural machinery icons are overlaid on the bottom area of ​​the agricultural machinery in the bird's-eye view (the invalid area marked with id=-1 in the LUT), clearly indicating the machinery's location and significantly improving the intuitiveness and ease of use of remote control.

[0072] 5. Offers dual-mode calibration, including a full-area version and a simplified version, significantly reducing deployment and maintenance costs. Users can flexibly choose the mode based on actual needs. The full-area version uses a 13×11 large calibration board stacked to cover the entire area around the agricultural machinery, along with 5×5 small calibration cloths at the four corners, suitable for high-precision factory calibration and initial installation scenarios. The simplified version calibrates the bird's-eye projection module separately in the field after the fisheye camera calibration, using only the 5×5 small calibration cloths at the four corners, suitable for rapid field calibration and on-site maintenance scenarios. By modifying four parameters—lateral and longitudinal stretching amplitude and lateral and longitudinal translation distance—camera installation errors can be quickly compensated on-site without rerunning complex global calibration algorithms, significantly reducing deployment and maintenance costs.

[0073] 6. The WSL environment enables seamless compatibility between Windows and native Linux CUDA programs, significantly reducing the differences between the development and final deployment environments, shortening the R&D iteration cycle, and improving system maintainability. Developers and operations personnel can use the Linux toolchain for code writing, debugging, and performance optimization within the familiar Windows interface. Once the algorithm verification is successful, the program can be directly ported to the native Linux system of the in-vehicle central control unit.

[0074] 7. Quantitative verification of synergistic effect: To prove the systematic emergent characteristics of the present invention, comparative experiments were conducted as shown in Table 3 to verify the performance difference between the independent existence and the synergistic existence of each technical feature.

[0075] Table 3 Comparison of performance differences between independent and synergistic existence of various technical features. Example 2 This invention also provides a method for implementing a CUDA-accelerated panoramic system for unmanned agricultural machinery in a WSL environment. The system is used to implement the method described in Embodiment 1, and the implementation method includes: Acquire calibration images from four fisheye cameras; After standardizing the calibration images of the four fisheye cameras, the generated camera intrinsic parameter matrix and distortion coefficients are stored separately in YAML format files. Obtain a YAML file, calculate the projection matrix by interactively selecting four marker points, and append the projection matrix to the same YAML file to form a complete camera parameter file. Based on the complete camera parameter file, after completing the pixel geometric mapping calculation between the bird's-eye view and the original image, a binary lookup table (LUT) is generated. It simultaneously receives real-time operational images and binary LUT files from four fisheye cameras, completes distortion correction and panoramic stitching, and generates a panoramic bird's-eye view for panoramic visualization output.

[0076] The specific implementation process includes the following steps: S101 fisheye camera calibration: A checkerboard calibration board with 12×10 internal corner points (corresponding to 13×11 squares) and a single square size of 20cm×20cm was used to calibrate fisheye cameras installed in the front, rear, left, and right directions of the agricultural machinery. During calibration image acquisition, the checkerboard was ensured to cover both the center and edges of the image, and to include multiple sets of viewpoints with different angles and distances. 50-65 calibration images were acquired for each camera. Corner point detection and parameter optimization were performed using the OpenCV fisheye camera distortion model to solve for the intrinsic parameter matrix and distortion coefficients of each camera, and the results were stored separately in YAML format files.

[0077] S102 Bird's-eye View Projection Calibration: Read the YAML file generated in S101, run the run_get_projection_maps.py script, and perform projection calibration in the front, back, left, and right directions respectively; after the script displays the fisheye image, click the four marker points in the order of upper left, upper right, lower left, and lower right. If an incorrect point is selected, press the d key to delete it; control the horizontal and vertical stretching of the projection using the -scale xy parameter (value range 0.3-0.6), and control the horizontal and vertical translation distance of the projection using the -shift ab parameter (value range -500 to 500 pixels); based on the coordinates of the four selected marker points and the preset ideal coordinates, combined with the camera intrinsic matrix and distortion coefficients from step 1, calculate the 3×3 projection matrix, and append this matrix to the corresponding YAML file to form a complete camera parameter file containing the intrinsic matrix, distortion coefficients, and projection matrix.

[0078] S103 Offline Mapping Table Generation: Read the complete YAML parameter file generated in S102, and read the 4-channel weight map weights.png (the four channels correspond to the four overlapping regions: left rear, right front, left front, and right rear, respectively); using the OpenCV fisheye module and PyTorch's grid_sample function, calculate the coordinate mapping relationship between each pixel of the output bird's-eye view and the original fisheye image; assign fusion weights (0-255) to the overlapping regions according to the 4-channel weight map; package all calculation results into a binary lookup table file surround_view.binary in a compact format of 12 bytes / pixel; the compact format differs from the storage and retrieval strategy optimization path and compression algorithm path in the existing technology. By analyzing the coordinate range (0-4000 pixels) and weight precision (0-255 levels) of the agricultural machinery surround view scene, the short type is selected to store coordinates, the uchar type to store fusion weights, and the byte type to store camera indexes, realizing a 12-byte compact structure that can be used directly without decompression.

[0079] S104 CUDA-accelerated real-time stitching: The binary lookup table file generated by S103 is loaded into GPU memory; the working images from four fisheye cameras are acquired in real time and uploaded to GPU global memory via cudaMemcpy, using asynchronous transmission and double buffering mechanisms to overlap data transmission and computation; a CUDA kernel function is started, allocating a GPU thread for each pixel of the output bird's-eye view, and each thread performs the following operations: reads the primary camera coordinates (u1, v1), secondary camera coordinates (u2, v2), camera index (id1, id2), and fusion weight w (weig) from the lookup table. If id1 is valid, obtain pixel value P1 from the corresponding camera image through bilinear interpolation; if id2 is valid, obtain pixel value P2 from the secondary camera image through bilinear interpolation, and execute the weighted fusion formula P_out=(P1×w+P2×(255-w))>>8 (using shift operation instead of division here); if id2 is invalid, then P_out=P1; if id1 is invalid, the pixel is set to the background color; write the fused pixel value to the corresponding position in the output image; download the stitched bird's-eye view back to CPU memory for visualization output using cudaMemcpy.

[0080] S105 Panoramic Visualization Output: Receives panoramic bird's-eye view data output from S104 and provides two visualization layouts: Option 1 is a "four-window + bird's-eye view" layout, which simultaneously displays the distortion-free real-time images in the front, back, left, and right directions, along with the central bird's-eye view; Option 2 is a "front window + bird's-eye view + panoramic view" layout; a highly transparent agricultural machinery icon is overlaid on the bottom area of ​​the agricultural machinery in the bird's-eye view (the invalid area marked as id=-1 in the LUT).

[0081] In this embodiment, the fisheye camera calibration in S101 (fisheye camera calibration) and S102 (bird's-eye projection calibration) adopts a dual-mode calibration strategy of full-field version and simplified version, specifically as follows: Figure 6 As shown, the full-area version involves simultaneously setting up a 13×11 large calibration board to cover the entire area around the agricultural machinery in the calibration site, and placing 5×5 small calibration cloths at the four corners of the agricultural machinery. The camera calibration and projection calibration are completed in one site setup. The simplified version involves using only the 5×5 small calibration cloths at the four corners for bird's-eye projection calibration after the fisheye camera calibration is completed.

[0082] In this embodiment, the binary lookup table (LUT) used in the offline mapping table generation in S103 adopts a compact 12-byte / pixel format. The data structure of each pixel includes: the primary camera coordinates (u1, v1) stored in short type, the secondary camera coordinates (u2, v2) stored in short type, the fusion weight stored in uchar type, the primary camera index id1 and the secondary camera index id2 stored in byte type, and a padding field for byte alignment.

[0083] In this embodiment, the asynchronous transmission and double buffering mechanism described in S104 CUDA-accelerated real-time stitching is implemented. Specifically, two independent image buffers are allocated in the GPU memory. Through the cudaMemcpyAsync asynchronous transmission function and CUDA streaming technology, while the image in the current buffer is being processed by the CUDA Kernel, the next frame image is being transmitted through the PCIe bus in the other buffer. The buffer roles are switched cyclically using pointers.

[0084] In this embodiment, the two visualization layouts described in the S105 panoramic visualization output are as follows: Scheme 1 is a "four-window + bird's-eye view" layout, which simultaneously displays the distortion-free real-time images in the front, back, left, and right directions, as well as the central bird's-eye view; Scheme 2 is a "front window + bird's-eye view + panoramic view" layout.

[0085] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A CUDA-accelerated panoramic surround-view system for unmanned agricultural machinery in a WSL environment, characterized in that, The system includes: a fisheye camera image acquisition module, a fisheye camera calibration module, a bird's-eye projection module, an offline mapping table generation module, a CUDA-accelerated image stitching module, and a panoramic visualization output module. The fisheye camera image acquisition module is used to transmit the acquired calibration images from the four fisheye cameras to the fisheye camera calibration module. The fisheye camera calibration module is used to standardize the calibration images of the four fisheye cameras and then store the generated camera intrinsic parameter matrix and distortion coefficients in YAML format files respectively. The bird's-eye view projection module is used to acquire a YAML format file, calculate the projection matrix by interactively selecting four marker points, and append the projection matrix to the same YAML format file to form a complete camera parameter file. Finally, the YAML format file containing the intrinsic parameter matrix, distortion coefficients and projection matrix is ​​transmitted to the offline mapping table generation module. The offline mapping table generation module is used to complete the pixel geometry mapping calculation between the bird's-eye view and the original image, and then transmit the generated binary lookup table (LUT) to the CUDA-accelerated image stitching module. The CUDA-accelerated image stitching module is used to simultaneously receive real-time working images and binary LUT files from four fisheye cameras, and after completing distortion correction and panoramic stitching, transmit the generated panoramic bird's-eye view to the panoramic visualization output module. The panoramic visualization output module is used to visualize the panoramic bird's-eye view for remote control by the driver.

2. The system according to claim 1, characterized in that, The fisheye camera calibration module uses a standard calibration grid of 12×10 inner corner points for agricultural machinery with a working distance of 0-4 meters. The width of a single grid is 20cm.

3. The system according to claim 1, characterized in that, The process of calculating the projection matrix by interactively selecting four marker points and appending the projection matrix to the same YAML file to form a complete camera parameter file includes: Read the generated YAML file and run the run_get_projection_maps.py script to perform projection calibration in the four directions of front, back, left, and right. After the script displays the fisheye image, click the four markers in the order of top left, top right, bottom left, and bottom right. If you click the wrong marker, press the d key to delete it. The -scale xy parameter controls the horizontal and vertical stretching of the projection, with a value range of 0.3-0.

6. The -shifta b parameter controls the horizontal and vertical translation distance of the projection, with a value range of -500 to 500 pixels. Based on the coordinates of the four selected marker points and the preset ideal coordinates, combined with the camera intrinsic parameter matrix and distortion coefficients read from the YAML format file, a 3×3 projection matrix is ​​calculated and appended to the corresponding YAML file to form a complete camera parameter file containing the intrinsic parameter matrix, distortion coefficients, and projection matrix.

4. The system according to claim 1, characterized in that, The generation process of the binary lookup table (LUT) includes: using the calibrated camera parameters, pre-calculating and packaging a binary lookup table containing geometric mapping relationships and fusion weights, specifically: The script 1.py reads the YAML parameter files generated by the fisheye camera calibration module and the bird's-eye projection module, and reads the 4-channel weight map weights.png, where the four channels correspond to the four overlapping regions of left rear, right front, left front, and right rear, respectively. Using the fisheye module in OpenCV and the grid_sample function in PyTorch, the coordinate mapping relationship of each pixel in the bird's-eye image to the original fisheye image is calculated, and the fusion weights are assigned to the overlapping regions according to the 4-channel weight map. All calculation results are packaged in a compact format of 12 bytes / pixel to generate a binary lookup table file surround_view.binary, which is the specific form of the binary lookup table LUT.

5. The system according to claim 1, characterized in that, The process of simultaneously receiving real-time images and binary LUT files from four fisheye cameras, completing distortion correction and panoramic stitching, and generating a panoramic bird's-eye view includes: The initialization function is called to load the binary lookup table (LUT) output by the offline mapping table generation module into the CPU memory and upload it to the GPU memory at the same time, while allocating the video memory space for the input and output images. After acquiring the operational images from the four fisheye cameras in real time, the image data is uploaded to the GPU global memory via cudaMemcpy. Asynchronous transmission and double buffering mechanisms are used to overlap the data transmission with the computation task. The CUDA kernel function is started, and a GPU thread is allocated for each pixel of the output bird's-eye view. Each thread performs the following operations: reads the primary camera coordinates (u1, v1), secondary camera coordinates (u2, v2), camera index (id1, id2), and fusion weight w from the lookup table; if the id1 field is valid, obtain the pixel value P1 from the corresponding camera image through bilinear interpolation; if the id2 field is valid, obtain the pixel value P2 from the secondary camera image through bilinear interpolation, and execute the weighted fusion formula P_out=(P1×w+P2×(255-w))>>8; if the id2 field is invalid, then P_out=P1; if the id1 field is invalid, the pixel is set to the background color; writes the fused pixel value to the corresponding position in the output image; and downloads the stitched bird's-eye view back to CPU memory for visualization output using cudaMemcpy.

6. The system according to claim 5, characterized in that, Asynchronous transfer mechanism: Through the cudaMemcpyAsync asynchronous transfer function and CUDA streaming technology, data transfer and GPU kernel function execution are made parallel. Specifically: a CUDA stream is created, cudaMemcpyAsync is called to asynchronously upload the image to the GPU memory, the function returns immediately without waiting for the transfer to complete, the kernel function is started in the same stream, the GPU hardware guarantees that the kernel function will execute automatically after the transfer is completed, and the CPU thread continues to acquire the next frame of image, so as to achieve complete parallelism between data transfer and GPU computing and eliminate waiting time.

7. The system according to claim 5, characterized in that, Double buffering mechanism: Two independent buffers are allocated in the GPU memory, namely Buffer A and Buffer B, which are used in a loop by switching between pointers, so that while the image in the current buffer is being processed by the CUDA Kernel, the next frame image is being transmitted through the PCIe bus in the other buffer.

8. The system according to claim 1, characterized in that, The process of visualizing a panoramic bird's-eye view includes: When the agricultural machinery is stationary, the turn signal is on, or the driver selects the "implementation attachment / field turn" mode, the "four-window + bird's-eye view" layout scheme is adopted, which simultaneously displays the distortion-free real-time images in the front, back, left, and right directions and the central bird's-eye view. When the agricultural machinery travels at a speed exceeding 5 km / h, without steering operation, or when the driver selects the "field travel / road travel" mode, a "front window + bird's-eye view + panoramic view" layout scheme is adopted to prioritize ensuring forward visibility and overall environmental perception. To address blind spots in agricultural machinery operations, icons for agricultural machinery are overlaid at the bottom of the machinery in the bird's-eye view to indicate their location.

9. A method for implementing a CUDA-accelerated panoramic surround-view system for unmanned agricultural machinery in a WSL environment, wherein the method is implemented using the system described in any one of claims 1-8, characterized in that, The implementation method includes: Acquire calibration images from four fisheye cameras; After standardizing the calibration images of the four fisheye cameras, the generated camera intrinsic parameter matrix and distortion coefficients are stored separately in YAML format files. Obtain a YAML file, calculate the projection matrix by interactively selecting four marker points, and append the projection matrix to the same YAML file to form a complete camera parameter file. Based on the complete camera parameter file, after completing the pixel geometric mapping calculation between the bird's-eye view and the original image, a binary lookup table (LUT) is generated. It simultaneously receives real-time operational images and binary LUT files from four fisheye cameras, completes distortion correction and panoramic stitching, and generates a panoramic bird's-eye view for panoramic visualization output.

Images