Panorama stitching method and system based on pre-computed mapping and operator fusion

Abstract

The application discloses a panoramic splicing method and system based on pre-computed mapping and operator fusion, and belongs to the technical field of unmanned driving. The method comprises the following steps: constructing a coordinate mapping table according to the pixel coordinate mapping relationship between a panoramic overhead view and a plurality of camera images; wherein the panoramic overhead view comprises a non-overlapping area and an overlapping fusion area; filling color values of corresponding pixel coordinates into the non-overlapping area based on the coordinate mapping table; in the case that the feature points of the overlapping fusion area are not offset by more than a set threshold value between the front and rear frames, obtaining the pixel coordinates corresponding to the overlapping fusion area based on the coordinate mapping table, and filling the weighted fusion result of the color values of the pixel coordinates into the overlapping fusion area; and splicing the non-overlapping area and the overlapping fusion area to obtain the panoramic overhead view. The application can provide a panoramic environment view with high precision, seamless and millisecond-level response for a remote driving vehicle.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, specifically relating to a panoramic stitching method and system based on pre-computation mapping and operator fusion. Background Technology

[0002] Autonomous driving technology is a research hotspot in the automotive industry and artificial intelligence fields. Remote driving, as an important supplement or transitional solution to autonomous driving systems, plays a crucial role in handling complex traffic environments, emergency takeovers, and specific application scenarios (such as ports, mining areas, and logistics parks). In remote driving scenarios, operators rely on environmental information acquired by onboard sensors to make driving decisions. Visual information is the most intuitive and important component. However, a single camera often has a limited field of view, failing to provide a complete view of the environment surrounding the vehicle.

[0003] To provide remote drivers with comprehensive 360-degree environmental perception, a multi-camera approach is typically used, stitching together images from multiple cameras mounted around the vehicle to create a complete panoramic or bird's-eye view. While existing image stitching technologies are diverse, they still face numerous challenges when applied to highly dynamic and safety-critical remote driving scenarios. First, high stitching accuracy is required; any noticeable misalignment or distortion could mislead the driver. Second, stitching seams need to be smooth to avoid visual discomfort and information loss due to differences in brightness, tone, or ghosting. More importantly, the entire stitching and rendering process must be highly real-time, with latency controlled to an extremely low level (e.g., milliseconds) to ensure the remote driver can perform timely and effective actions based on the current environmental conditions. Traditional stitching methods based on feature matching (such as SIFT and SURF) and fusion techniques, while achieving good accuracy and fusion effects, typically involve significant computational demands, making it difficult to implement high frame rates in real-time on in-vehicle embedded platforms. For example, Chinese patent application CN121019453A discloses a multi-camera collaborative vehicle panoramic imaging system, which requires distortion correction for each frame. Feature extraction (SIFT / ORB) match RANSAC The transformation process. This online full-scale calculation method results in extremely high computational redundancy, and in scenes with poor texture (such as road surfaces) or drastic lighting changes, feature point extraction is prone to failure, leading to abrupt changes in the stitching.

[0004] Therefore, designing a multi-camera image stitching system that can meet the requirements of high precision and seamless integration while achieving ultra-low latency processing and dynamic presentation is a key technical challenge for improving the safety and experience of remote driving. Summary of the Invention

[0005] Addressing the shortcomings of existing technologies in image stitching accuracy, real-time performance, and presentation quality in unmanned remote driving environment perception, this invention aims to provide a panoramic stitching method and system based on pre-computation mapping and operator fusion. This system optimizes feature extraction, matching, and fusion algorithms, and combines parallel computing and hardware acceleration technologies to provide remotely driven vehicles with a high-precision, seamless, and millisecond-level responsive panoramic environmental view.

[0006] To achieve the above objectives, the technical solution of the present invention includes the following:

[0007] A panoramic stitching method based on pre-computation mapping and operator fusion, the method comprising: A coordinate mapping table is constructed based on the pixel coordinate mapping relationship between the panoramic top view and multiple camera images; wherein, the panoramic top view includes non-overlapping areas and overlapping and blended areas; Based on the coordinate mapping table, the color values ​​of the corresponding pixel coordinates are filled into the non-overlapping areas; If the offset between consecutive frames does not exceed a set threshold, the pixel coordinates corresponding to the overlapping and fusion region are obtained based on the coordinate mapping table, and the weighted fusion result of the color value of the pixel coordinates is filled into the overlapping and fusion region. By stitching together the non-overlapping areas and the overlapping and merged areas, a panoramic top view is obtained.

[0008] Furthermore, based on the pixel coordinate mapping relationship between the panoramic top-down view and multiple camera images, a coordinate mapping table is constructed, including: Obtain the intrinsic parameter models of each camera; The extrinsic parameter models of each camera are obtained by calibrating the site or by initial feature matching based on the natural scene; wherein, the extrinsic parameter models include: the rotation matrix and translation vector of the camera relative to the vehicle center coordinate system; By defining the target resolution of the panoramic top view, the pixel coordinates of the panoramic top view are obtained; For each pixel coordinate in the panoramic top view, the corresponding pixel coordinate on the camera image is derived in reverse based on the camera's intrinsic and extrinsic parameter models. Based on the reverse derivation results, a coordinate mapping table is generated.

[0009] Furthermore, based on the coordinate mapping table, the pixel coordinates corresponding to the overlapping and blending region are obtained, and the weighted fusion result of the color value of the pixel coordinates is filled into the overlapping and blending region, including: Based on the coordinate mapping table, read the corresponding camera image; The camera images are distributed to parallel processing units so that each processing unit processes the camera images to obtain the color values ​​of the pixel coordinates. The color value of a pixel coordinate in the overlapping and blended region is obtained by weighting the color value of the pixel coordinates according to the preset alpha weight mask.

[0010] Further, the camera image is processed to obtain the color values ​​of the pixel coordinates, including: In the overlapping area of ​​images from adjacent cameras, feature points are extracted based on a feature extraction algorithm; wherein, the feature extraction algorithm includes: SIFT algorithm or ORB algorithm; The feature point is used for matching, and a robust estimation strategy is combined to calculate the transformation matrix between adjacent camera images; wherein, the robust estimation strategy includes: RANSAC strategy; Based on the transformation matrix, a perspective transformation is performed on the overlapping area of ​​images from adjacent cameras to obtain the color values ​​of the pixel coordinates.

[0011] Furthermore, the algorithm for obtaining the offset of the feature points in the preceding and following frames includes: a sparse optical flow algorithm.

[0012] Furthermore, if the feature points in the overlapping fusion region are offset by a set threshold between consecutive frames, the method further includes: Update the coordinate mapping table based on the current pixel coordinate mapping relationship between the panoramic top view and multiple camera images; Based on the updated coordinate mapping table, the pixel coordinates corresponding to the overlapping and blending region are obtained based on the coordinate mapping table, and the weighted fusion result of the color value of the pixel coordinates is filled into the overlapping and blending region.

[0013] Furthermore, after stitching together the non-overlapping areas and the overlapping and blended areas to obtain a panoramic top view, the method further includes: Obtain vehicle auxiliary lines, which include: trajectory lines and warning boxes; Overlay vehicle guide lines onto the panoramic top view; Transmit a panoramic top-down view with overlaid vehicle auxiliary lines to the remote control terminal.

[0014] A panoramic stitching system based on pre-computation mapping and operator fusion, the system comprising: The mapping table construction module is used to construct a coordinate mapping table based on the pixel coordinate mapping relationship between the panoramic top view and multiple camera images; wherein, the panoramic top view includes non-overlapping areas and overlapping and blended areas; The color filling module is used to fill the non-overlapping area with the color value of the corresponding pixel coordinate based on the coordinate mapping table; if the feature point of the overlapping fusion area does not exceed the set threshold in the frame offset, the pixel coordinate corresponding to the overlapping fusion area is obtained based on the coordinate mapping table, and the weighted fusion result of the color value of the pixel coordinate is filled into the overlapping fusion area. The stitching module is used to stitch together non-overlapping and overlapping areas to obtain a panoramic top view.

[0015] A computer device, comprising: a processor and a memory storing computer program instructions; wherein the processor, when executing the computer program instructions, implements the panoramic stitching method based on pre-computation mapping and operator fusion as described above.

[0016] A computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the panoramic stitching method based on pre-computation mapping and operator fusion as described above.

[0017] Compared with the prior art, the present invention has at least the following beneficial effects.

[0018] 1. This invention eliminates the computationally intensive feature extraction step, reducing the O(N) complexity of image analysis to an O(1) lookup table operation, which significantly reduces latency and improves stability in weak texture environments.

[0019] 2. High-precision stitching: Combining robust features such as SIFT / ORB and RANSAC optimization, high-precision image geometric alignment is achieved, reducing stitching errors and deformation.

[0020] 3. Seamless visual effect: The multi-band fusion technology effectively eliminates splicing gaps, providing a continuous panoramic view with natural color and brightness transitions.

[0021] 4. High real-time performance: Through parallel computing and hardware acceleration optimization, the processing latency is significantly reduced, and millisecond-level response can be achieved on high computing power platforms, meeting the stringent real-time requirements of remote driving.

[0022] 5. Enhanced environmental perception: Provides remote drivers with complete, clear, and real-time information about the vehicle's surrounding environment, greatly improving their environmental perception capabilities.

[0023] 6. Enhance human-machine interaction and safety: Enables remote drivers to better understand the vehicle's environment and the behavior of the autonomous driving system, increasing their confidence and ability to intervene when necessary, thereby improving the safety and reliability of the entire remote driving system. Attached Figure Description

[0024] Figure 1 Overall architecture diagram of the present invention.

[0025] Figure 2 A flowchart of a panoramic stitching method based on pre-computation mapping and operator fusion.

[0026] Figure 3Block diagram of a panoramic stitching system based on pre-computation mapping and operator fusion.

[0027] Figure 4 Example image of a stitched panoramic image dynamically displayed on a remote driving interface.

[0028] Figure 5 A block diagram of computer equipment. Detailed Implementation

[0029] The present invention will be further described below with reference to possible accompanying drawings and specific embodiments, but this does not constitute any limitation on the present invention.

[0030] The panoramic stitching system based on pre-computation mapping and operator fusion of the present invention, such as Figure 1 As shown, it includes a vehicle-side terminal and a remote control terminal. The vehicle-side terminal is used for image acquisition and preprocessing, and typically includes at least four wide-angle cameras mounted around the vehicle's perimeter (e.g., front, rear, left, and right views), as well as an onboard processing unit with high-performance computing capabilities (such as an embedded computer with an embedded GPU or an industrial control computer with a dedicated GPU). The remote control terminal includes corresponding display devices and network communication modules, which are used for parallel computing and hardware acceleration to integrate information and provide intuitive visualization.

[0031] Figure 2 This is a flowchart of a panoramic stitching method based on pre-computation mapping and operator fusion. The panoramic stitching method includes: Step 1: Construct a coordinate mapping table based on the pixel coordinate mapping relationship between the panoramic top view and multiple camera images. The panoramic top view includes non-overlapping areas and overlapping and blended areas. Step 2: Based on the coordinate mapping table, fill the non-overlapping area with the color value of the corresponding pixel coordinate; Step 3: If the offset between the feature points in the overlapping and fusion region does not exceed the set threshold, obtain the pixel coordinates corresponding to the overlapping and fusion region based on the coordinate mapping table, and fill the overlapping and fusion region with the weighted fusion result of the color value of the pixel coordinates. Step 4: Stitch together the non-overlapping areas and the overlapping areas to obtain a panoramic top view.

[0032] Figure 3 The diagram shows a panoramic stitching system based on pre-computed mapping and operator fusion. The panoramic stitching system includes a mapping table construction module, a color filling module, and a stitching module.

[0033] (a) Mapping table construction module.

[0034] The mapping table construction module is used to construct a coordinate mapping table (LUT) based on the pixel coordinate mapping relationship between the panoramic top-down view and multiple camera images; wherein, the panoramic top-down view includes non-overlapping areas and overlapping and blended areas. Specifically, the present invention simultaneously acquires image data from multiple (e.g., four) wide-angle cameras installed around the vehicle (e.g., front, rear, left, and right) and combines this data with the panoramic top-down view to construct the coordinate mapping table.

[0035] In one embodiment, upon startup or in calibration mode, the invention first performs a one-time geometric relationship calculation to obtain a coordinate mapping table. The steps include: 1) Obtain the intrinsic parameter models of each camera; 2) Obtain the extrinsic parameter model of each camera by calibrating the site or by initial feature matching based on the natural scene; wherein, the extrinsic parameter model includes: the rotation matrix and translation vector of the camera relative to the vehicle center coordinate system; 3) By defining the target resolution of the panoramic top view, the pixel coordinates of the panoramic top view are obtained; 4) For each pixel coordinate in the panoramic top-down view, the corresponding pixel coordinates on the camera image are derived in reverse based on the camera's intrinsic and extrinsic parameter models. Specifically, extrinsic parameter models such as the rotation matrix and translation vector of each camera relative to the vehicle's center coordinate system are obtained by calibrating the site (e.g., a checkerboard pattern) or by initial feature matching based on the natural scene (run only once). 5) Based on the reverse derivation results, a coordinate mapping table is generated. In this invention, the corresponding sampled coordinates are saved as a coordinate mapping table. This LUT table is essentially a matrix of the same size as the panoramic image, storing the source image index and source coordinates corresponding to each output pixel.

[0036] In a preferred embodiment, the present invention also performs weight mask pre-computation, that is, identifies the overlapping region of adjacent cameras, pre-calculates the alpha weight mask required for fusion, and stores it in video memory.

[0037] (ii) Color fill module.

[0038] The color filling module is used to fill the non-overlapping regions with the color values ​​of corresponding pixel coordinates based on a coordinate mapping table. For feature points in the overlapping fusion region, provided the frame offset does not exceed a set threshold, the corresponding pixel coordinates are obtained from the coordinate mapping table, and the weighted fusion result of the color values ​​of those pixel coordinates is filled into the overlapping fusion region. Specifically, during vehicle movement, the system enters a real-time processing loop. At this time, feature extraction is no longer performed; instead, pixel remapping is performed based on the generated LUT. To achieve an ultra-low latency of less than 15ms, this invention employs GPU / NPU-based kernel fusion technology.

[0039] In one embodiment, the color filling process includes the following steps.

[0040] 1. Zero-copy video memory data: YUV / RGB data captured by the camera is directly transferred to the accelerator's video memory via DMA, avoiding the memory copy overhead between the CPU and GPU.

[0041] 2. Region Partitioning Strategy: Different computational kernels are allocated to non-overlapping regions and overlapping / merged regions.

[0042] 2.1) Non-overlapping area: This area is covered by only a single camera. A lightweight copy kernel is activated, directly reading pixel values ​​from the source image based on the LUT and filling them into the target buffer, without performing any weighted calculations, greatly saving computing power.

[0043] 2.2) Fusion Region: For the fusion region, this invention does not employ step-by-step processing (i.e., not following the order of "distortion correction -> temporary storage -> perspective transformation -> temporary storage -> fusion"), but instead uses a unified computation kernel. The following operations are completed in a single GPU thread.

[0044] A. Read the coordinate mapping table to obtain the sampling coordinates of the current pixel in the left and right images respectively.

[0045] B. Distribute the camera images to parallel processing units so that each processing unit can process the camera images and obtain the color values ​​of the pixel coordinates.

[0046] Specifically, in the overlapping area of ​​images from adjacent cameras, this invention first extracts feature points based on the SIFT or ORB algorithm; then, it uses these feature points for matching and combines robust estimation strategies such as RANSAC to accurately calculate the transformation matrix (such as homography matrix) between adjacent images, effectively eliminating interference from mismatched point pairs; finally, it performs perspective transformation on the overlapping area of ​​images from adjacent cameras based on the transformation matrix to obtain the color values ​​of pixel coordinates.

[0047] C. Weighted fusion: Read the pre-computed alpha mask, perform weighted calculations on the color values ​​of the pixel coordinates, and obtain the color value of a pixel coordinate in the overlapping fusion region.

[0048] D. Write back: Write the final color value to the display memory.

[0049] This design reduces the number of memory read / write operations from the traditional 3-4 times to 1 time, breaking through the memory bandwidth bottleneck of high-resolution image processing and being a decisive factor in achieving a low latency of 15ms.

[0050] In a preferred embodiment, to prevent the LUT from failing due to minor changes in the physical position of the camera caused by severe vehicle vibrations, the system runs a low-frequency (e.g., once every 5 frames) monitoring thread in the background. This includes: selecting a small number of fixed feature points (e.g., 5-10 ROI regions) in the overlapping area of ​​adjacent cameras, calculating the displacement vectors of the feature points in the preceding and following frames using the Sparse Optical Flow method; then, calculating the average displacement vector; if the displacement is within a preset tolerance range (e.g., pixels), the extrinsic parameters are considered stable, and the current LUT is continued to be used; if the displacement continuously exceeds a threshold, it is determined that the camera has undergone physical displacement, triggering a background recalibration process to update the LUT table.

[0051] (iii) splicing module.

[0052] The stitching module is used to stitch together non-overlapping areas and overlapping areas to obtain a panoramic top-down view. In a preferred embodiment, the invention further superimposes vehicle auxiliary lines (such as trajectory lines and warning boxes) onto the panoramic top-down view before transmitting it to the vehicle display screen via an HDMI / FPD-Link interface.

[0053] The present invention will be further illustrated by a specific example below.

[0054] Camera calibration: Before the system runs, all cameras need to be precisely calibrated, both internally and externally. Internal parameter calibration is used to correct lens distortion, while external parameter calibration is used to determine the precise position and orientation of each camera relative to the vehicle's coordinate system. Precise calibration is the foundation for achieving high-quality stitching.

[0055] Feature extraction and matching: For each pair of adjacent camera images with overlapping regions, the SIFT or ORB algorithm is run in real time. For example, the ORB algorithm is a good choice for scenarios with high real-time requirements due to its faster speed and better rotation invariance. The extracted feature points and their descriptors are used for subsequent matching.

[0056] RANSAC Optimization and Transformation Calculation: By matching feature point pairs, the RANSAC algorithm iteratively estimates the transformation model (usually the homography matrix H) that best explains the geometric relationship between most matched point pairs. RANSAC can effectively eliminate interference caused by dynamic objects or mismatched features in the scene, resulting in robust transformation parameters.

[0057] Image Transformation and Fusion: Based on the calculated homography matrix H, one (or two) images are transformed to the coordinate system of the reference image. Then, a multi-band fusion algorithm is applied to the overlapping areas of the images. This algorithm decomposes the image into different spatial frequency bands (e.g., using a Laplacian pyramid), performs fusion independently in each frequency band (e.g., smooth transition in the low-frequency band, and retain more details in the high-frequency band), and finally reconstructs the fused images from each frequency band to obtain a seamless and detailed stitched result.

[0058] Parallelism and Hardware Acceleration: To achieve millisecond-level real-time performance, the entire processing flow needs to be optimized. For example, independent computing threads or tasks can be allocated to stitching together each pair of adjacent cameras. Computationally intensive parts such as feature extraction, image transformation, and fusion can be performed in large-scale parallel computing on GPUs using programming models such as CUDA (for NVIDIA GPUs) or OpenCL. Through carefully designed parallel strategies and hardware acceleration, the processing time of the entire stitching process can be made much shorter than the camera acquisition cycle (e.g., 33ms corresponds to 30FPS).

[0059] Dynamic presentation (refer to) Figure 4 The stitched panoramic image (which can be a bar chart, bird's-eye view, or other form) is transmitted in real time to the remote driving control console via the network and dynamically refreshed on the operator's interface. The operator can accurately grasp the dynamics of the environment around the vehicle through this continuously updated, seamless, wide-view image.

[0060] System performance: On an in-vehicle computing platform equipped with a high-performance GPU, this system can process input from four high-definition cameras and stably output stitched panoramic images at a frame rate of 30 FPS or higher. The end-to-end latency, as shown in Table 1, is controlled within tens of milliseconds. Table 1 In summary, through the implementation of this invention, remote drivers can obtain a wide, clear, and real-time environmental view as if they were actually there, which significantly improves the safety and efficiency of remote control and provides strong support for their understanding and assistance of the autonomous driving system.

[0061] This invention belongs to the category of "static mapping + dynamic monitoring". Utilizing the relatively fixed nature of the camera, complex geometric transformations are pre-calculated into a coordinate mapping table. During runtime, full-image feature extraction is not performed; mapping is completed directly through table lookup. Only a lightweight sparse optical flow is introduced to monitor whether the camera experiences physical displacement (such as severe jolts), and the LUT is only reset when a threshold is triggered. In this way, the computationally expensive feature extraction stage is eliminated, reducing the O(N) complexity of image analysis to an O(1) table lookup operation, significantly reducing latency and improving stability in weakly textured environments.

[0062] This invention proposes a specific memory bandwidth optimization strategy. Specifically, it divides the image memory into "non-overlapping areas" (direct copying) and "overlapping areas" (fusion calculations) through region divide-and-conquer, avoiding indiscriminate weighted calculations across the entire image. At the GPU / NPU level, "distortion correction," "perspective transformation," and "multi-band fusion" are merged into a single computational kernel. This avoids repeated read / write operations between memory and on-chip cache, achieving a low latency of 15ms, far superior to the 100ms of existing technologies.

[0063] Figure 5 This is a block diagram illustrating a computer device according to an exemplary embodiment. The computer device may be a terminal, laptop computer, desktop computer, server, computer cluster, or other type of computer device. The computer device may include at least one processor and memory. The processor can execute instructions stored in the memory. The processor is communicatively connected to the memory via a data bus. In addition to the memory, the processor may also be communicatively connected to input devices, output devices, and communication devices via the data bus.

[0064] The processor can be any conventional processor. Processors may include central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems on chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0065] Memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0066] In this embodiment of the invention, an executable instruction is stored in a memory. The processor can read the executable instruction from the memory and execute the instruction to implement all or part of the steps of the method of the invention.

[0067] In addition to the methods and systems described above, exemplary embodiments of this disclosure also include a computer program product or a computer-readable storage medium storing the computer program product. The computer product includes computer program instructions that can be executed by a processor to perform all or part of the steps described in the exemplary embodiments above.

[0068] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages, and scripting languages ​​(e.g., Python). The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0069] Computer-readable storage media can take the form of any combination of one or more readable media. A readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: static random access memory (SRAM) having one or more electrically connected wires; electrically erasable programmable read-only memory (EEPROM); erasable programmable read-only memory (EPROM); programmable read-only memory (PROM); read-only memory (ROM); magnetic storage; flash memory; magnetic disk or optical disk; or any suitable combination thereof.

[0070] The above embodiments are merely illustrative of the technical solutions of the present invention and are not intended to limit it. Those skilled in the art can modify or make equivalent substitutions to the above technical solutions based on the concept of the present invention, and such modifications or equivalent substitutions should all be covered within the protection scope of the present invention. The protection scope of the present invention is defined by the claims.

Claims

1. A panoramic stitching method based on pre-computation mapping and operator fusion, characterized in that, The method includes: A coordinate mapping table is constructed based on the pixel coordinate mapping relationship between the panoramic top view and multiple camera images; wherein, the panoramic top view includes non-overlapping areas and overlapping and blended areas; Based on the coordinate mapping table, the color values ​​of the corresponding pixel coordinates are filled into the non-overlapping areas; If the offset between consecutive frames does not exceed a set threshold, the pixel coordinates corresponding to the overlapping and fusion region are obtained based on the coordinate mapping table, and the weighted fusion result of the color value of the pixel coordinates is filled into the overlapping and fusion region. By stitching together the non-overlapping areas and the overlapping and merged areas, a panoramic top view is obtained.

2. The method according to claim 1, characterized in that, Based on the pixel coordinate mapping relationship between the panoramic top-down view and multiple camera images, a coordinate mapping table is constructed, including: Obtain the intrinsic parameter models of each camera; The extrinsic parameter models of each camera are obtained by calibrating the site or by initial feature matching based on the natural scene; wherein, the extrinsic parameter models include: the rotation matrix and translation vector of the camera relative to the vehicle center coordinate system; By defining the target resolution of the panoramic top view, the pixel coordinates of the panoramic top view are obtained; For each pixel coordinate in the panoramic top view, the corresponding pixel coordinate on the camera image is derived in reverse based on the camera's intrinsic and extrinsic parameter models. Based on the reverse derivation results, a coordinate mapping table is generated.

3. The method according to claim 1, characterized in that, The pixel coordinates corresponding to the overlapping and blending region are obtained based on the coordinate mapping table. The weighted blending result of the color value of the pixel coordinates is then filled into the overlapping and blending region, including: Based on the coordinate mapping table, read the corresponding camera image; The camera images are distributed to parallel processing units so that each processing unit processes the camera images to obtain the color values ​​of the pixel coordinates. The color value of a pixel coordinate in the overlapping and blended region is obtained by weighting the color value of the pixel coordinates according to the preset alpha weight mask.

4. The method according to claim 3, characterized in that, The camera image is processed to obtain the color values ​​at pixel coordinates, including: In the overlapping area of ​​images from adjacent cameras, feature points are extracted based on a feature extraction algorithm; wherein, the feature extraction algorithm includes: SIFT algorithm or ORB algorithm; The feature point is used for matching, and the transformation matrix between adjacent camera images is calculated by combining a robust estimation strategy; wherein, the robust estimation strategy includes: RANSAC strategy; Based on the transformation matrix, a perspective transformation is performed on the overlapping area of ​​images from adjacent cameras to obtain the color values ​​of the pixel coordinates.

5. The method according to claim 1, characterized in that, The algorithm for obtaining the offset of the feature points in the preceding and following frames includes: sparse optical flow algorithm.

6. The method according to any one of claims 1 to 5, characterized in that, If the feature points in the overlapping fusion region are offset by a set threshold between consecutive frames, the method further includes: Update the coordinate mapping table based on the current pixel coordinate mapping relationship between the panoramic top view and multiple camera images; Based on the updated coordinate mapping table, the pixel coordinates corresponding to the overlapping and blending region are obtained based on the coordinate mapping table, and the weighted fusion result of the color value of the pixel coordinates is filled into the overlapping and blending region.

7. The method according to any one of claims 1 to 5, characterized in that, After stitching together the non-overlapping areas and the overlapping and blended areas to obtain a panoramic top view, the method further includes: Obtain vehicle auxiliary lines, which include: trajectory lines and warning boxes; Overlay vehicle guide lines onto the panoramic top view; Transmit a panoramic top-down view with overlaid vehicle auxiliary lines to the remote control terminal.

8. A panoramic stitching system based on pre-computation mapping and operator fusion, characterized in that, The system includes: The mapping table construction module is used to construct a coordinate mapping table based on the pixel coordinate mapping relationship between the panoramic top view and multiple camera images; wherein, the panoramic top view includes non-overlapping areas and overlapping and blended areas; The color filling module is used to fill the non-overlapping area with the color value of the corresponding pixel coordinate based on the coordinate mapping table; if the feature point of the overlapping fusion area does not exceed the set threshold in the frame offset, the pixel coordinate corresponding to the overlapping fusion area is obtained based on the coordinate mapping table, and the weighted fusion result of the color value of the pixel coordinate is filled into the overlapping fusion area. The stitching module is used to stitch together non-overlapping and overlapping areas to obtain a panoramic top view.

9. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the panoramic stitching method based on pre-computation mapping and operator fusion as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the panoramic stitching method based on pre-computation mapping and operator fusion as described in any one of claims 1-7.

Images