Digital-twin-oriented high-speed service area video inverse perspective mapping method and system

Abstract

The application discloses a high-speed service area video inverse perspective mapping method and system for digital twinning, which comprises the following steps: step one: loading a target layer and a source layer and aligning geographic information; step two: defining a bounding box of a region to be calibrated and generating sparse grid points; step three: calculating the corresponding coordinates of each grid vertex in the source layer and performing displacement calculation; step four: encapsulating the grid vertex displacement result, the control point pair and the view size information; step five: calculating the weight coefficient and the affine coefficient according to the calibration parameter data packet; step six: calculating the corresponding coordinates of each pixel coordinate of the target layer in the source layer; step seven: performing inverse perspective transformation on each pixel by looking up a mapping lookup table; and step eight: calculating an overlapping area and removing a seam. Through TPS mapping and gridding calibration, the application realizes accurate docking and real-time processing of high-speed service area video images.

Description

Technical Field

[0001] This invention relates to the field of image mapping technology, and in particular to a method and system for inverse perspective mapping of high-speed service area video for digital twins. Background Technology

[0002] With the rapid development of intelligent transportation systems and smart parks, video surveillance systems are playing an increasingly important role in urban management. Especially in vast areas such as highway service areas, real-time monitoring data is crucial for traffic management, security monitoring, and accident early warning. However, existing video surveillance systems often face difficulties in fusing video images with Geographic Information System (GIS) data, particularly when dealing with complex monitoring scenarios. Accurately matching video images with actual geographic information remains a challenge. Many existing video inverse perspective mapping methods rely on simple geometric transformations, such as perspective transformations based on homography matrices. While these methods can handle flat scenes, they cannot achieve accurate mapping for complex non-planar scenes, such as slopes, undulating roads, and distortion from wide-angle cameras in highway service areas. Therefore, existing methods struggle to guarantee high-precision image overlay and geographic information matching, especially when using wide-angle or fisheye lenses. Due to optical distortion caused by the lens, video image deformation is more severe, leading to large registration errors between the actual monitoring image and the geographic map, affecting data accuracy. Furthermore, traditional calibration methods, such as using a checkerboard pattern for camera intrinsic parameter calibration, often require complex on-site operations. Moreover, since monitoring equipment is typically installed at high elevations, traditional calibration methods are difficult to implement in practical applications. This results in significant bottlenecks in the efficiency of real-time video stream processing and calibration in dynamic scenarios.

[0003] Therefore, how to provide a method and system for high-speed service area video inverse perspective mapping for digital twins is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] One objective of this invention is to propose a method and system for inverse perspective mapping of high-speed service area videos for digital twins. This invention solves the image stitching error problems caused by wide-angle lens distortion and non-planar terrain in traditional technologies by introducing a TPS model and a gridded calibration method. By pre-generating a mapping lookup table (LUT), the efficiency of video stream processing is improved, achieving low-latency, high-precision inverse perspective mapping. Furthermore, a weighted fusion algorithm is used to process the stitching of multiple video streams, eliminating seams at the stitching points and generating a seamless panoramic video stream, greatly improving the quality of video images and the real-time performance and accuracy of the monitoring system.

[0005] The high-speed service area video inverse perspective mapping method for digital twins according to embodiments of the present invention includes the following steps:

[0006] Step 1: Load the orthophoto or CAD plan with geographic coordinate information as the target layer, and load the source image of the highway service area monitoring video to be calibrated as the source layer, and perform geographic information alignment to obtain the initial layer alignment result;

[0007] Step 2: Define the bounding box of the area to be calibrated on the target layer based on the initial layer alignment result, and generate sparse grid points according to the set step size to obtain the gridded calibration point set;

[0008] Step 3: Based on the set of meshed calibration points, calculate the corresponding coordinates of each mesh vertex in the source layer, perform displacement calculations, and generate the mesh vertex displacement results;

[0009] Step 4: Encapsulate the mesh vertex displacement results, control point pairs, and view dimension information to obtain the calibration parameter data package;

[0010] Step 5: Based on the calibration parameter data package, calculate the weight coefficients and affine coefficients using the known control point pairs and their corresponding coordinate relationships, establish a nonlinear mapping relationship between the target layer and the source layer, and obtain the TPS mapping model;

[0011] Step 6: Calculate the coordinates of each pixel in the target layer in the source layer based on the TPS mapping model, and generate a mapping lookup table;

[0012] Step 7: Access the video stream to obtain the current video frame, and perform inverse perspective transformation on each pixel by looking up the mapping lookup table to generate an inverse perspective mapped video stream;

[0013] Step 8: Receive the inverse perspective mapping video stream after multi-channel inverse perspective mapping processing, calculate the overlapping area, use weighted fusion to eliminate the stitching seams at the joints, and generate a seamless panoramic video stream.

[0014] Optionally, step one specifically includes:

[0015] An orthophoto or CAD plan with geographic coordinate information is loaded as the target layer. This target layer serves as the reference layer in the geographic information system and includes geospatial data of highway service areas.

[0016] Load the source image of the highway service area monitoring video to be calibrated as the source layer. The source layer is a single frame image of the actual monitoring video of the highway service area, including the real-time traffic conditions and monitoring information in the highway service area.

[0017] Set the source layer to semi-transparent until the source and target layers can be displayed simultaneously and visually contrasted by layer overlay.

[0018] In the same view, the position, rotation angle and scaling of the source layer are adjusted manually until the features in the monitoring video are aligned with the geographic information in the target layer, generating the initial layer alignment result.

[0019] Optionally, step two specifically involves:

[0020] Define a bounding box for the area to be calibrated on the target layer. The bounding box is delineated according to the boundary of the area to be calibrated. The calibrated area covers the monitoring range of the entire highway service area.

[0021] A sparse grid point set is generated within the bounding box according to the set step size, where the step size is the spacing between grid points and is set according to the calibration accuracy requirements.

[0022] Each grid point in the generated sparse grid point set is used as a control point. Each control point corresponds to a specific geographical location on the target layer. The control points establish a mapping relationship between the target layer and the source layer to obtain a gridded calibration point set.

[0023] Optionally, step three specifically includes:

[0024] Based on the gridded calibration point set, the position of each grid point in the target layer is determined as the initial coordinates for reference during the calibration process;

[0025] For each grid point, the coordinates of the grid point in the source image are determined by calculating its corresponding position in the source layer. The coordinate calculation is based on the preliminary alignment results between the target layer and the source layer.

[0026] After obtaining the corresponding coordinates of the grid points in the source image, the displacement of each grid vertex is calculated to generate the grid vertex displacement result. The displacement is the difference between the coordinates in the source image and the coordinates of the corresponding grid point in the target layer.

[0027] Optionally, step four specifically involves:

[0028] The calibration parameters are obtained by collecting the grid vertex displacement results, the control point pairs of each grid point, and the view size information. The control point pairs include the target layer coordinates and the corresponding coordinates of the source layer. The view size information is used to define the image display area and coordinate range.

[0029] The calibration parameters are encapsulated and verified to obtain a calibration parameter data package.

[0030] Optionally, step five specifically includes:

[0031] Extract known control point pairs from the calibration parameter data package;

[0032] Based on the extracted control point pairs and their corresponding coordinate relationships, the coordinates of each control point are solved using the least squares method to obtain the weight coefficients and affine coefficients of each control point, thus obtaining the mapping parameters of the TPS mapping model.

[0033] Using the calculated weight coefficients and affine coefficients, a non-linear mapping relationship is established between the target layer and the source layer until the coordinates of the target layer and the source layer undergo a one-to-one spatial transformation and mapping under the TPS mapping model.

[0034] By solving the inverse mapping relationship of the original source layer coordinates, the inverse perspective transformation mapping from the source image to the target image is obtained, forming a complete TPS mapping model.

[0035] Optionally, step six specifically includes:

[0036] Based on the TPS mapping model, the coordinates of each pixel in the target layer are traversed to calculate the corresponding coordinates of the current pixel in the source layer.

[0037] When generating the mapping relationship from target layer pixels to source layer pixels, for each target layer pixel, the corresponding pixel coordinates of the source layer are solved by the weight coefficients and affine coefficients of the TPS model, and each pixel coordinate in the target layer is mapped to the corresponding position in the source layer.

[0038] The calculated pixel coordinates of each target layer and its corresponding source layer coordinates are recorded in a mapping lookup table. Each entry in the mapping lookup table includes the mapping relationship between the pixel coordinates of the target layer and the pixel coordinates of the source layer.

[0039] Optionally, step seven specifically includes:

[0040] Access the video stream and obtain the current frame image from the video stream to perform inverse perspective transformation. The steps of performing inverse perspective transformation include:

[0041] The video stream is a real-time monitoring video stream, and each frame of the video image contains multiple pixels, and these pixels correspond to the coordinates of the target layer.

[0042] Based on the mapping lookup table, find the corresponding coordinates in the source layer for each pixel coordinate in the current video frame;

[0043] Transform each pixel coordinate in the target layer to the corresponding position in the source layer;

[0044] By performing an inverse perspective transformation on each pixel in the current video frame, a transformed video frame is generated, and coordinate mapping is performed from the source layer to the target layer.

[0045] All transformed video frames are combined to form an inverse perspective-mapped video stream.

[0046] Optionally, step eight specifically includes:

[0047] Access inverse perspective mapping video streams from multiple cameras or multiple perspectives, and calculate the overlapping area between each inverse perspective mapping video stream by analyzing the geospatial range of each inverse perspective mapping video stream.

[0048] The overlapping area refers to the part where the image content in different inverse perspective mapping video streams intersects in the actual geographic space;

[0049] For overlapping areas, a preset weighted fusion method is used to stitch them together, eliminating the seams between different inverse perspective mapping video streams and generating a seamless panoramic video stream.

[0050] A high-speed service area video inverse perspective mapping system for digital twins according to an embodiment of the present invention includes the following modules:

[0051] The layer alignment module is used to load orthophotos or CAD plans with geographic coordinate information as the target layer, load the source image of the highway service area monitoring video to be calibrated as the source layer, and perform geographic information alignment to obtain the initial layer alignment result.

[0052] The mesh calibration module is used to define the bounding box of the region to be calibrated on the target layer and generate sparse mesh points according to the set step size to obtain the mesh calibration point set.

[0053] The mesh vertex displacement calculation module is used to calculate the corresponding coordinates of each mesh vertex in the source layer based on the meshed calibration point set, perform displacement calculation, and generate mesh vertex displacement results.

[0054] The calibration parameter calculation module is used to encapsulate the mesh vertex displacement results, control point pairs, and view dimension information to obtain a calibration parameter data package;

[0055] The mapping model construction module is used to calculate weight coefficients and affine coefficients based on the calibration parameter data package, using known control point pairs and corresponding coordinate relationships, to establish a non-linear mapping relationship between the target layer and the source layer, and obtain the TPS mapping model.

[0056] The mapping lookup table building module is used to calculate the coordinates of each pixel coordinate in the target layer in the source layer based on the TPS mapping model, and generate a mapping lookup table;

[0057] The inverse perspective transformation module is used to access the video stream to obtain the current video frame, and to perform inverse perspective transformation on each pixel by looking up the mapping lookup table to generate an inverse perspective mapped video stream.

[0058] The stitching elimination module receives inverse perspective mapped video streams after multi-channel inverse perspective mapping processing, calculates overlapping areas, and uses weighted fusion to eliminate stitching seams at the joints, generating a seamless panoramic video stream.

[0059] The beneficial effects of this invention are:

[0060] This invention proposes a video inverse perspective mapping method for high-speed service areas for digital twins. Building upon existing technologies, it overcomes the limitations of traditional methods and achieves significant beneficial effects. Firstly, this invention solves the optical distortion problem caused by wide-angle or fisheye lenses in existing technologies through a nonlinear mapping technique based on the TPS algorithm. This effectively eliminates errors caused by camera distortion, ensuring accurate alignment between video images and actual geographic information. Traditional perspective transformation methods cannot handle these complex distortions, resulting in large image alignment errors. This invention, however, accurately fits the nonlinear transformation using a TPS model, adapting to video image processing in complex scenarios, especially in the variable terrain environment of high-speed service areas, providing high-precision inverse perspective mapping. Simultaneously, the gridded calibration method and dynamic calibration parameter transmission proposed in this invention overcome the difficulties of traditional calibration methods. In traditional methods, the calibration board needs to be placed in the camera's field of view for precise calibration. However, in practical applications, especially for cameras installed at high altitudes, traditional checkerboard calibration cannot be used, and existing calibration processes are complex and time-consuming. This invention performs gridded calibration based on control points of the target and source layers, which not only avoids the cumbersome calibration process but also ensures calibration accuracy through a simplified calibration procedure, greatly improving calibration efficiency and adapting to practical application needs in the field. Furthermore, the data encapsulation and transmission methods improve data fluency and stability during processing, enabling the calibration process to be performed efficiently and in real-time.

[0061] Another significant advantage of this invention lies in the Mapping Lookup Table (LUT) method, which greatly improves the processing efficiency of video streams. Unlike traditional methods that calculate frame by frame, this invention stores the mapping relationship between the target layer and the source layer by pre-generating a LUT, avoiding complex calculations for each frame, significantly reducing computational load, and ensuring low-latency processing of real-time video streams. This optimization enables the system to support parallel processing of multiple video streams, making it suitable for large-scale monitoring scenarios, such as panoramic video monitoring of highway service areas. Finally, combined with a weighted fusion algorithm, it effectively handles the stitching problem between multiple video streams. Traditional multi-video stream stitching often results in seams in overlapping areas, affecting the quality of panoramic video. However, this invention, through a weighted fusion algorithm, eliminates these seams, generating a seamless panoramic video stream, resulting in a smoother, more natural final output video image and providing a high-quality visual experience. In summary, this invention not only achieves new heights in accuracy, efficiency, and real-time performance but also overcomes many challenges in existing technologies, providing a more reliable and efficient solution for monitoring complex scenarios such as highway service areas. Attached Figure Description

[0062] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0063] Figure 1 This is an overall flowchart of the high-speed service area video inverse perspective mapping method for digital twins proposed in this invention;

[0064] Figure 2 This is a schematic diagram of the structure of the high-speed service area video inverse perspective mapping system for digital twins proposed in this invention. Detailed Implementation

[0065] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0066] refer to Figure 1 A method for inverse perspective mapping of high-speed service area videos for digital twins, including:

[0067] Step 1: Load the orthophoto or CAD plan with geographic coordinate information as the target layer, and load the source image of the highway service area monitoring video to be calibrated as the source layer, and perform geographic information alignment to obtain the initial layer alignment result;

[0068] Step 2: Define the bounding box of the area to be calibrated on the target layer based on the initial layer alignment result, and generate sparse grid points according to the set step size to obtain the gridded calibration point set;

[0069] Step 3: Based on the set of meshed calibration points, calculate the corresponding coordinates of each mesh vertex in the source layer, perform displacement calculations, and generate the mesh vertex displacement results;

[0070] Step 4: Encapsulate the mesh vertex displacement results, control point pairs, and view dimension information to obtain the calibration parameter data package;

[0071] Step 5: Based on the calibration parameter data package, calculate the weight coefficients and affine coefficients using the known control point pairs and their corresponding coordinate relationships, establish a nonlinear mapping relationship between the target layer and the source layer, and obtain the TPS mapping model;

[0072] Step 6: Calculate the coordinates of each pixel in the target layer in the source layer based on the TPS mapping model, and generate a mapping lookup table;

[0073] Step 7: Access the video stream to obtain the current video frame, and perform inverse perspective transformation on each pixel by looking up the mapping lookup table to generate an inverse perspective mapped video stream;

[0074] Step 8: Receive the inverse perspective mapping video stream after multi-channel inverse perspective mapping processing, calculate the overlapping area, use weighted fusion to eliminate the stitching seams at the joints, and generate a seamless panoramic video stream.

[0075] In this embodiment, step one specifically includes:

[0076] An orthophoto or CAD plan with geographic coordinate information is loaded as the target layer. This target layer serves as the base layer in the Geographic Information System (GIS) and contains detailed geospatial data of the highway service area. Specifically, the target layer includes, but is not limited to, topographic elevation data, road networks, building locations, locations of landmarks such as parking areas and gas stations, and detailed information on traffic facilities such as lanes and intersections within the service area. The loaded orthophoto or CAD plan must have a resolution of 1:5000 to ensure accurate spatial positioning, with the deviation between geographic coordinates and actual spatial location not exceeding 2 meters.

[0077] The source image of the highway service area surveillance video to be calibrated is loaded as the source layer. The source layer is a single frame image of the actual surveillance video of the highway service area, containing real-time traffic conditions, vehicle driving information, road markings, traffic signs, and other monitoring information within the highway service area. The loaded source layer image is a high-resolution video frame, with an image resolution of no less than 1920×1080 pixels, ensuring that the details of the monitoring screen are clearly visible.

[0078] Set the source layer to semi-transparent with an alpha value of 0.6. This allows the source and target layers to be displayed simultaneously and visually contrasted through layer overlay. The 0.6 alpha value ensures sufficient contrast between the source and target layers for precise alignment without compromising the readability of the target layer's geographic information.

[0079] Within the same view, the position, rotation angle, and scaling of the source layer are adjusted manually until the features in the surveillance video align with the geographic information in the target layer, generating an initial layer alignment result. Specifically, the rotation angle is adjusted within ±10 degrees, and the scaling range is 50% to 200% to accurately align the geographic information between the video source image and the target layer. During the adjustment process, the system displays the alignment effect between the source and target layers in real time until the roads, buildings, parking lots, and other facilities in the source and target layers are precisely aligned, achieving the desired initial layer alignment result.

[0080] In this embodiment, step two specifically includes:

[0081] Define a bounding box for the area to be calibrated on the target layer. The bounding box is delineated based on the boundaries of the area to be calibrated, and the calibration area covers the entire monitoring range of the highway service area. Specifically, the bounding box should cover all important monitoring areas within the highway service area, such as entrances / exits, parking areas, and gas stations. The size of the bounding box should ensure coverage of an area of ​​at least 500 meters × 500 meters to accommodate highway service areas of different sizes. The boundaries of the bounding box should be precisely defined based on the actual site conditions of the specific service area to ensure that the calibration process comprehensively covers the required area.

[0082] A sparse grid set is generated within the bounding box based on a set step size, where the step size is the spacing between grid points. The value of the step size should be set according to the calibration accuracy requirements. Generally, a step size of 20cm to 50cm is set, which is suitable for the calibration needs of most highway service areas. For scenarios with higher calibration accuracy requirements, the step size can be set to 10cm to increase the density of grid points and ensure calibration accuracy. By setting a reasonable step size, the number of grid points generated can be controlled to balance accuracy and computational resource consumption.

[0083] Each grid point in the generated sparse grid set serves as a control point, corresponding to a specific geographic location on the target layer. A mapping relationship is established between the control points in the target and source layers, resulting in a gridded calibration point set. Each control point corresponds to a precise geographic location on the target layer and is matched one-to-one with feature points in the surveillance video footage of the source layer, ensuring high-precision registration between video images and geographic information. This mapping relationship provides the foundational data for subsequent nonlinear transformation calculations, ultimately generating the gridded calibration point set. The mapping relationship of each grid point provides precise data support for displacement calculations in subsequent steps, ensuring the accuracy of the calibration process.

[0084] In this embodiment, step three specifically includes:

[0085] Based on a gridded calibration point set, the position of each grid point in the target layer is determined, serving as the initial reference coordinates during the calibration process. Specifically, each grid point in the grid set corresponds one-to-one with geographic coordinates in the target layer. Through manual adjustment or automatic identification on the target layer, it is ensured that the coordinates of each grid point accurately reflect the geographic location of the target area. These grid point coordinates will serve as reference coordinates in subsequent calculations, ensuring the alignment consistency of all points during the calibration process.

[0086] For each grid point, its coordinates in the source image are determined by calculating its corresponding position in the source layer. This coordinate calculation is based on the preliminary alignment results between the target and source layers. The key to this step is effectively matching the surveillance video images in the source layer with the geographic information in the target layer. Using the alignment of the source and target layers completed in the previous steps, the accurate corresponding position of each grid point in the target layer in the source layer is calculated. This process utilizes image registration techniques or the mapping relationship of known control point pairs to ensure the mapping accuracy between the source and target layers.

[0087] After obtaining the corresponding coordinates of the grid points in the source image, the displacement of each grid vertex is calculated to generate the grid vertex displacement result. This displacement is the difference between the coordinates in the source image and the coordinates of the corresponding grid point in the target layer; that is, the displacement is determined by calculating the deviation between the actual coordinates of each grid point in the source layer and the preset coordinates in the target layer. This displacement reflects the geometric deformation of each point in the source layer relative to the target layer, effectively representing the spatial difference between the two layers. Through this calculation method, the final generated grid vertex displacement result provides a high-precision mapping relationship for the input data of the subsequent transformation model, ensuring that the subsequent image deformation process can accurately and realistically restore the original scene.

[0088] In this embodiment, step four specifically includes:

[0089] The calibration parameters are obtained by collecting the grid vertex displacement results, control point pairs for each grid point, and view dimension information. Specifically, the grid vertex displacement results are the coordinate differences between the source and target layers, reflecting the spatial transformation relationship between the source and target layers. The control point pairs for each grid point, including the target layer coordinates and the corresponding coordinates of the source layer, are the core data in the calibration process, ensuring accurate registration between the target and source layers. The collection of control point pairs requires high alignment accuracy; typically, the error of the control points should not exceed 2 pixels to ensure calibration precision. View dimension information includes the specific range of the display area, the range of the coordinate system, and the image resolution, used to define the image's display area and coordinate range. This view dimension information ensures error-free image adaptation and display in subsequent steps, especially when calibrating images at different resolutions, ensuring the universality and adaptability of the calibration process.

[0090] The calibration parameters are encapsulated and verified to obtain a calibration parameter data packet. During encapsulation, all calibration parameters, including mesh vertex displacement results, control point pairs, and view dimension information, are integrated into a single data packet, which forms the basis for subsequent processing. The calibration parameter data packet is encapsulated using a standardized format to ensure data integrity and consistency. The encapsulated data packet undergoes verification during transmission to ensure data accuracy and avoid calibration errors caused by data loss or corruption. The verification process uses a hash algorithm to check the data packet, ensuring that its content has not been altered. If the verification passes, the data packet is successfully transmitted to the next processing module for subsequent mapping calculations and processing.

[0091] In this embodiment, step five specifically includes:

[0092] Known control point pairs are extracted from the calibration parameter data package. Specifically, the control point pairs in the calibration parameter data package are the data set of mesh vertex displacement results and control point pairs obtained in step three above. Each control point pair includes a one-to-one correspondence between the target layer and the source layer. These control points are crucial benchmarks in the calibration process, used to establish the spatial mapping relationship between the source and target layers. The extraction process for each control point pair must ensure accuracy; typically, the control point error should not exceed 1 pixel to ensure high precision requirements in subsequent calculations.

[0093] Based on the extracted control point pairs and their corresponding coordinate relationships, the coordinates of each control point are solved using the least squares method to obtain the weight coefficients and affine coefficients of each control point, thus yielding the mapping parameters of the TPS mapping model. Specifically, the least squares method is used to solve an optimization problem: minimizing the error between the transformed target layer and the source layer based on the control point pairs and their corresponding coordinate relationships. By minimizing the error, the weight coefficients and affine coefficients of the control points are obtained. These coefficients are key parameters of the TPS mapping model, reflecting the spatial transformation relationship from the target layer to the source layer. Typically, in practice, the calculation error of the weight coefficients and affine coefficients during the least squares method calculation is guaranteed to be less than 0.5 pixels to ensure the accuracy of the model.

[0094] By utilizing the calculated weighting and affine coefficients, a non-linear mapping relationship is established between the target layer and the source layer until the coordinates of the target and source layers undergo a one-to-one spatial transformation and mapping under the TPS mapping model. In this process, the combined weighting and affine coefficients enable the generated TPS mapping model to perform precise coordinate transformations between the target and source layers. This non-linear mapping relationship is established based on the coordinate transformation of each control point, ensuring a high degree of consistency in the transformed coordinates and accurately mapping the coordinates of the target layer to the source layer.

[0095] By solving the inverse mapping relationship of the original source layer coordinates, the inverse perspective transformation mapping from the source image to the target image is obtained, forming a complete TPS mapping model. In this step, the inverse perspective transformation relationship from the source layer to the target layer is obtained through inverse mapping. The inverse mapping process ensures that the pixel coordinates of each target layer can be accurately transformed back to the corresponding coordinates of the source layer through the TPS model, thus forming a complete TPS mapping model. This model is not only used for coordinate mapping from the target layer to the source layer, but also for inverse mapping from the source layer to the target layer, ensuring the comprehensiveness and accuracy of image transformation.

[0096] In this embodiment, step six specifically includes:

[0097] Based on the TPS mapping model, the coordinates of each pixel in the target layer are traversed to calculate the corresponding coordinates of the current pixel in the source layer. Specifically, at each pixel position in the target layer, the weight coefficients and affine coefficients calculated in the TPS mapping model are used to calculate the corresponding coordinates of the pixel in the source layer based on the non-linear mapping relationship between the source and target layers. This step requires iterative calculation for each pixel in the target layer to ensure accurate mapping for each pixel. The calculation process considers the spatial deformation of each pixel in the target layer to ensure that the mapping relationship between the source and target layers is globally consistent and can accurately reflect the actual geometric deformation in space.

[0098] When generating the mapping relationship from target layer pixels to source layer pixels, for each target layer pixel, the corresponding pixel coordinates in the source layer are calculated using the weight coefficients and affine coefficients of the TPS model. For the target image, the mapping function of the TPS model, using the previously calculated weight coefficients and affine coefficients, transforms the coordinates of each pixel in the target layer to the corresponding position in the source layer. This process, based on the non-linear mapping characteristics of the TPS model, transforms the coordinates of the target layer to ensure that each pixel in the source layer accurately matches the pixel in the target layer. This step generates the corresponding source layer coordinates for each frame pixel in the video stream and is the foundation of the inverse perspective transformation of the video stream.

[0099] The calculated pixel coordinates of each target layer and its corresponding source layer coordinates are recorded in a mapping lookup table. Each entry in the mapping lookup table includes the mapping relationship between the target layer pixel coordinates and the source layer pixel coordinates. The generated mapping lookup table (LUT) stores the mapping relationship between each target layer pixel and source layer pixel, providing a corresponding source layer coordinate for each target layer pixel coordinate. The storage method of the LUT must ensure fast reading and access so that the correspondence between the target layer and the source layer can be efficiently found during the subsequent inverse perspective mapping process of the video stream. The mapping relationship in each entry contains the correspondence between the target layer pixel coordinates and the source layer coordinates. The generation of the LUT ensures the efficiency and accuracy of subsequent image processing.

[0100] In this embodiment, step seven specifically includes:

[0101] Access the video stream and obtain the current frame image from the video stream to perform inverse perspective transformation. The steps of performing inverse perspective transformation include:

[0102] The video stream is a real-time monitoring video stream. Each frame of the video image contains multiple pixels, and these pixels correspond to the coordinates of the target layer. Specifically, the accessed video stream is a real-time video stream from a surveillance camera in a highway service area. Each frame of the video stream consists of millions of pixels, and each pixel has corresponding coordinates in the target layer. The resolution of the video image is generally no less than 1920×1080 pixels to ensure that the details of the monitored image are clearly visible, and each pixel corresponds to specific geographic coordinates in the target layer. The coordinates of these pixels need to be transformed from the target layer coordinates to the source layer coordinates through inverse perspective transformation.

[0103] Based on the mapping lookup table, the corresponding coordinates in the source layer are found for each pixel coordinate in the current video frame. The mapping lookup table (LUT) is a lookup table established based on the mapping relationship between the target layer and the source layer generated in step six. For each pixel coordinate in the current video frame, the corresponding coordinates of the target layer pixel in the source layer can be found by looking up the LUT table. This process allows each pixel in the target layer to quickly obtain its corresponding pixel coordinates in the source layer through the lookup table, avoiding redundant calculations and improving processing efficiency. Each pixel coordinate in the target layer is then transformed to its corresponding position in the source layer. After obtaining the corresponding coordinates of the target layer pixel coordinates and the source layer, an inverse perspective transformation is used to accurately map each pixel coordinate in the target layer to its corresponding position in the source layer. This transformation process is based on the aforementioned LUT mapping lookup table, ensuring that the position of each pixel in the source layer is accurate. The inverse perspective transformation takes into account the geometric deformation between the source and target layers, ensuring that the mapping relationship of each pixel is consistent and conforms to the actual geographical location.

[0104] By performing an inverse perspective transformation on each pixel in the current video frame, a transformed video frame is generated, and coordinate mapping from the source layer to the target layer is performed. For each pixel in the video frame, its position in the source layer is calculated sequentially through the inverse perspective transformation, thus generating the transformed video frame. Pixels in each video frame are mapped to ensure that the content in the video image accurately reflects the spatial relationships of the source layer. This step ensures the consistency and accuracy of the coordinate transformation from the source layer to the target layer in every video frame.

[0105] All transformed video frames are aggregated to form an inverse perspective mapping video stream. By performing an inverse perspective transformation on each video frame, all the generated transformed video frames are aggregated to form a complete inverse perspective mapping video stream. After inverse perspective transformation, this video stream not only maintains spatial consistency between the source and target layers but also provides a high-quality view, meeting the real-time video stream processing needs of applications such as intelligent monitoring and digital twins.

[0106] In this embodiment, step eight specifically includes:

[0107] This system receives inverse perspective video streams from multiple cameras or viewpoints and analyzes the geospatial extent of each stream to calculate the overlapping area. Specifically, the inverse perspective video streams from multiple cameras or viewpoints cover different areas within the highway service area. By analyzing the geospatial information of each video stream, the spatial overlap between them is determined. The overlapping area between video streams is typically formed by the overlap of adjacent camera viewpoints or monitored areas. This overlapping area can be obtained by comparing and calculating the spatial coordinates of the image content in the video streams. To ensure the accuracy of the overlapping area, the calculation is usually based on the actual location in the geographic coordinate system, avoiding errors caused by different camera angles or irregular monitoring areas.

[0108] The overlapping region refers to the portion of image content from different inverse perspective video streams that intersect in actual geographic space. Specifically, it refers to the portion of the surveillance content captured by multiple cameras or viewpoints in different inverse perspective video streams that overlaps in actual geographic space. This region is determined through geographic alignment between cameras and the spatial intersection of image content. Within this region, the surveillance data provided by multiple cameras will share the same geographic information, forming an intersection area. By calculating these overlapping regions, it is possible to determine which video streams share spatial commonalities, providing a reference for subsequent stitching operations.

[0109] For overlapping areas, a pre-defined weighted fusion algorithm is used to stitch the images together, eliminating seams between different inverse perspective mapping video streams and generating a seamless panoramic video stream. Specifically, for overlapping areas, a weighted fusion algorithm is used to stitch together the image content of multiple video streams. The weighted fusion algorithm calculates a weighted average of the images based on the weight value of each video stream within the overlapping area, smoothly transitioning the image content between different video streams and eliminating potential seams or gaps at the stitching points. The weighting values ​​can be adjusted based on the image quality of the cameras, the degree of overlap in viewing angles, and other factors. Finally, through weighted fusion processing, a seamless panoramic video stream is generated, enabling seamless integration of monitoring data from different perspectives and providing users with a continuous and high-quality view. The seamless panoramic video stream provides an efficient visual display for real-time monitoring and digital twin systems in high-speed service areas, ensuring that users can clearly and without interference view the real-time status of the entire area.

[0110] refer to Figure 2 A high-speed service area video inverse perspective mapping system for digital twins includes the following modules:

[0111] The layer alignment module is used to load orthophotos or CAD plans with geographic coordinate information as the target layer, load the source image of the highway service area monitoring video to be calibrated as the source layer, and perform geographic information alignment to obtain the initial layer alignment result.

[0112] The mesh calibration module is used to define the bounding box of the region to be calibrated on the target layer and generate sparse mesh points according to the set step size to obtain the mesh calibration point set.

[0113] The mesh vertex displacement calculation module is used to calculate the corresponding coordinates of each mesh vertex in the source layer based on the meshed calibration point set, perform displacement calculation, and generate mesh vertex displacement results.

[0114] The calibration parameter calculation module is used to encapsulate the mesh vertex displacement results, control point pairs, and view dimension information to obtain a calibration parameter data package;

[0115] The mapping model construction module is used to calculate weight coefficients and affine coefficients based on the calibration parameter data package, using known control point pairs and corresponding coordinate relationships, to establish a non-linear mapping relationship between the target layer and the source layer, and obtain the TPS mapping model.

[0116] The mapping lookup table building module is used to calculate the coordinates of each pixel coordinate in the target layer in the source layer based on the TPS mapping model, and generate a mapping lookup table;

[0117] The inverse perspective transformation module is used to access the video stream to obtain the current video frame, and to perform inverse perspective transformation on each pixel by looking up the mapping lookup table to generate an inverse perspective mapped video stream.

[0118] The stitching elimination module receives inverse perspective mapped video streams after multi-channel inverse perspective mapping processing, calculates overlapping areas, and uses weighted fusion to eliminate stitching seams at the joints, generating a seamless panoramic video stream.

[0119] Example 1:

[0120] To verify the feasibility of this invention in practice, it was applied to a highway service area as the application scenario. With the continuous increase in traffic volume, traditional video surveillance systems can no longer meet the demand for real-time monitoring of various areas within the service area. This service area is a multi-functional parking and rest area, including multiple restaurants, restrooms, rest areas, gas stations, and other facilities. The frequent changes in the positions of vehicles, pedestrians, and goods within the service area pose significant challenges to safety management and operational scheduling. Traditional camera surveillance videos can mostly only provide fixed-point monitoring of a specific area, and due to limitations in viewing angle and space, they cannot provide complete real-time data of the service area, especially in cases of congestion or special circumstances, making it difficult to respond and handle situations quickly.

[0121] By using video inverse perspective mapping and combining it with digital twin technology, the spatial mapping relationship between the target layer and the source layer is utilized to transform multiple surveillance video feeds into a unified digital virtual space that can accurately reflect all activity areas in the entire service area, thereby providing managers with comprehensive real-time monitoring, scheduling, and emergency response support.

[0122] Surveillance cameras installed throughout the highway service area capture video streams in real time. Using an inverse perspective transformation algorithm, the cameras generate a target layer that matches the service area's ground layout. Digital twin technology is then used to create and update a virtual map of the service area in real time, precisely locating vehicles, pedestrians, and other information from the video images. During this process, based on a TPS mapping model, each pixel in the video undergoes an inverse perspective transformation according to its corresponding coordinates in the source layer, accurately restoring moving objects from the video feed to the virtual space. Through this mapping relationship, all video image data is converted and integrated in real time into a high-precision dynamic digital twin space, allowing managers to view the entire service area in real time and perform dynamic monitoring.

[0123] In this process, especially when congestion occurs in service areas or special events (such as accidents or fires), managers can immediately view the situation through the mapped virtual space and obtain detailed data on the affected area, such as traffic flow, pedestrian density, and parking conditions. This is crucial for emergency response and dispatch decisions. For example, when a sudden queue congestion occurs in a food court, the system can automatically identify the congested area and display relevant information on a map, helping managers quickly deploy manpower to intervene and avoid long wait times. In practical applications, we have demonstrated the beneficial effects of this invention through test data. When congestion occurred in a certain area of ​​the service area, traditional monitoring systems failed to identify the problem in a timely manner and failed to provide real-time monitoring data to managers, resulting in queue times exceeding 10 minutes. However, the system based on this invention can capture and report congestion in real time. After detecting congestion, managers can use the system to control and dispatch staff to manage the flow of people, reducing queue times to 3 minutes and significantly improving customer satisfaction and service efficiency.

[0124] Table 1. Comparison of Monitoring and Dispatch Performance in Highway Service Areas

[0125] Comparison Dimensions Local monitoring method based on a single camera Manual scheduling method based on human monitoring and observation Regional Scheduling Method Based on Fixed-Area Video Surveillance Method of the present invention Surveillance perspective Single perspective, local area No real-time monitoring Fixed-point camera surveillance cannot fully cover all areas. Multi-angle video stream, fully covering all areas of the service area. Monitoring response time 10 minutes 20 minutes 15 minutes 3 minutes Event recognition accuracy 60% 59% 70% 95% Queue time 10 minutes ( 15 minutes 12 minutes (3 minutes should)

[0126] As shown in Table 1, local monitoring methods based on a single camera typically only monitor a specific area, resulting in a limited monitoring perspective and an inability to comprehensively understand the real-time status of the service area. This method has a long response time, and due to the limited perspective, the accuracy of event recognition is low, leading to a failure to promptly detect and address problems, especially during peak hours when customer queues are long and customer satisfaction is low. Manual dispatching methods based on human monitoring and observation rely entirely on human experience, resulting in slow response times and difficulty in timely problem detection in emergencies. Manual monitoring lacks efficient automated systems, leading to long queues and low customer satisfaction due to slow crowd control. Area dispatching methods based on fixed-area video monitoring can cover a larger area, but due to the limited perspective of each camera, it still cannot cover all activity areas, and the accuracy of event recognition is also low. Management efficiency is limited, and it remains difficult to respond quickly and effectively under high traffic or congestion conditions. The method of this invention, by introducing digital twin technology and inverse perspective mapping, uniformly maps and integrates various video data of the service area into a virtual space in real time, enabling real-time feedback and rapid response, significantly reducing customer queue times and greatly improving management efficiency. The system's flexibility and intelligence enable managers to adjust service area operations promptly and accurately, significantly improving customer satisfaction. Compared to traditional methods, the method of this invention has significant advantages in several aspects, solving problems such as monitoring blind spots, delayed feedback, slow response, and low customer satisfaction in existing methods, thereby improving the overall operational efficiency of the service area and the customer experience.

[0127] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for inverse perspective mapping of high-speed service area video for digital twins, characterized in that, include: Step 1: Load the orthophoto or CAD plan with geographic coordinate information as the target layer, and load the source image of the highway service area monitoring video to be calibrated as the source layer, and perform geographic information alignment to obtain the initial layer alignment result; Step 2: Define the bounding box of the area to be calibrated on the target layer based on the initial layer alignment result, and generate sparse grid points according to the set step size to obtain the gridded calibration point set; Step 3: Based on the set of meshed calibration points, calculate the corresponding coordinates of each mesh vertex in the source layer, perform displacement calculations, and generate the mesh vertex displacement results; Step 4: Encapsulate the mesh vertex displacement results, control point pairs, and view dimension information to obtain the calibration parameter data package; Step 5: Based on the calibration parameter data package, calculate the weight coefficients and affine coefficients using the known control point pairs and their corresponding coordinate relationships, establish a nonlinear mapping relationship between the target layer and the source layer, and obtain the TPS mapping model; Step 6: Calculate the coordinates of each pixel in the target layer in the source layer based on the TPS mapping model, and generate a mapping lookup table; Step 7: Access the video stream to obtain the current video frame, and perform inverse perspective transformation on each pixel by looking up the mapping lookup table to generate an inverse perspective mapped video stream; Step 8: Receive the inverse perspective mapping video stream after multi-channel inverse perspective mapping processing, calculate the overlapping area, use weighted fusion to eliminate the stitching seams at the joints, and generate a seamless panoramic video stream.

2. The high-speed service area video inverse perspective mapping method for digital twins according to claim 1, characterized in that, Step one specifically involves: An orthophoto or CAD plan with geographic coordinate information is loaded as the target layer. This target layer serves as the reference layer in the geographic information system and includes geospatial data of highway service areas. Load the source image of the highway service area monitoring video to be calibrated as the source layer. The source layer is a single frame image of the actual monitoring video of the highway service area, including the real-time traffic conditions and monitoring information in the highway service area. Set the source layer to semi-transparent until the source and target layers can be displayed simultaneously and visually contrasted by layer overlay. In the same view, the position, rotation angle and scaling of the source layer are adjusted manually until the features in the monitoring video are aligned with the geographic information in the target layer, generating the initial layer alignment result.

3. The method for inverse perspective mapping of high-speed service area video based on digital twins according to claim 1, characterized in that, Step two specifically involves: Define a bounding box for the area to be calibrated on the target layer. The bounding box is delineated according to the boundary of the area to be calibrated. The calibrated area covers the monitoring range of the entire highway service area. A sparse grid point set is generated within the bounding box according to the set step size, where the step size is the spacing between grid points and is set according to the calibration accuracy requirements. Each grid point in the generated sparse grid point set is used as a control point. Each control point corresponds to a specific geographical location on the target layer. The control points establish a mapping relationship between the target layer and the source layer to obtain a gridded calibration point set.

4. The method for inverse perspective mapping of high-speed service area video based on digital twins according to claim 1, characterized in that, Step three specifically involves: Based on the gridded calibration point set, the position of each grid point in the target layer is determined as the initial coordinates for reference during the calibration process; For each grid point, the coordinates of the grid point in the source image are determined by calculating its corresponding position in the source layer. The coordinate calculation is based on the preliminary alignment results between the target layer and the source layer. After obtaining the corresponding coordinates of the grid points in the source image, the displacement of each grid vertex is calculated to generate the grid vertex displacement result. The displacement is the difference between the coordinates in the source image and the coordinates of the corresponding grid point in the target layer.

5. The method for inverse perspective mapping of high-speed service area video based on digital twins according to claim 1, characterized in that, Step four specifically involves: The calibration parameters are obtained by collecting the grid vertex displacement results, the control point pairs of each grid point, and the view size information. The control point pairs include the target layer coordinates and the corresponding coordinates of the source layer. The view size information is used to define the image display area and coordinate range. The calibration parameters are encapsulated and verified to obtain a calibration parameter data package.

6. The method for inverse perspective mapping of high-speed service area video based on digital twins according to claim 1, characterized in that, Step five specifically involves: Extract known control point pairs from the calibration parameter data package; Based on the extracted control point pairs and their corresponding coordinate relationships, the coordinates of each control point are solved using the least squares method to obtain the weight coefficients and affine coefficients of each control point, thus obtaining the mapping parameters of the TPS mapping model. Using the calculated weight coefficients and affine coefficients, a non-linear mapping relationship is established between the target layer and the source layer until the coordinates of the target layer and the source layer undergo a one-to-one spatial transformation and mapping under the TPS mapping model. By solving the inverse mapping relationship of the original source layer coordinates, the inverse perspective transformation mapping from the source image to the target image is obtained, forming a complete TPS mapping model.

7. The method for inverse perspective mapping of high-speed service area video based on digital twins according to claim 1, characterized in that, Step six specifically involves: Based on the TPS mapping model, the coordinates of each pixel in the target layer are traversed to calculate the corresponding coordinates of the current pixel in the source layer. When generating the mapping relationship from target layer pixels to source layer pixels, for each target layer pixel, the corresponding pixel coordinates of the source layer are solved by the weight coefficients and affine coefficients of the TPS model, and each pixel coordinate in the target layer is mapped to the corresponding position in the source layer. The calculated pixel coordinates of each target layer and its corresponding source layer coordinates are recorded in a mapping lookup table. Each entry in the mapping lookup table includes the mapping relationship between the pixel coordinates of the target layer and the pixel coordinates of the source layer.

8. The method for inverse perspective mapping of high-speed service area video based on digital twins according to claim 1, characterized in that, Step seven specifically involves: Access the video stream and obtain the current frame image from the video stream to perform inverse perspective transformation. The steps of performing inverse perspective transformation include: The video stream is a real-time monitoring video stream, and each frame of the video image contains multiple pixels, and these pixels correspond to the coordinates of the target layer. Based on the mapping lookup table, find the corresponding coordinates in the source layer for each pixel coordinate in the current video frame; Transform each pixel coordinate in the target layer to the corresponding position in the source layer; By performing an inverse perspective transformation on each pixel in the current video frame, a transformed video frame is generated, and coordinate mapping is performed from the source layer to the target layer. All transformed video frames are combined to form an inverse perspective-mapped video stream.

9. The method for inverse perspective mapping of high-speed service area video based on digital twins according to claim 1, characterized in that, Step eight specifically involves: Access inverse perspective mapping video streams from multiple cameras or multiple perspectives, and calculate the overlapping area between each inverse perspective mapping video stream by analyzing the geospatial range of each inverse perspective mapping video stream. The overlapping area refers to the part where the image content in different inverse perspective mapping video streams intersects in the actual geographic space; For overlapping areas, a preset weighted fusion method is used to stitch them together, eliminating the seams between different inverse perspective mapping video streams and generating a seamless panoramic video stream.

10. A high-speed service area video inverse perspective mapping system for digital twins, comprising the high-speed service area video inverse perspective mapping method for digital twins as described in any one of claims 1 to 9, characterized in that, Includes the following modules: The layer alignment module is used to load orthophotos or CAD plans with geographic coordinate information as the target layer, load the source image of the highway service area monitoring video to be calibrated as the source layer, and perform geographic information alignment to obtain the initial layer alignment result. The mesh calibration module is used to define the bounding box of the region to be calibrated on the target layer and generate sparse mesh points according to the set step size to obtain the mesh calibration point set. The mesh vertex displacement calculation module is used to calculate the corresponding coordinates of each mesh vertex in the source layer based on the meshed calibration point set, perform displacement calculation, and generate mesh vertex displacement results. The calibration parameter calculation module is used to encapsulate the mesh vertex displacement results, control point pairs, and view dimension information to obtain a calibration parameter data package; The mapping model construction module is used to calculate weight coefficients and affine coefficients based on the calibration parameter data package, using known control point pairs and corresponding coordinate relationships, to establish a non-linear mapping relationship between the target layer and the source layer, and obtain the TPS mapping model. The mapping lookup table building module is used to calculate the coordinates of each pixel coordinate in the target layer in the source layer based on the TPS mapping model, and generate a mapping lookup table; The inverse perspective transformation module is used to access the video stream to obtain the current video frame, and to perform inverse perspective transformation on each pixel by looking up the mapping lookup table to generate an inverse perspective mapped video stream. The stitching elimination module receives inverse perspective mapped video streams after multi-channel inverse perspective mapping processing, calculates overlapping areas, and uses weighted fusion to eliminate stitching seams at the joints, generating a seamless panoramic video stream.

Images