A method, system and medium for video surveillance of valet parking

Abstract

This invention relates to a valet parking video monitoring method, system, and medium. The method includes: L1. Multi-source camera raw data acquisition: acquiring raw video data from a 120° front-view camera, a rear-view camera, and four surround-view cameras, and inputting the raw video data into the application module of the HAD controller; L2. Camera data differentiation processing: performing distortion correction processing on the raw data from the 120° front-view camera and the rear-view camera, outputting a corrected single-channel video stream, and outputting the undistorted raw frames from the raw data from the four surround-view cameras, which are then transmitted to the surround-view stitching module. This invention not only ensures that users can obtain clear and complete video images of the vehicle's surroundings in real time via a mobile APP, but also its modular design facilitates the addition of new camera types or adjustment of processing logic to adapt to the needs of different valet parking scenarios.

Description

Technical Field

[0001] This invention relates to the field of autonomous valet parking technology, and in particular to a valet parking video monitoring method, system and medium. Background Technology

[0002] Valet parking, as a typical scenario for advanced autonomous driving, enables vehicles to automatically park and exit within a designated area, providing users with a convenient parking experience. During valet parking, users typically use a mobile app to remotely view the vehicle's surroundings to confirm its parking status and safety. Therefore, real-time, clear video monitoring capabilities are crucial for ensuring user trust and driving safety.

[0003] In the prior art, Chinese patent (publication number: CN117459746A) discloses a monitoring method and device for autonomous learning parking, the monitoring method including the following steps: response During the autonomous parking learning process, the system receives a video opening request from the user terminal, acquires external video footage, and sends a request to the cloud to open a live stream. It then receives the corresponding live stream number generated by the cloud and, based on this number, streams the external video footage to that live stream. Simultaneously, the user terminal receives the live stream number and, based on it, pulls the external video footage from that live stream to its interface for display. However, this solution does not provide a view switching function for the external video monitoring; users cannot view the surroundings of the vehicle on the terminal interface, and cannot fully browse the vehicle's real-time driving environment.

[0004] In the prior art, Chinese patent (publication number: CN114710689A) discloses a method and system for real-time audio and video remote monitoring of vehicles, which sends remote control commands to the vehicle via a mobile terminal. When a vehicle is being remotely moved or used for valet parking, the system allows viewing real-time video feeds and switching between different monitoring angles, as well as adjusting the video resolution. However, this solution does not perform distortion correction on the video stream, resulting in poor video monitoring performance during valet parking.

[0005] The existing parking video surveillance solution has the following defects: 1) The multi-camera data processing logic is chaotic. The distortion correction process of the front-view, rear-view and surround-view cameras is not differentiated, resulting in wasted computing power or image distortion; 2) The surround-view image stitching and distortion correction process is disconnected, which easily leads to problems such as stitching gaps and image blurring, affecting the user's perception of the vehicle's surrounding environment. The low efficiency of integrating video streaming with multi-source video data makes it impossible to guarantee that the mobile app can obtain multiple clear video streams in real time, resulting in monitoring delays and stuttering. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the present invention provides a valet parking video monitoring method, system and medium, which not only ensures that users can obtain clear and complete video images of the vehicle's surroundings in real time through a mobile APP, but also facilitates the addition of camera types or adjustment of processing logic in the future to adapt to the needs of different valet parking scenarios.

[0007] To achieve the above and other related objectives, the present invention provides the following technical solution: A valet parking video surveillance method, the method comprising: L1. Acquisition of raw data from multiple cameras: Acquire raw video data from the 120° front-view camera, the rear-view camera, and the four-way surround-view camera, and input the raw video data into the application module of the HAD controller; L2. Camera data differentiation processing: Perform distortion correction processing on the raw data of the forward-looking 120° camera and the rear-looking camera, output the corrected single-channel video stream, and output the undistorted raw frames of the raw data of the four surround-view cameras, and transmit them to the surround-view stitching module. L3. Surround view image stitching and distortion correction: The surround view stitching module receives four undistorted surround view raw frames, performs surround view stitching operation to generate a complete surround view image of the vehicle's surroundings, and then performs stitching distortion correction processing on the stitched image to output the corrected surround view video stream. L4. Multi-channel video stream integration and streaming: The single-channel video streams after distortion correction of the front and rear views, as well as the surround view video stream after distortion correction of the surround view stitching, are uniformly input into the video streaming SDK. The video streaming SDK encapsulates and encodes the multiple video streams and executes video streaming to the mobile APP through the vehicle network channel for real-time monitoring of the valet parking process.

[0008] Furthermore, the method also includes: L5. Mobile Video Display: The mobile app receives multiple video streams, decodes and renders them, and displays real-time monitoring footage of the front, rear, and surround views to the user.

[0009] Furthermore, in step L3, the surround view stitching operation includes using an image registration algorithm based on multi-scale feature fusion and adaptive deformation field to perform image registration on the four undistorted surround view original frames.

[0010] Furthermore, the image registration algorithm based on multi-scale feature fusion and adaptive deformation field includes: L31. The input image is first subjected to intensity normalization to eliminate grayscale differences between different scanning devices; L32. Construct a four-layer Gaussian pyramid, with each layer having a resolution of one-half, one-quarter, one-eighth, and one-sixteenth of the original image, respectively. Extract multi-channel features in each layer, including gray-level gradient, Laplacian response, and local binary pattern features. Feature point matching adopts an improved nearest neighbor search algorithm, and spatial consistency constraints are introduced to eliminate mismatched points. L33. An iterative optimization strategy is adopted, with fifty iterations performed on each pyramid level. In each iteration, the similarity measure gradient under the current deformation field is calculated, and the deformation field is updated in combination with an adaptive regularization term. The regularization weight is dynamically adjusted according to the local deformation gradient. L34. After registration is completed, a topology check is performed on the deformation field to detect and correct any possible folds or tears, and bicubic interpolation is used to generate the final registered image.

[0011] Furthermore, in step L33, the iterative optimization strategy includes determining convergence and proceeding to the next resolution level when the deformation field change between two adjacent iterations is less than a preset threshold.

[0012] Furthermore, the detection and correction of possible folds or tears involves evaluating the registration quality by calculating overlap indices, which include the Dice coefficient, the Jaccard index, and the target registration error.

[0013] To achieve the above and other related objectives, the present invention also provides a valet parking video monitoring system for implementing the aforementioned valet parking video monitoring method, the system comprising: The multi-camera acquisition unit includes a 120° forward-looking camera, a rear-view camera, and four surround-view cameras, used to acquire raw video data from different perspectives around the vehicle. The HAD controller includes an application module for receiving raw data from multiple cameras, performing distortion correction on the front / rear view camera data, and outputting undistorted raw frames for the surround view camera data; a surround view stitching module for receiving four surround view raw frames and performing surround view stitching and distortion correction; and a video streaming SDK module for integrating the corrected video streams from the front, rear, and surround views and performing encoding and streaming operations.

[0014] Furthermore, the system also includes a mobile app for receiving video streaming data, decoding, rendering, and displaying real-time monitoring footage.

[0015] Furthermore, the multi-camera acquisition unit is connected to the application module, the application module is connected to the surround view video stitching module and the video streaming SDK module respectively, and the surround view video stitching module is connected to the video streaming SDK module.

[0016] To achieve the above and other related objectives, the present invention also provides a computer-readable storage medium storing a computer program programmed or configured to perform the valet parking video monitoring method described above.

[0017] The present invention has the following positive effects: 1. The differentiated processing of this invention improves efficiency: For different usage scenarios of front-view / rear-view cameras and surround-view cameras, a differentiated process of "direct distortion removal" and "stitching first and then distortion removal" is adopted to avoid redundant calculations and improve processing efficiency.

[0018] 2. The panoramic stitching quality of the present invention is higher: the process of stitching first and then removing distortion can effectively eliminate stitching gaps and distortion, ensuring the integrity and clarity of the panoramic image.

[0019] 3. The present invention has stronger streaming stability: By uniformly encapsulating multiple video streams through the video streaming SDK, it ensures that the mobile APP can obtain low-latency and high-smoothness monitoring images in real time, thereby improving user experience and driving safety.

[0020] 4. The architecture of this invention has good scalability: the modular design facilitates the addition of new camera types or adjustment of processing logic, adapting to the needs of different valet parking scenarios. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a flowchart illustrating the image registration algorithm based on multi-scale feature fusion and adaptive deformation field of the present invention. Figure 3 This is a schematic diagram of the system framework of the present invention. Detailed Implementation

[0022] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] Example 1: As Figure 1 As shown, a valet parking video monitoring method includes: L1. Acquisition of raw data from multiple cameras: Acquire raw video data from the 120° front-view camera, the rear-view camera, and the four-way surround-view camera, and input the raw video data into the application module of the HAD controller; L2. Camera data differentiation processing: Perform distortion correction processing on the raw data of the forward-looking 120° camera and the rear-looking camera, output the corrected single-channel video stream, and output the undistorted raw frames of the raw data of the four surround-view cameras, and transmit them to the surround-view stitching module. L3. Surround view image stitching and distortion correction: The surround view stitching module receives four undistorted surround view raw frames, performs surround view stitching operation to generate a complete surround view image of the vehicle's surroundings, and then performs stitching distortion correction processing on the stitched image to output the corrected surround view video stream. L4. Multi-channel video stream integration and streaming: The single-channel video streams after distortion correction of the front and rear views, as well as the surround view video stream after distortion correction of the surround view stitching, are uniformly input into the video streaming SDK. The video streaming SDK encapsulates and encodes the multiple video streams and executes video streaming to the mobile APP through the vehicle network channel for real-time monitoring of the valet parking process.

[0024] In this embodiment, the method further includes: L5. Mobile Video Display: The mobile app receives multiple video streams, decodes and renders them, and displays real-time monitoring footage of the front, rear, and surround views to the user.

[0025] In this embodiment, in step L3, the surround view stitching operation includes using an image registration algorithm based on multi-scale feature fusion and adaptive deformation field to perform image registration on the four undistorted surround view original frames.

[0026] In this embodiment, as Figure 2 As shown, the image registration algorithm based on multi-scale feature fusion and adaptive deformation field includes: L31. The input image is first subjected to intensity normalization to eliminate grayscale differences between different scanning devices; L32. Construct a four-layer Gaussian pyramid, with each layer having a resolution of one-half, one-quarter, one-eighth, and one-sixteenth of the original image, respectively. Extract multi-channel features in each layer, including gray-level gradient, Laplacian response, and local binary pattern features. Feature point matching adopts an improved nearest neighbor search algorithm, and spatial consistency constraints are introduced to eliminate mismatched points. L33. An iterative optimization strategy is adopted, with fifty iterations performed on each pyramid level. In each iteration, the similarity measure gradient under the current deformation field is calculated, and the deformation field is updated in combination with an adaptive regularization term. The regularization weight is dynamically adjusted according to the local deformation gradient. L34. After registration is completed, a topology check is performed on the deformation field to detect and correct any possible folds or tears, and bicubic interpolation is used to generate the final registered image.

[0027] In this embodiment, in step L33, the iterative optimization strategy includes determining convergence and entering the next resolution level when the deformation field change between two adjacent iterations is less than a preset threshold.

[0028] In this embodiment, the detection and correction of possible folds or tears is achieved by evaluating the registration quality through the calculation of overlap indices, which include the Dice coefficient, the Jaccard index, and the target registration error.

[0029] Multi-scale feature extraction module: Extracts feature representations of the input image at different resolutions through a pyramid structure, capturing multi-level information from local details to global structure.

[0030] Adaptive deformation field generation module: Dynamically adjusts the smoothness parameters of the deformation field based on feature differences, allowing large local deformations while maintaining the continuity of the anatomical structure.

[0031] Similarity Measure Optimization Module: An improved composite measure that fuses local normalized mutual information and gradient correlation is adopted to enhance the robustness of multimodal registration.

[0032] Iterative optimization strategy module: Introduces a coarse-to-fine multi-resolution optimization framework and combines momentum acceleration technology to improve convergence efficiency.

[0033] In this embodiment, the mathematical model of this algorithm can be expressed as finding the optimal spatial transformation function that maximizes the similarity between the floating image and the reference image after transformation. The objective function is defined as follows: Objective function = similarity measure + regularization term The similarity measure uses an improved locally normalized mutual information, calculated using the following formula: Local mutual information = Σ(joint probability distribution × log(product of joint probability distribution and marginal probability distribution)) The regularization term employs an adaptive smoothing constraint, dynamically adjusting weights based on local deformation gradients to avoid over-smoothing leading to detail loss or under-smoothing causing artifacts. A multi-scale feature extraction strategy effectively captures multi-level image information, improving registration robustness. Secondly, the adaptive regularization mechanism allows for large local deformations while maintaining topological consistency of the anatomical structure, resolving the inherent contradictions of traditional algorithms. Thirdly, the composite similarity metric design significantly improves multimodal registration accuracy, overcoming the challenges posed by differences in grayscale distribution. Finally, a coarse-to-fine iterative optimization framework balances efficiency and accuracy, enabling practical deployment.

[0034] Example 2: Based on the valet parking video monitoring method in Example 1, the present invention will be further explained and described below.

[0035] like Figure 3As shown, the present invention provides a valet parking video monitoring system for implementing the aforementioned valet parking video monitoring method. The system includes: The multi-camera acquisition unit includes a 120° forward-looking camera, a rear-view camera, and four surround-view cameras, used to acquire raw video data from different perspectives around the vehicle. The HAD controller includes an application module for receiving raw data from multiple cameras, performing distortion correction on the front / rear view camera data, and outputting undistorted raw frames for the surround view camera data; a surround view stitching module for receiving four surround view raw frames and performing surround view stitching and distortion correction; and a video streaming SDK module for integrating the corrected video streams from the front, rear, and surround views and performing encoding and streaming operations.

[0036] In this embodiment, the system also includes a mobile APP, which is used to receive video streaming data, decode, render and display real-time monitoring footage.

[0037] In this embodiment, the multi-camera acquisition unit is connected to the application module, the application module is connected to the surround view video stitching module and the video streaming SDK module, and the surround view video stitching module is connected to the video streaming SDK module.

[0038] In this embodiment, the hardware configuration scheme The vehicle is equipped with six cameras, including one forward-facing 120-degree wide-angle camera (1920×1080 resolution, 30fps), one rear-view camera (1920×1080 resolution, 30fps), and four surround-view cameras (1280×720 resolution, 25fps). The HAD controller uses an automotive-grade chip and features multiple video input interfaces and hardware acceleration capabilities. The video streaming SDK is integrated into the vehicle communication module, supporting 4G / 5G dual-mode networks.

[0039] Software processing flow After system startup, the system first initializes each camera module and establishes data acquisition channels. Data from the forward-looking and rear-looking cameras enters the distortion correction unit, where radial and tangential distortion correction is performed using calibration parameters. Data from the surround-view cameras enters the stitching unit, where feature point extraction and matching are first performed, followed by perspective transformation and image fusion, and finally overall distortion correction. The processed three video streams then enter the encoding module, using H.265 encoding. The bitrate is dynamically adjusted based on network conditions, ranging from 2Mbps to 8Mbps.

[0040] Data transmission scheme Video streaming uses the RTMP protocol, transmitting data to a cloud server via a dedicated vehicle-to-everything (V2X) channel, and then the server distributes it to the user's mobile app. Transmission latency is controlled within 500ms, and the image resolution reaches 720P or higher. The system supports a reconnection mechanism, automatically resuming transmission after a network interruption.

[0041] In this embodiment, in weak network environments such as underground parking lots, the system automatically switches to a low bitrate mode (720P, 15fps) and enables a video frame interpolation algorithm to ensure smooth playback. Simultaneously, it prioritizes pushing surround-view feeds, while front-view and rear-view feeds are loaded on demand to further reduce bandwidth consumption. Tests show that in an environment with a signal strength of -95dBm, the video stuttering rate is less than 3%, meeting practical requirements.

[0042] In this embodiment, the video streaming SDK adopts the H.265 encoding standard, supports dynamic bitrate adjustment for multiple video streams, and automatically optimizes transmission parameters based on network conditions. Encrypted video streams are pushed to the user's mobile app via a 4G / 5G vehicle-to-everything (V2X) channel, achieving real-time monitoring with millisecond-level latency.

[0043] In this embodiment, the present invention provides a computer-readable storage medium storing a computer program programmed or configured to perform the valet parking video monitoring method described above.

[0044] Any references to memory, storage, database, or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0045] In summary, this invention not only ensures that users can obtain clear and complete video footage of the vehicle's surroundings in real time through a mobile app, but its modular design also facilitates the addition of new camera types or adjustments to the processing logic, adapting to the needs of different valet parking scenarios.

[0046] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video monitoring method for valet parking, characterized in that, The method includes: L1. Acquisition of raw data from multiple cameras: Acquire raw video data from the 120° front-view camera, the rear-view camera, and the four-way surround-view camera, and input the raw video data into the application module of the HAD controller; L2. Camera data differentiation processing: Perform distortion correction processing on the raw data of the forward-looking 120° camera and the rear-looking camera, output the corrected single-channel video stream, and output the undistorted raw frames of the raw data of the four surround-view cameras, and transmit them to the surround-view stitching module. L3. Surround view image stitching and distortion correction: The surround view stitching module receives four undistorted surround view raw frames, performs surround view stitching operation to generate a complete surround view image of the vehicle's surroundings, and then performs stitching distortion correction processing on the stitched image to output the corrected surround view video stream. L4. Multi-channel video stream integration and streaming: The single-channel video streams after distortion correction of the front and rear views, as well as the surround view video stream after distortion correction of the surround view stitching, are uniformly input into the video streaming SDK. The video streaming SDK encapsulates and encodes the multiple video streams and executes video streaming to the mobile APP through the vehicle network channel for real-time monitoring of the valet parking process.

2. The valet parking video monitoring method according to claim 1, characterized in that, The method further includes: L5. Mobile Video Display: The mobile app receives multiple video streams, decodes and renders them, and displays real-time monitoring footage of the front, rear, and surround views to the user.

3. The valet parking video monitoring method according to claim 1, characterized in that, In step L3, the surround view stitching operation includes using an image registration algorithm based on multi-scale feature fusion and adaptive deformation field to perform image registration on the four undistorted surround view original frames.

4. The valet parking video monitoring method according to claim 3, characterized in that, The image registration algorithm based on multi-scale feature fusion and adaptive deformation field includes: L31. The input image is first subjected to intensity normalization to eliminate grayscale differences between different scanning devices; L32. Construct a four-layer Gaussian pyramid, with each layer having a resolution of one-half, one-quarter, one-eighth, and one-sixteenth of the original image, respectively. Extract multi-channel features in each layer, including gray-level gradient, Laplacian response, and local binary pattern features. Feature point matching adopts an improved nearest neighbor search algorithm, and spatial consistency constraints are introduced to eliminate mismatched points. L33. An iterative optimization strategy is adopted, with fifty iterations performed on each pyramid level. In each iteration, the similarity measure gradient under the current deformation field is calculated, and the deformation field is updated in combination with an adaptive regularization term. The regularization weight is dynamically adjusted according to the local deformation gradient. L34. After registration is completed, a topology check is performed on the deformation field to detect and correct any possible folds or tears, and bicubic interpolation is used to generate the final registered image.

5. The valet parking video monitoring method according to claim 4, characterized in that, In step L33, the iterative optimization strategy includes determining convergence and proceeding to the next resolution level when the deformation field change between two adjacent iterations is less than a preset threshold.

6. The valet parking video monitoring method according to claim 4, characterized in that: The detection and correction of possible folds or tears involves evaluating registration quality by calculating overlap indices, including the Dice coefficient, Jaccard index, and target registration error.

7. A valet parking video monitoring system, characterized in that, The system for implementing the valet parking video monitoring method according to any one of claims 1-6, the system comprising: The multi-camera acquisition unit includes a 120° forward-looking camera, a rear-view camera, and four surround-view cameras, used to acquire raw video data from different perspectives around the vehicle. The HAD controller includes an application module for receiving raw data from multiple cameras, performing distortion correction on the front / rear view camera data, and outputting undistorted raw frames for the surround view camera data; a surround view stitching module for receiving four surround view raw frames and performing surround view stitching and distortion correction; and a video streaming SDK module for integrating the corrected video streams from the front, rear, and surround views and performing encoding and streaming operations.

8. The valet parking video monitoring system according to claim 7, characterized in that, The system also includes a mobile app for receiving video streaming data, decoding, rendering, and displaying real-time monitoring footage.

9. The valet parking video monitoring system according to claim 7, characterized in that: The multi-camera acquisition unit is connected to the application module, and the application module is connected to the surround view video stitching module and the video streaming SDK module, respectively. The surround view video stitching module is connected to the video streaming SDK module.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is programmed or configured to perform the valet parking video monitoring method according to any one of claims 1 to 6.

Images