Parking space tracking method and system adapted to bumpy road surface

Abstract

The invention provides a parking space tracking method and system suitable for a bumpy road surface, and belongs to the technical field of parking control, and the method comprises the steps: carrying out the preprocessing of multi-modal data obtained in real time, predicting the bumpy degree of the road surface according to the preprocessed multi-modal data, and carrying out the prediction of the bumpy degree of the road surface; adjusting the collection frame rate of the road surface image according to the road surface bumping degree, so that a camera collects the road surface image at the collection frame rate; wherein the multi-modal data comprises a road surface image, IMU data, vehicle position information and vehicle speed information; performing single-frame compensation on each image frame based on the IMU data, and processing a continuous image frame sequence based on the IMU data to obtain a prediction feature of a next frame; performing dynamic weight distribution on each image frame based on the IMU data, and performing weighted fusion to obtain a target image frame; and according to the prediction characteristics of the next frame, identifying and tracking the parking space line of the target image frame.

Description

Technical Field

[0001] This invention relates to the field of parking control technology, and in particular to a parking space tracking method, system, electronic device, and computer-readable medium adapted to bumpy roads. Background Technology

[0002] With the rapid development of autonomous driving technology, parking space detection and tracking plays a crucial role in automated parking systems. However, complex road conditions, especially bumpy roads, pose significant challenges to parking space detection and tracking algorithms. When a vehicle travels on a bumpy road, images captured by cameras or other sensors may experience severe vibrations, causing deformation or displacement of key features such as parking lines and signs, thus affecting the accuracy of detection and tracking.

[0003] Image stabilization (IMU) technology is one of the fundamental techniques for addressing camera shake. It typically reduces the impact of camera shake by using inter-frame alignment or image smoothing algorithms (such as optical flow and Kalman filters). However, these methods often rely on feature matching within the image itself, making them susceptible to drastic changes in road conditions or dynamic scenes (such as the sudden appearance of obstacles), and they are prone to failure, especially under severe bumpy conditions. IMU sensors measure vehicle acceleration and angular velocity, helping to compensate for camera shake caused by vehicle motion. In existing technologies, IMUs are usually combined with image data for simple attitude compensation to reduce the impact of shake on the image. However, most existing research focuses primarily on compensating for planar motion of the vehicle (i.e., horizontal movement), neglecting multidimensional motion under complex road conditions (such as pitch, roll, and other three-dimensional rotations), resulting in limited compensation effectiveness.

[0004] Therefore, existing technologies rely on image processing algorithms or simple sensor fusion techniques, which are difficult to achieve good results in complex and bumpy environments. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art, and proposes a parking space tracking method, system, electronic device and computer-readable medium adapted to bumpy roads.

[0006] In a first aspect, embodiments of the present invention provide a parking space tracking method adapted to bumpy road surfaces, the method comprising:

[0007] The real-time acquired multimodal data is preprocessed, the road surface bumpiness is predicted based on the preprocessed multimodal data, and the acquisition frame rate of the road surface image is adjusted according to the road surface bumpiness so that the camera can acquire the road surface image at the acquisition frame rate; wherein, the multimodal data includes road surface image, IMU data, vehicle location information and vehicle speed information;

[0008] Based on the IMU data, single-frame compensation is performed on each acquired image frame, and based on the IMU data, a continuous sequence of image frames is processed to obtain the predicted features for the next frame.

[0009] Based on the IMU data, dynamic weights are assigned to each image frame, and weighted fusion is performed to obtain the target image frame;

[0010] Based on the predicted features for the next frame, the parking lines in the target image frame are identified and tracked.

[0011] Optionally, predicting the road surface bumpiness based on the preprocessed multimodal data includes:

[0012] The preprocessed multimodal data is input into a pre-trained neural network model to predict the degree of road bumps that the vehicle may encounter in the future. The neural network model is a real-time vehicle speed-road coupling model obtained by fusing at least vehicle speed information, IMU data and road type information.

[0013] Optionally, the step of performing single-frame compensation on each acquired image frame based on the IMU data includes:

[0014] For each image frame, the attitude information corresponding to that image frame in the IMU data is mapped to the image coordinate system, and geometric correction is performed on the image frame to eliminate perspective distortion caused by changes in vehicle attitude.

[0015] Optionally, the step of processing the continuous image frame sequence based on the IMU data to obtain the predicted features for the next frame includes:

[0016] Spatial features are extracted from each image frame using an image feature extraction module, and motion compensation is performed in conjunction with the IMU data.

[0017] A time-series analysis and prediction module is used to process consecutive image frames, capture dynamic changes between frames, and obtain predictive features for the next frame.

[0018] Optionally, the step of extracting spatial features from each image frame using the image feature extraction module and performing motion compensation in conjunction with the IMU data includes:

[0019] Spatial features of each image frame are extracted through multi-layer convolution.

[0020] The vehicle motion state corresponding to the image frame in the IMU data is processed by a fully connected layer, which maps it to the same dimension as the spatial features of the image frame, and the features of the IMU data are fused with the spatial features of each image frame channel by channel.

[0021] Optionally, a temporal analysis and prediction module is used to process consecutive image frames, capture dynamic changes between frames, and obtain predictive features for the next frame, including:

[0022] The self-attention mechanism is used to model the correlation between the features of the current frame and the historical frames. The correlation between each historical frame and the current frame is calculated and weighted to obtain the temporal features.

[0023] The temporal features are processed by a temporal convolutional network to generate predictive features for the next frame.

[0024] Optionally, the dynamic weight allocation of each image frame based on the IMU data includes:

[0025] The weights of each image frame are determined using the following formula:

[0026]

[0027] in, Indicates time The weights of the acquired image frames, and Representing time respectively The acceleration and angular velocity in the IMU data, and These represent the vehicle's acceleration and angular velocity in a steady state, respectively. and This represents the hyperparameters that adjust the effects of acceleration and angular velocity.

[0028] Optionally, the method further includes: optimizing the hyperparameters through online learning. and .

[0029] Optionally, the step of identifying and tracking parking lines in the target image frame based on the predicted features of the next frame includes:

[0030] Parking line detection is performed on the target image frame using a convolutional neural network;

[0031] Based on the predicted features for the next frame, a tracking algorithm is used to continuously track parking lines in multiple consecutively acquired target image frames.

[0032] Optionally, when performing single-frame compensation on each acquired image frame based on the IMU data, auxiliary correction is performed by combining the vehicle position information and vehicle speed information.

[0033] Optionally, when performing the process of processing a continuous sequence of image frames based on the IMU data to obtain the predicted features for the next frame, auxiliary correction is performed by combining the vehicle position information and vehicle speed information.

[0034] Optionally, the preprocessing of the real-time acquired multimodal data includes:

[0035] Using the GPS time synchronization mechanism, timestamps are added to various types of data in the real-time acquired multimodal data;

[0036] An interpolation algorithm is used to align different types of data at different frequencies, so that the data of different types are synchronized in the time dimension.

[0037] Secondly, embodiments of the present invention provide a parking space tracking system adapted to bumpy road surfaces, comprising:

[0038] The prediction module is used to synchronize the real-time acquired multimodal data, predict the road surface bumpiness based on the synchronized multimodal data, and adjust the acquisition frame rate of the road surface image according to the road surface bumpiness, so that the camera can acquire the road surface image at the acquisition frame rate; wherein, the multimodal data includes road surface image, IMU data, vehicle location information, and vehicle speed information;

[0039] The compensation module is used to perform single-frame compensation on each acquired image frame based on the IMU data, and to process the continuous image frame sequence based on the IMU data to obtain the prediction features for the next frame.

[0040] The fusion module is used to dynamically assign weights to each image frame based on the IMU data, and then perform weighted fusion to obtain the target image frame.

[0041] The tracking module is used to identify and track parking lines in the target image frame based on the predicted features for the next frame.

[0042] Thirdly, embodiments of the present invention provide an electronic device, including:

[0043] One or more processors;

[0044] Memory, used to store one or more programs;

[0045] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-described parking space tracking method adapted to bumpy roads.

[0046] Fourthly, embodiments of the present invention provide a computer-readable medium storing a computer program, which, when executed by a processor, implements the steps in the above-described method for tracking parking spaces on bumpy roads.

[0047] The parking space tracking method adapted to bumpy roads provided by this invention can predict bumps in advance through multimodal data, dynamically optimize the image acquisition frame rate, and enhance tracking stability. Through the fusion of multi-frame historical data and nonlinear temporal prediction, it can accurately predict the position of parking lines and signs under complex road conditions. Through an adaptive frame weight allocation mechanism, the importance of image frames is dynamically adjusted according to the vehicle's motion state, and the most reliable data source is selected, which significantly improves the detection accuracy and robustness of the system in continuous bumpy or sharp turning scenarios. Attached Figure Description

[0048] Figure 1 A flowchart illustrating a parking space tracking method adapted to bumpy roads, provided by an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram illustrating the multimodal data synchronization and fusion method provided in the embodiments of the present invention;

[0050] Figure 3 This is a flowchart illustrating an optional specific implementation method of step S2 in an embodiment of the present invention.

[0051] Figure 4 A structural block diagram of a parking space tracking system adapted to bumpy roads provided in an embodiment of the present invention;

[0052] Figure 5 This is an optional structural block diagram of the compensation module 20 in an embodiment of the present invention;

[0053] Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0054] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0055] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0056] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0057] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0058] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0059] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0060] Current parking space tracking methods have many shortcomings when dealing with complex road conditions, especially when vehicles pass over bumps or make sharp turns. Image jitter and motion blur can easily lead to instantaneous recognition errors, reducing the tracking accuracy of parking lines and markers. Most existing methods rely on passive compensation of single-frame images, which is difficult to cope with dynamically changing road conditions and fails to fully utilize vehicle motion data for prediction optimization. In addition, simple averaging of historical frame data cannot effectively handle nonlinear motion in complex scenes.

[0061] To address at least one of the technical problems existing in the aforementioned related technologies, the present invention provides a parking space tracking method and system adapted to bumpy road surfaces. Figure 1 A flowchart illustrating a parking space tracking method adapted to bumpy road surfaces, provided by an embodiment of the present invention, includes the following steps:

[0062] S1, synchronize the real-time acquired multimodal data, predict the road surface bumpiness based on the synchronized multimodal data, and adjust the acquisition frame rate of the road surface image according to the road surface bumpiness, so that the camera can acquire the road surface image at the acquisition frame rate; wherein, the multimodal data includes road surface image, IMU data, vehicle location information and vehicle speed information.

[0063] In this embodiment of the invention, the vehicle is equipped with a camera, an IMU (Inertial Measurement Unit) sensor, a position sensor (such as GPS), and a vehicle speed sensor. A high-resolution camera is used to acquire real-time road surface images. These image data include parking lines, road markings, and other visual information that may influence parking decisions. The IMU sensor acquires the vehicle's acceleration, angular velocity, and three-axis attitude information in space in real time, including pitch, roll, and yaw angles. This IMU data is crucial for understanding the vehicle's dynamic behavior on bumpy roads because it reflects the vehicle's instantaneous motion state under complex road conditions. By combining the vehicle position and speed information provided by the position and speed sensors, the complete motion state of the vehicle in complex dynamic environments and the corresponding image sequence can be obtained.

[0064] The various types of data collected by the aforementioned data acquisition devices constitute multimodal data, which requires integration and preprocessing. Firstly, because the data acquisition frequencies of different devices vary, these data need to be synchronized using timestamps. For example... Figure 2 As shown, GPS time synchronization is used to accurately timestamp various types of data. Then, interpolation algorithms are used to align different types of data at different frequencies, ensuring they work together in the time dimension. Secondly, the acquired image sequences require image preprocessing, such as filtering and denoising to remove unnecessary noise. Simultaneously, considering the potential impact of environmental factors like lighting, shadows, and weather on the camera during vehicle movement, adaptive brightness and contrast adjustments are made to ensure image quality. Furthermore, histogram equalization, contrast stretching, and sharpening image enhancement techniques are employed to make the images visually clearer, brighter, and with stronger contrast.

[0065] The multi-modal data processed as described above can be used for subsequent data analysis. In one embodiment of the invention, the pre-processed multimodal data is input into a pre-trained neural network model to predict the degree of road bumps a vehicle may encounter in the future. The neural network model is a real-time vehicle speed-road coupling model obtained by fusing vehicle speed information, IMU data, and road type information. The neural network model is trained using historical data and integrates vehicle speed information, vehicle motion state information, and road type information. Road type information can be obtained through a preset map or real-time detection, identifying the road's flatness, unevenness, etc., using road cameras or other sensing devices. By fusing the multimodal data, a real-time vehicle speed-road coupling model is obtained to predict the degree of road bumps a vehicle may encounter in the future. For relatively flat roads, the model predicts a lower degree of bumps; while on uneven road sections or when passing speed bumps, the model can predict increased bumps. Through high-frequency feedback from the IMU sensor, the model can capture subtle vehicle vibration trends in real time and, combined with historical data, make a more accurate prediction of the degree of bumps.

[0066] As an embodiment of the present invention, once the system predicts that it will encounter a bumpy road section, it will adjust the image acquisition frame rate in advance so that the camera can acquire road images at the predicted frame rate. When a minor bump is predicted, the system will slightly increase the image acquisition frame rate, thereby increasing the update frequency of the subsequent compensation algorithm. When encountering more severe bumpy road sections (such as speed bumps or unpaved roads), the system will significantly increase the frame rate to reduce the jitter effect in a single frame image. At the same time, the subsequent compensation algorithm will also add nonlinear motion compensation according to the predicted bump level, so as to minimize the impact of vehicle movement on the image, thereby ensuring the stability of parking space tracking and sign recognition.

[0067] It is understood that this invention proposes an intelligent prediction model based on vehicle speed and road conditions. By designing an intelligent prediction model that combines vehicle speed, road surface type, and IMU data, it can predict the degree of road bumpiness that a vehicle is about to pass through, and dynamically adjust image processing parameters and tracking algorithm accuracy based on this prediction. Under high-bump road conditions, the system automatically increases the image acquisition frame rate and compensation frequency, enhancing the stability of parking space tracking. Unlike traditional methods that rely on passive compensation after bumps occur, this intelligent prediction model can optimize algorithm parameters in advance based on future road condition changes, reducing instantaneous recognition errors caused by bumps or changes in road conditions, thereby improving the continuity and robustness of tracking.

[0068] S2, perform single-frame compensation on each acquired image frame based on the IMU data, and process the continuous image frame sequence based on the IMU data to obtain the prediction features for the next frame.

[0069] In the processing, this invention first utilizes vehicle motion data acquired by an IMU sensor for single-frame compensation. Using IMU data, geometric correction can be performed on single-frame images captured by the camera, correcting image distortion caused by vehicle shaking or tilting. Specifically, for each image frame, the attitude information corresponding to that image frame in the IMU data is mapped to the image coordinate system, and geometric correction is performed on that image frame. This adjusts the camera's virtual viewing angle in real time, eliminating perspective distortion caused by changes in vehicle attitude. This compensation step effectively improves the accuracy of single-frame images; however, in dynamic and complex environments, single-frame compensation may still experience instantaneous error accumulation.

[0070] To better capture these dynamic features, this invention proposes an innovative temporal deep learning model by combining IMU data with continuous image frame sequences. This model can effectively capture the vehicle's motion trend and predict the camera's future motion state in advance. Unlike traditional LSTM or RNN models, this temporal model combines a self-attention mechanism and an image feature extraction network to construct a novel Temporal-Spatial Recursive Network (TSR). This model includes an image feature extraction module and a temporal analysis and prediction module.

[0071] like Figure 3 As shown, the step of processing a continuous sequence of image frames based on the IMU data to obtain the predicted features for the next frame specifically includes: S21, using an image feature extraction module to extract spatial features from each image frame and combining them with the IMU data for motion compensation; S22, using a temporal analysis and prediction module to process the continuous image frames, capture the dynamic changes between frames, and obtain the predicted features for the next frame.

[0072] In step S21, the image feature extraction module extracts high-level spatial features from each image frame and performs motion compensation using IMU data. The input data includes original image frames of size H×W×C and 6-dimensional IMU data (including acceleration and angular velocity). Specifically, spatial features of each image frame are first extracted using multi-layer convolution; then, the vehicle motion state corresponding to the image frame in the IMU data is processed through a fully connected layer, mapping it to the same dimension as the spatial features of the image frame, and the features of the IMU data are fused with the spatial features of each image frame channel by channel.

[0073] More specifically, the image frame is first input into a lightweight convolutional neural network (MobileNet can be used) to extract spatial features. The first layer of the convolutional network maps the image from H×W×3 to a higher-dimensional feature space. Assuming the first layer uses a 3×3 kernel size, a stride of 2, and padding of 1, the size of the output feature map after this layer becomes (H / 2)×(W / 2)×C1, where C1 is the number of kernels. After multiple convolutions, the output shape of the convolutional network is (H / 16)×(W / 16)×Cn, where Cn is the number of channels in the last layer of the network. This feature map encodes the spatial structure information of the image.

[0074] Then, the spatial structure information of the obtained image is embedded into the IMU data. The IMU data contains acceleration and angular velocity information, represented as follows: These data reflect the vehicle's motion state. First, a fully connected (FC) layer processes the vehicle motion state corresponding to the image frame in the IMU data, mapping it to the same dimension as the image features. Its vector representation is as follows: Then, this IMU feature is fused with the spatial features of each frame of the image channel by channel (e.g., through weighted summation or dot product). The fused feature is represented as: ,in, These are the image features output from the convolutional network. It is a learnable scalar weight used to adjust the degree of influence of IMU information.

[0075] In step S22, the goal of the temporal analysis and prediction module is to process consecutive image frames, capture dynamic changes between frames, and obtain predictive features of the parking space status in the next frame to compensate for motion blur or camera shake. Specifically, firstly, a self-attention mechanism is used to model the correlation between the features of the current frame and historical frames, calculate the correlation between each historical frame and the current frame, and obtain temporal features after weighting; then, the temporal features are processed through a temporal convolutional network to generate predictive features for the next frame.

[0076] More specifically, in time series analysis, a self-attention mechanism is used to model the correlation between the features of the current frame and historical frames, setting the features of the current frame as... The characteristic sequence of historical frames is , where k is the number of historical frames. To focus on the most relevant time steps, this invention uses a multi-head self-attention mechanism to calculate the correlation between different frames. The features of each frame are mapped to query, key, and value through a linear transformation. Then, the correlation between each historical frame and the current frame is calculated. By summarizing the information from all historical frames, the self-attention-weighted temporal features are represented as follows: .

[0077] Then, after the self-attention operation, these temporal features are further processed by a Temporal Convolutional Network (TCN) to generate predicted features for the next frame: ,in, It is the hidden state at the current moment. It is a prediction of the features at the next moment, which is used to compensate for the camera's motion state when identifying parking lines.

[0078] It is understood that, in addition to motion compensation for single-frame images, this invention also proposes a multi-frame historical data fusion and nonlinear temporal prediction mechanism. By designing a TSR deep learning network that combines a self-attention mechanism and an image feature extraction network, it fuses and analyzes multi-frame historical images and IMU data. By introducing a nonlinear temporal prediction model, the system can capture the dynamic changes of vehicle motion and road markings in time and space, predicting the future shape and position of parking lines and markers. This mechanism ensures that the system maintains high accuracy in parking space tracking even under complex road conditions such as continuous bumps or sharp turns.

[0079] S3, based on the IMU data, dynamically assign weights to each image frame and perform weighted fusion to obtain the target image frame.

[0080] The motion state of a vehicle can usually be measured by acceleration from IMU data. and angular velocity To describe. Let's assume... Given the vehicle's attitude rotation matrix at time t, and combining the vehicle's acceleration and angular velocity, the following nonlinear motion model can be used to describe the vehicle's motion in complex scenarios:

[0081]

[0082] in, It is determined by angular velocity The corresponding antisymmetric matrix is ​​defined as follows:

[0083]

[0084] This formula can describe the vehicle's performance over a period of time. Nonlinear attitude changes within the body. Combined with acceleration data. This allows for the further calculation of the vehicle's nonlinear motion trajectory.

[0085] To compensate for image jitter caused by the nonlinear motion of the vehicle, weighted fusion of data from different image frames is performed. Let... Indicates time The acquired image frames need to be selected for compensation based on IMU feedback. Assuming the frame weights are determined by the vehicle's motion state, the weights... It can be expressed as a function of the vehicle's stable state:

[0086]

[0087] Among them, the weight function It can be defined as:

[0088]

[0089] in, and Representing time respectively The acceleration and angular velocity in the IMU data, and These represent the vehicle's acceleration and angular velocity in a steady state, respectively. and This is a hyperparameter that adjusts the effects of acceleration and angular velocity. The meaning of this weighting function is that when the vehicle's acceleration and angular velocity deviate from a steady state, the weights... The weight of the most recent frame will decrease when the vehicle is violently bumpy, while the weight of stable frames in the past will increase.

[0090] To ensure the system selects the most reliable image frames for marker detection and parking line tracking, the system performs weighted fusion of multiple image frames. Combining the weighted fusion of historical frames and the current frame, the system can utilize a deep learning network to perform marker detection and parking line tracking on the fused image in one step. It is understood that the adaptive frame weight allocation mechanism proposed in this embodiment dynamically allocates weights to different image frames based on the vehicle's real-time motion state (such as acceleration and angular velocity fed back by the IMU). When the vehicle experiences severe vibrations, the system tends to rely on the most recent stable frame rather than the current jittery frame. In this way, the system can select the most reliable historical frame data for marker detection and parking line tracking, thereby significantly improving the robustness and stability of detection under complex road conditions.

[0091] To further improve system performance, the weight function can also be optimized through online learning. Parameters in and These parameters are adjusted based on the accuracy of historical detection results to ensure dynamic adaptation to changing vehicle motion under different road conditions. This adaptive optimization can be achieved by minimizing a loss function that minimizes the detection error.

[0092]

[0093] By minimizing the loss L, the system can continuously adjust the frame weight allocation mechanism to better adapt to nonlinear motion in complex road conditions.

[0094] In addition to spatiotemporal compensation combining IMU data and image frames, a multi-sensor collaborative compensation mechanism can also be employed. When a vehicle is traveling at high speed or in complex road conditions, IMU data may experience noise or drift. In such cases, data from GPS and vehicle speed sensors can be introduced for auxiliary correction. By inferring the vehicle's motion trend from the vehicle speed data and combining it with GPS positioning to accurately compensate for IMU data drift, the overall accuracy and robustness of the system can be improved.

[0095] S4, Based on the predicted features for the next frame, identify and track parking lines in the target image frame.

[0096] First, a convolutional neural network is used to detect parking lines in the compensated image (i.e., the target image frame). By extracting feature points and edge information from the image, the shape and position of the parking lines can be accurately located. Unlike traditional detection methods, this invention uses a compensated, stable image for recognition, greatly improving detection accuracy in complex scenes.

[0097] After detecting the parking lines, the system further uses the Kalman filter algorithm or other tracking algorithms based on the temporal information of the preceding and following frames (i.e., the predicted features for the next frame) to continuously track the parking lines on the continuously acquired target image frames. This not only improves the robustness of the system but also predicts the future movement trend of the parking lines, ensuring that parking spaces can be continuously identified and tracked during vehicle movement.

[0098] In summary, the parking space tracking method adapted to bumpy roads provided by the embodiments of the present invention can predict bumps in advance through multimodal data, dynamically optimize the image acquisition frame rate, and enhance tracking stability. Through the fusion of multi-frame historical data and nonlinear temporal prediction, it can accurately predict the position of parking lines and signs under complex road conditions. Through an adaptive frame weight allocation mechanism, it can dynamically adjust the importance of image frames according to the vehicle's motion state and select the most reliable data source, significantly improving the detection accuracy and robustness of the system in continuous bumpy or sharp turning scenarios.

[0099] Based on the same inventive concept, embodiments of the present invention also provide a parking space tracking system adapted to bumpy roads. Figure 4 This is a structural block diagram of a parking space tracking system adapted to bumpy roads, provided as an embodiment of the present invention. Figure 4 As shown, an embodiment of the present invention provides a parking space tracking system adapted to bumpy roads, comprising:

[0100] The prediction module 10 is used to synchronize the real-time acquired multimodal data, predict the road surface bumpiness based on the synchronized multimodal data, and adjust the acquisition frame rate of the road surface image according to the road surface bumpiness, so that the camera can acquire the road surface image at the acquisition frame rate; wherein, the multimodal data includes road surface image, IMU data, vehicle location information, and vehicle speed information;

[0101] The compensation module 20 is used to perform single-frame compensation on each acquired image frame based on the IMU data, and to process the continuous image frame sequence based on the IMU data to obtain the prediction features for the next frame.

[0102] The fusion module 30 is used to dynamically assign weights to each image frame based on the IMU data and then perform weighted fusion to obtain the target image frame.

[0103] The tracking module 40 is used to identify and track parking lines in the target image frame based on the predicted features for the next frame.

[0104] Optionally, the prediction module 10 predicts the road surface bumpiness based on the preprocessed multimodal data, specifically by inputting the preprocessed multimodal data into a pre-trained neural network model to predict the road surface bumpiness that the vehicle may encounter in the future. The neural network model is a real-time vehicle speed-road surface coupling model obtained by fusing at least vehicle speed information, IMU data, and road surface type information.

[0105] Optionally, the compensation module 20 performs single-frame compensation on each acquired image frame based on the IMU data. Specifically, for each image frame, the attitude information corresponding to the image frame in the IMU data is mapped to the image coordinate system, and geometric correction is performed on the image frame to eliminate perspective distortion caused by changes in vehicle attitude.

[0106] Optionally, such as Figure 5 As shown, the compensation module 20 includes:

[0107] The first compensation unit 201 is used to extract spatial features from each image frame using an image feature extraction module and perform motion compensation in combination with the IMU data.

[0108] The second compensation unit 202 is used to process consecutive image frames using a time-series analysis and prediction module, capture dynamic changes between frames, and obtain prediction features for the next frame.

[0109] Optionally, the first compensation unit 201 is specifically used to: extract the spatial features of each image frame through multi-layer convolution; process the vehicle motion state corresponding to the image frame in the IMU data through a fully connected layer, map it to the same dimension as the spatial features of the image frame, and fuse the features of the IMU data with the spatial features of each image frame channel by channel.

[0110] Optionally, the second compensation unit 202 is specifically used to: model the correlation between the features of the current frame and the historical frames using a self-attention mechanism, calculate the correlation between each historical frame and the current frame, and obtain temporal features after weighting; process the temporal features through a temporal convolutional network to generate predictive features for the next frame.

[0111] Optionally, the fusion module 30 dynamically assigns weights to each image frame based on the IMU data, specifically by determining the weight of each image frame according to the following formula:

[0112]

[0113] in, Indicates time The weights of the acquired image frames, and Representing time respectively The acceleration and angular velocity in the IMU data, and These represent the vehicle's acceleration and angular velocity in a steady state, respectively. and This represents the hyperparameters that adjust the effects of acceleration and angular velocity.

[0114] Optionally, the fusion module 30 is further configured to: optimize the hyperparameters through online learning. and .

[0115] Optionally, the tracking module 40 identifies and tracks parking lines in the target image frame based on the predicted features for the next frame. Specifically, it uses a convolutional neural network to detect parking lines in the target image frame and employs a tracking algorithm based on the predicted features for the next frame to continuously track parking lines in multiple consecutively obtained target image frames.

[0116] Optionally, the compensation module 20 is further configured to perform auxiliary correction by combining the vehicle position information and vehicle speed information when performing single-frame compensation based on the IMU data for each acquired image frame.

[0117] Optionally, the compensation module 20 is further configured to perform auxiliary correction by combining the vehicle position information and vehicle speed information when processing the continuous image frame sequence based on the IMU data to obtain the prediction features for the next frame.

[0118] Optionally, the prediction module 10 preprocesses the real-time acquired multimodal data, specifically by: using the GPS time synchronization mechanism to timestamp each type of data in the real-time acquired multimodal data; and using an interpolation algorithm to align each type of data at different frequencies, so that each type of data is synchronized in the time dimension.

[0119] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 6 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement any of the parking space tracking methods adapted to bumpy roads as described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0120] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0121] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0122] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0123] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of any of the parking space tracking methods adapted to bumpy road surfaces described in the above embodiments. The computer-readable storage medium may be volatile or non-volatile.

[0124] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described parking space tracking method adapted to bumpy roads.

[0125] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0126] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0127] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0128] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0129] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0130] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0131] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0132] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0134] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A parking space tracking method adapted to bumpy road surfaces, characterized in that, The method includes: The real-time acquired multimodal data is preprocessed, the road surface bumpiness is predicted based on the preprocessed multimodal data, and the acquisition frame rate of the road surface image is adjusted according to the road surface bumpiness so that the camera can acquire the road surface image at the acquisition frame rate; wherein, the multimodal data includes road surface image, IMU data, vehicle location information and vehicle speed information; Based on the IMU data, single-frame compensation is performed on each acquired image frame, and based on the IMU data, a continuous sequence of image frames is processed to obtain the predicted features for the next frame. Based on the IMU data, dynamic weights are assigned to each image frame, and weighted fusion is performed to obtain the target image frame; Based on the predicted features for the next frame, the parking lines in the target image frame are identified and tracked.

2. The method according to claim 1, characterized in that, The step of predicting road surface bumpiness based on the preprocessed multimodal data includes: The preprocessed multimodal data is input into a pre-trained neural network model to predict the degree of road bumps that the vehicle may encounter in the future. The neural network model is a real-time vehicle speed-road coupling model obtained by fusing at least vehicle speed information, IMU data and road type information.

3. The method according to claim 1, characterized in that, The single-frame compensation based on the IMU data for each acquired image frame includes: For each image frame, the attitude information corresponding to that image frame in the IMU data is mapped to the image coordinate system, and geometric correction is performed on the image frame to eliminate perspective distortion caused by changes in vehicle attitude.

4. The method according to claim 1, characterized in that, The process of processing a continuous sequence of image frames based on the IMU data to obtain predictive features for the next frame includes: Spatial features are extracted from each image frame using an image feature extraction module, and motion compensation is performed in conjunction with the IMU data. A time-series analysis and prediction module is used to process consecutive image frames, capture dynamic changes between frames, and obtain predictive features for the next frame.

5. The method according to claim 4, characterized in that, The step of extracting spatial features from each image frame using an image feature extraction module and performing motion compensation in conjunction with the IMU data includes: Spatial features of each image frame are extracted through multi-layer convolution. The vehicle motion state corresponding to the image frame in the IMU data is processed by a fully connected layer, which maps it to the same dimension as the spatial features of the image frame, and the features of the IMU data are fused with the spatial features of the image frame channel by channel.

6. The method according to claim 4, characterized in that, A temporal analysis and prediction module is used to process consecutive image frames, capturing dynamic changes between frames to obtain predictive features for the next frame, including: The self-attention mechanism is used to model the correlation between the features of the current frame and the historical frames. The correlation between each historical frame and the current frame is calculated and weighted to obtain the temporal features. The temporal features are processed by a temporal convolutional network to generate predictive features for the next frame.

7. The method according to claim 1, characterized in that, The dynamic weight allocation of each image frame based on the IMU data includes: The weights of each image frame are determined using the following formula: in, Indicates time The weights of the acquired image frames, and Representing time respectively The acceleration and angular velocity in the IMU data, and These represent the vehicle's acceleration and angular velocity in a steady state, respectively. and This represents the hyperparameters that adjust the effects of acceleration and angular velocity.

8. The method according to claim 7, characterized in that, The method further includes: optimizing the hyperparameters through online learning. and .

9. The method according to claim 1, characterized in that, The step of identifying and tracking parking lines in the target image frame based on the predicted features of the next frame includes: Parking line detection is performed on the target image frame using a convolutional neural network; Based on the predicted features for the next frame, a tracking algorithm is used to continuously track parking lines in multiple consecutively acquired target image frames.

10. The method according to claim 1, characterized in that, When performing single-frame compensation on each acquired image frame based on the IMU data, auxiliary correction is performed by combining the vehicle position information and vehicle speed information.

11. The method according to claim 1, characterized in that, When processing a continuous sequence of image frames based on the IMU data to obtain the predicted features for the next frame, auxiliary correction is performed by combining the vehicle position information and vehicle speed information.

12. The method according to claim 1, characterized in that, The preprocessing of the real-time acquired multimodal data includes: Using the GPS time synchronization mechanism, timestamps are added to various types of data in the real-time acquired multimodal data; An interpolation algorithm is used to align different types of data at different frequencies, so that the data of different types are synchronized in the time dimension.

13. A parking space tracking system adapted to bumpy roads, characterized in that, include: The prediction module is used to synchronize the real-time acquired multimodal data, predict the road surface bumpiness based on the synchronized multimodal data, and adjust the acquisition frame rate of the road surface image according to the road surface bumpiness, so that the camera can acquire the road surface image at the acquisition frame rate; wherein, the multimodal data includes road surface image, IMU data, vehicle location information, and vehicle speed information; The compensation module is used to perform single-frame compensation on each acquired image frame based on the vehicle motion state, and to process the continuous image frame sequence based on the vehicle motion state to obtain the prediction features for the next frame. The fusion module is used to dynamically assign weights to each image frame based on the vehicle's motion state, and then perform weighted fusion to obtain the target image frame. The tracking module is used to identify and track parking lines in the target image frame based on the predicted features for the next frame.

14. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 12.

15. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 12.

Images