Parking visual positioning method and system based on parking space semantic information and vehicle-mounted terminal

Abstract

The invention provides a parking visual positioning method and system based on parking space semantic information and a vehicle-mounted terminal.The method comprises the steps that an aerial view image of a vehicle in a parking environment and a global parking space semantic map are obtained, and the global parking space semantic map is constructed according to the parking environment; inputting the aerial view image into a parking space detection model for parking space detection, and obtaining first parking space semantic information under an image coordinate system; according to the relative position relation between the vehicle and the parking space in the aerial view image, global coordinate conversion is carried out on the first parking space semantic information, second parking space semantic information under a global coordinate system is obtained, and the global coordinate system is constructed based on the parking environment; and matching the second parking space semantic information with the global parking space semantic map, and determining the position of the vehicle in the parking environment. Parking visual positioning is carried out based on the corresponding parking space semantic information of the vehicle in the current parking environment, the position of the vehicle in the parking environment can be accurately determined, and the accuracy of vehicle positioning during parking is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle parking technology, and in particular to a parking visual positioning method, system and vehicle-mounted terminal based on parking space semantic information. Background Art

[0002] In modern vehicle automatic parking systems, visual positioning technology is widely used to improve the accuracy and safety of automatic parking. Traditional visual positioning methods mainly include methods based on image feature point matching and object detection methods based on deep learning. Among them, the method based on image feature point matching relies on extracting and matching feature points from the image for visual positioning, such as SIFT (Scale-invariant feature transform), SURF (Speeded Up Robust Features) or ORB (Oriented FAST and RotatedBRIEF, FAST corner detection and BRIEF feature description), etc. The object detection method based on deep learning relies on training CNN (Convolutional Neural Networks) to identify environmental features such as parking spaces and lane lines for visual positioning.

[0003] However, in APA (Automated Parking Assist) and AVP (Automated Valet Parking) scenarios, unstable feature tracking is often affected by light reflection and frequently moving objects. Methods based on image feature point matching perform poorly when dealing with changes in illumination, changes in perspective, or partial occlusion. They are usually unable to extract enough effective feature points in parking environments with sparse textures and dynamically changing light, resulting in a significant decrease in positioning accuracy. In addition, this method usually requires high-quality feature point extraction and matching algorithms, which have high computational complexity and poor real-time positioning. Target detection methods based on deep learning rely on a large amount of annotated data for training, and have limited adaptability to different environments and conditions. In addition, the reasoning process of deep learning models requires large computing resources and is difficult to meet real-time requirements. Therefore, the parking visual positioning methods in related technologies still have obvious defects in accuracy, robustness, and real-time performance, which leads to inaccurate vehicle positioning during parking. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, the present application discloses a parking visual positioning method, system and vehicle-mounted terminal based on parking space semantic information, which are used to solve the technical problem of inaccurate vehicle positioning during parking in the prior art.

[0005] In a first aspect, the present application provides a parking visual positioning method based on parking space semantic information, the method comprising: obtaining a bird's-eye view image of a vehicle in a parking environment and a global parking space semantic map, the global parking space semantic map being constructed according to the parking environment; inputting the bird's-eye view image into a parking space detection model for parking space detection to obtain first parking space semantic information in an image coordinate system, the parking space detection model being trained based on a sample bird's-eye view image; performing a global coordinate transformation on the first parking space semantic information according to a relative position relationship between the vehicle and the parking space in the bird's-eye view image to obtain second parking space semantic information in a global coordinate system, the global coordinate system being constructed with the parking environment as a reference; matching the second parking space semantic information with the global parking space semantic map to determine the position of the vehicle in the parking environment.

[0006] In one embodiment of the present application, the method for generating the bird's-eye view image includes: calibrating the camera extrinsic parameters of the vehicle camera and capturing multiple surround-view images of the vehicle at different perspectives; performing a bird's-eye view conversion on each of the surround-view images according to the camera extrinsic parameters to obtain multiple bird's-eye view sub-images at the different perspectives; and splicing the multiple bird's-eye view sub-images according to a preset bird's-eye view image size to generate the bird's-eye view image.

[0007] In one embodiment of the present application, the camera extrinsic calibration of the vehicle camera includes: obtaining a preset camera intrinsic distortion coefficient of the vehicle camera, and obtaining a plurality of calibration images at different viewing angles based on a checkerboard calibration plate; for each of the calibration images, performing a dedistortion process according to the camera intrinsic distortion coefficient, and gray-scaling the dedistorted calibration image to obtain a grayscale image; identifying the corner points of the checkerboard in each of the grayscale images to obtain a plurality of groups of checkerboard corner point coordinates; and calculating the plurality of groups of checkerboard corner point coordinates and the calibration point coordinates of the corner points preset in the checkerboard calibration plate to obtain the camera extrinsic parameters.

[0008] In one embodiment of the present application, the first parking space semantic information is subjected to a global coordinate transformation according to the relative position relationship between the vehicle and the parking space in the bird's-eye view image to obtain the second parking space semantic information in the global coordinate system, including: transforming the first parking space semantic information according to the relative position relationship between the vehicle and the parking space in the bird's-eye view image to obtain the third parking space semantic information in the vehicle body coordinate system; calculating the third parking space semantic information according to the visual-inertial fusion parameters and the vehicle forward distance to obtain the second parking space semantic information, the visual-inertial fusion parameters including a rotation matrix and a translation vector, which are obtained by calibrating the relative position and posture relationship between the vehicle camera and the vehicle inertial sensor, and the vehicle forward distance is the vehicle forward distance relative to the origin of the global coordinate system, which is obtained by monitoring the wheel speed sensor.

[0009] In one embodiment of the present application, the calibration method of the rotation matrix includes: collecting synchronized camera data and inertial sensor data, the camera data including first image data at different timestamps, and the inertial sensor data including first vehicle motion data at different timestamps; calculating a first relative rotation matrix between image data according to the first image data at different timestamps, and calculating a second relative rotation matrix between vehicle motion data according to the first vehicle motion data at different timestamps; constructing a rotation matrix equation according to the first relative rotation matrix and the second relative rotation matrix; converting the rotation matrix equation into a quaternion equation, and constructing an overdetermined equation according to the quaternion equation, and solving a singular vector corresponding to a minimum singular value; performing matrix transformation on the singular vector to obtain the rotation matrix.

[0010] In one embodiment of the present application, a method for calibrating the translation vector includes: selecting a time sliding window comprising multiple frames, wherein the time sliding window includes second image data at different timestamps and second vehicle motion data synchronized with the second image data; initializing the visual-inertial fusion parameters and the vehicle inertial sensor bias, wherein the vehicle inertial sensor bias includes an accelerometer bias and a gyroscope bias; constructing a linear optimization cost function according to the second image data, the second vehicle motion data, and the initialized visual-inertial fusion parameters and the vehicle inertial sensor bias, wherein the linear optimization cost function includes a corner reprojection error equation, an accelerometer measurement error equation, a gyroscope measurement error equation, and a bias random walk noise equation; solving the linear optimization cost function with the goal of minimizing the linear optimization cost function to obtain the translation vector.

[0011] In one embodiment of the present application, a training method of the parking space detection model includes: obtaining an image training set, wherein the image training set includes various sample bird's-eye images in different parking environments; annotating parking space information on each sample bird's-eye view image, wherein the parking space information includes parking space corner point position coordinates, parking space entrance line coordinates, parking space status and parking space type, wherein the parking space corner point position coordinates and the parking space entrance line coordinates are coordinates on an image coordinate system; constructing an initial model and a loss function of the initial model, wherein the initial model is a target detection algorithm model under a convolutional neural network architecture, and the loss function includes a positioning loss function and a classification loss function; training the initial model using the annotated image training set to converge the initial model, and obtaining the parking space detection model when the sum of the positioning loss and the classification loss reaches a preset loss value.

[0012] In one embodiment of the present application, the method for constructing the global parking space semantic map includes: in the parking environment, obtaining a set of vehicle surrounding images through vehicle driving to generate an environment bird's-eye view image set; inputting each environment bird's-eye view image in the environment bird's-eye view image set into the parking space detection model to obtain multiple fourth parking space semantic information in the image coordinate system; map modeling of the parking environment, and converting the multiple fourth parking space semantic information into the global coordinate system to obtain global initial parking space semantic information; calculating the similarity between the two parking space semantic information in the global initial parking space semantic information, and optimizing the global initial parking space semantic information according to the similarity to obtain global target parking space semantic information; drawing parking spaces on the map according to the global target parking space semantic information to generate the global parking space semantic map.

[0013] In a second aspect, the present application provides a parking visual positioning system based on parking space semantic information, the system comprising: an acquisition module, used to acquire a bird's-eye view image of a vehicle in a parking environment and a global parking space semantic map, wherein the global parking space semantic map is constructed according to the parking environment; a detection module, used to input the bird's-eye view image into a parking space detection model for parking space detection, and obtain first parking space semantic information in an image coordinate system, wherein the parking space detection model is trained based on a sample bird's-eye view image; a conversion module, used to perform a global coordinate conversion on the first parking space semantic information according to the relative position relationship between the vehicle and the parking space in the bird's-eye view image, and obtain second parking space semantic information in a global coordinate system, wherein the global coordinate system is constructed with the parking environment as a reference; a positioning module, used to match the second parking space semantic information with the global parking space semantic map, and determine the position of the vehicle in the parking environment.

[0014] In a third aspect, the present application provides a vehicle-mounted terminal, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the vehicle-mounted terminal implements parking visual positioning based on parking space semantic information as described in the first aspect.

[0015] As described above, the parking visual positioning method, system and vehicle-mounted terminal based on parking space semantic information provided by the embodiments of the present application have the following beneficial effects:

[0016] First, a bird's-eye view image of the vehicle in the parking environment and a global parking space semantic map are obtained, wherein the global parking space semantic map is constructed according to the parking environment, and the bird's-eye view image is input into a parking space detection model for parking space detection to obtain first parking space semantic information in an image coordinate system, wherein the parking space detection model is trained based on a sample bird's-eye view image, and then the first parking space semantic information is transformed into a global coordinate system according to the relative position relationship between the vehicle and the parking space in the bird's-eye view image to obtain second parking space semantic information in a global coordinate system, wherein the global coordinate system is constructed with the parking environment as a reference, and finally the second parking space semantic information is transformed into a global coordinate system. The information is matched with the global parking space semantic map to determine the position of the vehicle in the parking environment. Combined with semantic vision technology, it effectively combines semantic understanding and efficient computing, which helps to accurately identify parking space semantic information in complex parking environments and changing lighting conditions. It uses bird's-eye view information to comprehensively capture the parking space situation around the vehicle, and under the global standard of the parking environment, matches the parking space semantic information currently corresponding to the vehicle in the parking environment with the constructed global parking space semantic map in the parking environment, which can effectively determine the position of the vehicle in the parking environment and improve the accuracy of vehicle positioning when parking.

[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0019] Figure 1 is a schematic diagram of an implementation environment of a parking vision positioning system based on parking space semantic information, shown in an exemplary embodiment of the present application;

[0020] Figure 2 is a flow chart of a parking visual positioning method based on parking space semantic information shown in an exemplary embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a calibration image shown in an exemplary embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a grayscale image shown in an exemplary embodiment of the present application;

[0023] Figure 5 is a schematic diagram of corner point recognition shown in an exemplary embodiment of the present application;

[0024] Figure 6 is a schematic diagram of a bird's-eye view image corresponding to a calibration image shown in an exemplary embodiment of the present application;

[0025] Figure 7 is a schematic diagram of a bird's-eye view image of a chessboard calibration plate shown in an exemplary embodiment of the present application;

[0026] Figure 8 is a schematic diagram of a bird's-eye view image of a vehicle in a parking environment shown by an exemplary embodiment of the present application;

[0027] Fig. 9 is a schematic diagram of a parking space detection shown in an exemplary embodiment of the present application;

[0028] Fig.10 is a schematic diagram of a global parking space semantic map shown in an exemplary embodiment of the present application;

[0029] Fig.11 is a flowchart of a specific parking visual positioning method based on parking space semantic information shown in an exemplary embodiment of the present application;

[0030] Fig.12 is a block diagram of a parking visual positioning system based on parking space semantic information shown in an exemplary embodiment of the present application;

[0031] Fig.13 It is a structural schematic diagram of a vehicle-mounted terminal provided in one embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0033] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present application in a schematic manner, and therefore the illustrations only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the form, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0034] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0035] First of all, it should be explained that parking visual positioning is a technology that uses visual sensors such as cameras to help the vehicle determine its position relative to the surrounding environment (such as parking spaces, obstacles, etc.). It can help the car accurately judge the relative distance and angle between itself and the target position, thereby achieving safe and efficient parking operations.

[0036] Traditional visual positioning mainly includes methods based on image feature point matching and target detection methods based on deep learning. Among them, the method based on image feature point matching relies on extracting and matching feature points from images for visual positioning, and the target detection method based on deep learning relies on training CNN to identify environmental features such as parking spaces and lane lines for visual positioning. The inventors of this application have found that the method based on image feature point matching cannot extract enough effective feature points in a parking environment with sparse texture and dynamically changing light, resulting in a significant decrease in positioning accuracy. In addition, the high-quality feature point extraction and matching algorithm has a high computational complexity, resulting in poor real-time positioning. The target detection method based on deep learning relies on a large amount of labeled data for training, has limited adaptability to different environments and conditions, and the reasoning process of the deep learning model requires large computing resources, and it is difficult to meet real-time requirements. Therefore, the accuracy of vehicle positioning during parking is affected by weights.

[0037] Therefore, see Figure 1 , Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a parking vision positioning system based on parking space semantic information, shown in an exemplary embodiment of the present application. Figure 1As shown, the implementation environment includes a vehicle 110 and a parking visual positioning system 120 based on parking space semantic information, wherein the parking visual positioning system 120 based on parking space semantic information is embedded in the vehicle 110, and is used to realize the parking visual positioning of the vehicle 110. The parking visual positioning system 120 based on parking space semantic information includes but is not limited to a vehicle system, a vehicle-mounted computer, etc., and by combining semantic vision technology, effectively combining semantic understanding and efficient computing, it helps to accurately identify parking space semantic information in complex parking environments and changing lighting conditions, and utilizes bird's-eye view information to comprehensively capture the parking space situation around the vehicle, and under the global standard of the parking environment, matches the parking space semantic information currently corresponding to the vehicle in the parking environment with the constructed global parking space semantic map in the parking environment, which can effectively determine the position of the vehicle in the parking environment and improve the accuracy of vehicle positioning during parking.

[0038] See also Figure 2 , Figure 2 is a flowchart of a parking visual positioning method based on parking space semantic information, which is shown in an exemplary embodiment of the present application. The method can be applied to Figure 1 The implementation environment shown is a schematic diagram. It should be understood that the method may also be applicable to other exemplary implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.

[0039] like Figure 2 As shown, in an exemplary embodiment, the parking visual positioning method based on parking space semantic information includes at least steps S210 to S240, which are described in detail as follows:

[0040] Step S210, obtaining a bird's-eye view image of the vehicle in the parking environment and a global parking space semantic map, wherein the global parking space semantic map is constructed according to the parking environment.

[0041] It should be noted that the configuration of sensors in the vehicle includes IMU (Inertial Measurement Unit), WSS (Wheel Speed ​​Sensor) and vehicle external cameras. Among them, the position of each sensor on the vehicle is not limited in the embodiment of the present application. As a possible embodiment, the IMU is installed in the center of the vehicle, the WSS is installed near the right rear wheel of the vehicle, and the vehicle external cameras include multiple cameras installed around the vehicle, such as four cameras installed on the front, rear, left and right sides of the vehicle, wherein the cameras include but are not limited to surround fisheye cameras for obtaining AVM (Around View Monitor) images around the vehicle.

[0042] Among them, the bird's eye view image, namely BEV (Bird's Eye View), is obtained based on the monitoring of the vehicle's external camera; the global parking space semantic map is constructed according to the parking environment, and different global parking space semantic maps are maintained for different parking environments.

[0043] Specifically, the method for generating a bird's-eye view image includes: calibrating the camera extrinsic parameters of the vehicle camera and capturing multiple surround view images of the vehicle at different perspectives; converting each surround view image into a bird's-eye view according to the camera extrinsic parameters to obtain multiple bird's-eye view sub-images at different perspectives; and splicing multiple bird's-eye view sub-images according to a preset bird's-eye view image size to generate a bird's-eye view image.

[0044] In this embodiment, the bird's-eye view image is obtained by converting and stitching multiple surround view images of the vehicle at different perspectives, that is, stitching the images captured by multiple cameras into a panoramic bird's-eye view. In this way, the surround view image is converted into a bird's-eye view image, which provides a bird's-eye view around the vehicle, can fully capture the parking space conditions around the vehicle, and facilitate subsequent comprehensive detection of the parking spaces around the vehicle.

[0045] In this embodiment, considering that even if it is the same model of camera, if the installation position and angle are different on different vehicles or the same vehicle, the camera extrinsic parameters will show great differences, therefore, the camera extrinsic parameters of the camera in the vehicle are recalibrated in advance, and the surround view image is converted to the bird's-eye view image through the calibrated camera extrinsic parameters, so that the accuracy of the generated bird's-eye view image can be guaranteed.

[0046] It should be noted that both the surround view image and the bird's-eye view image are recorded at a certain resolution. In the process of converting the surround view image to the bird's-eye view image, the mapping relationship between the pixels in the surround view image and the pixels in the bird's-eye view image is determined. This mapping relationship is the camera extrinsic parameter mentioned above. For example, the surround view image is recorded at a resolution of 1920×1080 pixels, and the bird's-eye view image is recorded at a resolution of 896×896 pixels, wherein each pixel in the surround view image has a mapping relationship with each pixel in the bird's-eye view image.

[0047] In addition, for the converted bird's-eye view image, in addition to the pre-set size, a coverage area is also set, that is, the entire bird's-eye view area, for example, the bird's-eye view area is an area of ​​16 meters×16 meters.

[0048] In one embodiment, a vehicle camera is calibrated for camera extrinsics, including: obtaining a preset camera intrinsic distortion coefficient of the vehicle camera, and obtaining a plurality of calibration images at different viewing angles based on a checkerboard calibration plate; for each calibration image, performing a dedistortion process according to the camera intrinsic distortion coefficient, and gray-scaling the dedistorted calibration image to obtain a grayscale image; identifying the corner points of the checkerboard in each grayscale image to obtain a plurality of sets of checkerboard corner point coordinates; and calculating the plurality of sets of checkerboard corner point coordinates and the calibration point coordinates of the corner points in the preset checkerboard calibration plate to obtain camera extrinsics.

[0049] Among them, the in-camera distortion coefficient of the vehicle camera directly uses the data corresponding to the phase itself; the checkerboard calibration plate is selected from a suitable checkerboard size and number of grids, and its black and white grids provide good contrast, geometric regularity, easy corner detection, and are suitable for various environments; the calibration point coordinates of the corner points in the preset checkerboard calibration plate, that is, the position of the corner points on the chessboard in the vehicle body coordinate system, are measured by measuring the actual distance from the real chessboard corner points to the vehicle (such as the center of the rear axle).

[0050] In this embodiment, multiple calibration images are first processed by using the camera's intrinsic distortion coefficients, and then the checkerboard corner point coordinates are identified, and the calibration point coordinates of the corner points in the preset checkerboard calibration plate are calculated to complete the camera extrinsic calibration. In this way, the accuracy and reliability of the camera extrinsic calibration are effectively improved.

[0051] Taking the surround fisheye camera as an example, the images at each stage in the camera extrinsic calibration are as follows: Figure 3-Figure 5 As shown. Among them, Figure 3 is a schematic diagram of a calibration image shown in an exemplary embodiment of the present application, that is, an original image under a fisheye camera, Figure 4 is a schematic diagram of a grayscale image shown in an exemplary embodiment of the present application, that is, an image after dedistortion and grayscale processing of a calibration image, Figure 5 It is a schematic diagram of corner point recognition shown in an exemplary embodiment of the present application, that is, an image for detecting and analyzing checkerboard corner points, wherein the checkerboard corner points are shown as red areas in the figure.

[0052] As a possible embodiment, checkerboard corner point recognition is performed on each grayscale image, including CV (Computer Vision) corner point recognition, sub-pixel level recognition and point sequence rearrangement, so that high-precision positioning of checkerboard corner points is achieved.

[0053] As a possible embodiment, the solvePnP (Perspective-n-Point Problem Solving) method may be used to calculate multiple sets of chessboard corner point coordinates and calibration point coordinates of the corner points in a preset chessboard calibration plate.

[0054] In a possible embodiment, after the camera extrinsic parameters are obtained through calibration, the camera extrinsic parameters are also verified, that is, each calibration image is converted into a bird's-eye view image according to the camera extrinsic parameters obtained through calibration, and the complete bird's-eye view image is obtained by stitching, and a standard bird's-eye view image of a checkerboard calibration plate is obtained from a top-down perspective, and the two are compared. If the comparison is successful, the accuracy and reliability of the calibration result can be ensured.

[0055] Among them, each calibration image is converted into a bird's-eye view image according to the camera extrinsic parameters obtained by calibration. Figure 6 As shown, Figure 6 is a schematic diagram of a bird's-eye view image corresponding to a calibration image according to an exemplary embodiment of the present application; the bird's-eye view images converted from each calibration image are spliced ​​to obtain a complete bird's-eye view image as shown in FIG. Figure 7 As shown, Figure 7 It is a schematic diagram of a bird's-eye view image of a chessboard calibration plate shown in an exemplary embodiment of the present application.

[0056] Step S220, inputting the bird's-eye view image into a parking space detection model to perform parking space detection, and obtaining first parking space semantic information in the image coordinate system, wherein the parking space detection model is trained based on the sample bird's-eye view image.

[0057] The first parking space semantic information includes the parking space corner position coordinates and parking space entrance line coordinates in the image coordinate system, the parking space status and the parking space type. The parking space status includes occupied or free, and the parking space type includes but is not limited to horizontal, vertical and diagonal. The detected parking space status will help the automatic parking plan select a free parking space.

[0058] In this embodiment, multi-target detection is performed on the parking space, namely, the parking space corner point, the parking space entrance line, the parking space status and the parking space type, thereby enhancing the accuracy of vehicle positioning.

[0059] See also Figure 8 , Figure 8 is a schematic diagram of a bird's-eye view image of a vehicle in a parking environment shown in an exemplary embodiment of the present application. Figure 8 The following is a bird's-eye view of the vehicle in the parking environment, which shows the vehicle and its surrounding environment from a bird's-eye view. Fig. 9 , Fig. 9 is a schematic diagram of a parking space detection shown in an exemplary embodiment of the present application, such as Fig. 9 As shown, it includes the semantic information of parking spaces around the vehicle.

[0060] Specifically, the training method of the parking space detection model includes: obtaining an image training set, the image training set includes various sample bird's-eye images in different parking environments; annotating the parking space information on each sample bird's-eye view image, the parking space information includes the parking space corner point position coordinates, the parking space entrance line coordinates, the parking space status and the parking space type, and the parking space corner point position coordinates and the parking space entrance line coordinates are coordinates on the image coordinate system; constructing an initial model and a loss function of the initial model, the initial model is a target detection algorithm model under the convolutional neural network architecture, and the loss function includes a positioning loss function and a classification loss function; using the annotated image training set to train the initial model, and performing the convergence of the initial model, when the sum of the positioning loss and the classification loss reaches a preset loss value, the parking space detection model is obtained.

[0061] In this embodiment, CNN is used for parking space detection. Based on target detection algorithms, such as YOLOv5 (You Only Look Once v5, the fifth generation of target detection algorithm), a lightweight parking space detector is developed to detect parking spaces in bird's-eye views and predict their states, as well as detailed parking space information (coordinates of each corner point of the parking space, coordinates of the entrance line of the parking space, and parking space type).

[0062] In this embodiment, the loss function includes a positioning loss function and a classification loss function, wherein the positioning loss function includes two parts: a parking space corner point positioning loss function and a parking space entrance line positioning loss function, and the classification loss function includes two parts: a parking space state loss function and a parking space type loss function, that is, the positioning loss includes the parking space corner point positioning loss and the parking space entrance line positioning loss, and the classification loss includes the parking space state loss and the parking space type loss.

[0063] As a possible embodiment, a weighted sum is performed on the parking space corner point positioning loss, the parking space entrance line positioning loss, the parking space state loss and the parking space type loss to obtain a total loss value.

[0064] In this way, based on the parking space corner points, parking space entrance lines, parking space status and parking space types, multi-target task detection training is carried out on the parking space, and multiple loss functions such as parking space corner point positioning loss, parking space entrance line positioning loss, parking space status loss and parking space type loss are integrated. This allows the model to learn the various characteristics of the parking space more comprehensively, significantly improves the detection accuracy and robustness of the parking space detection model, and further ensures the accuracy of vehicle positioning.

[0065] Step S230 , performing global coordinate transformation on the first parking space semantic information according to the relative position relationship between the vehicle and the parking space in the bird's-eye view image, and obtaining second parking space semantic information in a global coordinate system, where the global coordinate system is constructed with reference to the parking environment.

[0066] It should be noted that the first parking space semantic information identified from the bird's-eye view image is relative to the image coordinate system, while the second parking space semantic information is relative to the global coordinate system. In the embodiment of the present application, it involves an image coordinate system, a vehicle coordinate system and a global coordinate system, wherein the image coordinate system is a two-dimensional coordinate system, such as a coordinate system with the lower left corner of the bird's-eye view image as the coordinate origin, the vehicle coordinate system is a three-dimensional coordinate system, such as a coordinate system with the center of the rear axle of the vehicle as the coordinate origin, and the global coordinate system is a three-dimensional coordinate system, such as a coordinate system with the entrance of the parking environment as the coordinate origin.

[0067] Specifically, according to the relative position relationship between the vehicle and the parking space in the bird's-eye view image, the first parking space semantic information is transformed into a global coordinate system to obtain the second parking space semantic information in the global coordinate system, including: according to the relative position relationship between the vehicle and the parking space in the bird's-eye view image, the first parking space semantic information is transformed to obtain the third parking space semantic information in the vehicle body coordinate system; according to the visual inertial fusion parameters and the vehicle's forward distance, the third parking space semantic information is calculated to obtain the second parking space semantic information, the visual inertial fusion parameters include a rotation matrix and a translation vector, which are obtained by calibrating the relative position and posture relationship between the vehicle camera and the vehicle inertial sensor, and the vehicle's forward distance is the vehicle's forward distance relative to the origin of the global coordinate system, which is obtained by monitoring the wheel speed sensor.

[0068] Please continue to see Figure 8 In the bird's-eye view image, the vehicle is taken as the coordinate origin, for example, the center of the rear axle of the vehicle is taken as the origin. Based on the resolution of the bird's-eye view image, the position coordinates of each corner point of the parking space and the parking space line relative to the vehicle can be directly read, thereby obtaining the relative position relationship between the vehicle and the parking space. The first parking space semantic information and the relative position relationship are calculated to obtain the third parking space semantic information.

[0069] In addition, the semantic information of the third parking space is calculated according to the visual inertial fusion parameters and the vehicle's forward distance to obtain the semantic information of the second parking space. That is, the semantic information of the third parking space is first calculated according to the rotation matrix, and then the translation vector and the vehicle's forward distance are added to the above calculation results to obtain the semantic information of the second parking space.

[0070] Among them, the rotation matrix is ​​a 3×3 orthogonal matrix; the translation vector is a three-dimensional vector; the vehicle's forward distance is also a three-dimensional vector (a, b, c), which represents the vehicle's forward distance on the X-axis, Y-axis, and Z-axis based on the origin of the global coordinate system.

[0071] It should be noted that IMU is mainly used to measure the angular velocity and acceleration of the vehicle during movement. Since the pictures taken by the camera will be blurred when it moves too fast, the common area between the two frames will become smaller, and the visual algorithm cannot determine the direction of movement of the robot. Therefore, the use of IMU can continuously provide reliable rotation matrix and translation vector estimation. For the visual algorithm, IMU plays a complementary role.

[0072] Therefore, in this embodiment, considering that for low-texture, motion blur and occluded environments, scale information is usually lacking in pure visual SLAM (Simultaneous Localization and Mapping), and the combination of visual and inertial sensors has strong robustness, the IMU can continuously provide reliable rotation matrix and translation vector to restore the absolute scale. Therefore, multimodal sensors, namely cameras, IMU and WSS, are fused to perform parking space detection and global parking space conversion, and the parking space semantic information is fused into SLAM to form a tightly coupled SLAM system, thereby achieving high-precision positioning of the vehicle.

[0073] It should also be noted that the multimodal sensor data corresponding to the multimodal sensor are the angular velocity and acceleration measured by the IMU, the vehicle forward distance measured by the WSS, and the parking space semantic information obtained from the bird's-eye view image.

[0074] Specifically, the calibration method of the rotation matrix includes: collecting synchronized camera data and inertial sensor data, the camera data including first image data at different timestamps, and the inertial sensor data including first vehicle motion data at different timestamps; calculating a first relative rotation matrix between image data according to the first image data at different timestamps, and calculating a second relative rotation matrix between vehicle motion data according to the first vehicle motion data at different timestamps; constructing a rotation matrix equation according to the first relative rotation matrix and the second relative rotation matrix; converting the rotation matrix equation into a quaternion equation, and constructing an overdetermined equation according to the quaternion equation, and solving a singular vector corresponding to a minimum singular value; performing matrix transformation on the singular vector to obtain a rotation matrix.

[0075] In this embodiment, the first relative rotation matrix of two images of the camera at different timestamps is calculated by the five-point algorithm, which is recorded as The second relative rotation matrix between two frames at different timestamps of IMU is obtained by gyroscope integration calculation, which is recorded as for and There exists a rotation matrix R bc , so that the two establish an equation. That is and Following the equation:

[0076]

[0077] Then, for ease of calculation, the rotation matrix is ​​represented by a quaternion, and the rotation matrix equation is converted into a quaternion equation as follows:

[0078]

[0079] Define Q k,k+1 for in, is a quaternion The left multiplication matrix of is a quaternion The right multiplication matrix of .

[0080] Right now, Translates to:

[0081]

[0082] For multiple pairs of consecutive frame images, an overdetermined equation can be constructed:

[0083]

[0084] Where N represents the number of frames required for the rotation matrix to converge. It is multiple pairs of Q k,k+1 The matrix composed of N-1,b is the weight of exception handling, which can be set to 0, q bc is the quaternion to be solved.

[0085] By solving the overdetermined equation, we can find the solution to Q N The singular vector with the smallest singular value (unit singular vector) q bc , then q bc Convert to a matrix, that is, obtain the rotation matrix, denoted as R bv , to represent the attitude change of the IMU coordinate system relative to the camera coordinate system.

[0086] Specifically, the calibration method of the translation vector includes: selecting a time sliding window containing multiple frames, the time sliding window includes second image data at different timestamps and second vehicle motion data synchronized with the second image data; initializing visual inertial fusion parameters and vehicle inertial sensor bias, the vehicle inertial sensor bias includes accelerometer bias and gyroscope bias; constructing a linear optimization cost function according to the second image data, the second vehicle motion data, and the initialized visual inertial fusion parameters and vehicle inertial sensor bias, the linear optimization cost function includes a corner reprojection error equation, an accelerometer measurement error equation, a gyroscope measurement error equation and a bias random walk noise equation; solving the linear optimization cost function with the goal of minimizing the linear optimization cost function to obtain the translation vector.

[0087] In this embodiment, a set of consecutive frames are selected as the time sliding window to ensure that these frames contain sufficient image data and vehicle motion data to accurately estimate the translation vector between the camera and the IMU, denoted as t, to represent the position offset of the origin of the IMU coordinate system relative to the origin of the camera coordinate system.

[0088] As a possible embodiment, the parameters are initialized by maximum likelihood estimation, that is, minimizing the sum of the Mahalanobis norms of all measurement errors from the IMU and the monocular camera in the sliding window. After the initial parameters are obtained, a linear optimization cost function is constructed by combining the information of the IMU and visual measurements, including the corner reprojection error equation, the accelerometer measurement error equation, the gyroscope measurement error equation, and the bias random walk noise equation. The linear optimization cost function is minimized as the goal, and the translation vector between the camera and the IMU is obtained by solving it.

[0089] Step S240: Match the second parking space semantic information with the global parking space semantic map to determine the position of the vehicle in the parking environment.

[0090] Specifically, the parking space corner point position coordinates and parking space line coordinates in the second parking space semantic information are matched with all parking space corner point position coordinates and parking space line coordinates in the global parking space semantic map to determine the target parking space corner point position coordinates and target parking space line coordinates in the global parking space semantic map, that is, the target parking space currently corresponding to the vehicle is determined in the global parking space semantic map, thereby determining the position of the vehicle in the parking environment according to the target parking space.

[0091] In one embodiment, a method for constructing a global parking space semantic map includes: in a parking environment, obtaining a set of vehicle surrounding images through vehicle driving to generate an environment bird's-eye view image set; inputting each environment bird's-eye view image in the environment bird's-eye view image set into a parking space detection model to obtain multiple fourth parking space semantic information in the image coordinate system; map modeling of the parking environment, and converting the multiple fourth parking space semantic information into a global coordinate system to obtain global initial parking space semantic information; calculating the similarity between the two parking space semantic information in the global initial parking space semantic information, and optimizing the global initial parking space semantic information according to the similarity to obtain global target parking space semantic information; drawing parking spaces on the map according to the global target parking space semantic information to generate a global parking space semantic map.

[0092] The method of converting the plurality of fourth parking space semantic information on the image coordinate system to the global coordinate system is the same as the aforementioned conversion method, which will not be described in detail here.

[0093] In this embodiment, in order to avoid repeated detection and drawing of each parking space, the similarity between each parking space is calculated. If the similarity exceeds a first threshold, the semantic information of one of the parking spaces is discarded and not drawn on the map. If the similarity is lower than the first threshold, the semantic information of both parking spaces will be drawn on the map, thereby ensuring the reliability of the global parking space semantic map.

[0094] It should be noted that for each parking environment, a global parking space semantic map is drawn in advance, that is, the vehicle is driven in the entire parking environment, and the camera is continuously used to capture the surrounding images of the vehicle to draw a global parking space semantic map. First, the vehicle enters the parking environment, and the global coordinate system is initialized at the beginning of the vehicle trajectory. The system records the current position as the coordinate origin of the global coordinate system. Then, during the driving process, the vehicle uses the wheel speed pulse and IMU signal to infer the vehicle track, and the parking environment is mapped according to the vehicle trajectory, including but not limited to the construction of lanes, obstacles, etc. At the same time, the vehicle continuously uses the camera to capture the surrounding images of the vehicle during the driving process, calculates the semantic information of the parking space in the global coordinate system, and completes the parking space drawing on the map according to the semantic information. After the vehicle's driving trajectory covers the entire parking environment, it is considered that all parking spaces have been drawn, thereby obtaining a global parking space semantic map.

[0095] As a possible embodiment, the vehicle drives multiple times in the parking environment to complete map modeling and parking space drawing to ensure the accuracy of the global parking space semantic map.

[0096] In addition, in the global parking space semantic map, each parking space can be regarded as a different rectangle at a different position in the global coordinate system. In order to solve the allocation problem of each parking space, in a possible embodiment, a multi-target tracking framework SORT (Simple Online and Realtime Tracking) is used to manage each parking space.

[0097] In parking space mapping, in order to assign detections to existing parking spaces, the tracker state needs to be updated in a timely manner, the bounding box geometry of each target is predicted through a Kalman filter, and the assignment cost matrix of the IoU (Intersection over Union) distance between the detection and all predicted bounding boxes of existing targets is calculated, and the assignment problem is solved by the Hungarian algorithm. In addition, a minimum IoU is set to reject assignments where the detection overlaps the target less than 0.3, and the matched parking space detection will share the same id as the existing parking space. A hard match association method is also implemented and evaluated, which creates predefined thresholds for previous parking space associations and new parking spaces.

[0098] In this way, the status of all parking spaces is robustly tracked and maintained under the multi-target tracking framework to achieve improved parking space positioning accuracy, which helps the downstream tasks of automatic parking.

[0099] See also Fig.10 , Fig.10 FIG. 1 is a schematic diagram of a global parking space semantic map shown in an exemplary embodiment of the present application. Fig.10 As shown, a global parking space semantic map corresponding to an exemplary parking environment is shown, that is, a global parking space semantic map obtained by drawing parking spaces based on parking space semantic information on the basis of parking environment map modeling.

[0100] After the global parking space semantic map is generated, when the vehicle enters the parking environment again, if the APA parking space function is activated, the parking visual positioning of the vehicle can be achieved according to the above steps S210 to S240.

[0101] In a possible embodiment, the vehicle will record the real-time parking space semantic information obtained during the positioning process and modify the global parking space semantic map.

[0102] In one possible embodiment, a multi-map multiplexing system is used to effectively cope with long-term and weak-texture environments. If the system loses positioning, it can restart the map and seamlessly merge with the original map when passing through previously visited locations, and can utilize information from long ago or from different maps.

[0103] See also Fig.11 , Fig.11FIG. 1 is a flowchart of a specific parking visual positioning method based on parking space semantic information, shown in an exemplary embodiment of the present application. Fig.11 As shown, first, a multimodal sensor is configured, namely, a camera, an IMU, and a WSS; visual-inertial calibration is performed on the camera and the IMU to obtain the rotation matrix and translation vector between the camera and the IMU. For the WSS, the vehicle's forward distance is monitored when the vehicle is driving in a parking environment, and for the camera, multiple surround images are captured; during parking visual positioning, multiple surround images are converted according to the camera extrinsics obtained by the camera-vehicle calibration to generate a bird's-eye view image; and parking space detection is performed on the bird's-eye view image, and parking space semantic information in the image coordinate system is output; then, the parking space semantic information in the image coordinate system is converted into parking space semantic information in the vehicle body coordinate system, and then, combined with the rotation matrix and translation vector between the camera and the IMU and the vehicle's forward distance, the parking space semantic information in the vehicle body coordinate system is converted into parking space semantic information in the global coordinate system; then Kalman filter optimization is performed; finally, loop detection, mapping, and positioning are performed to complete the vehicle's parking visual positioning.

[0104] The parking visual positioning method based on parking space semantic information first obtains a bird's-eye view image of the vehicle in the parking environment and a global parking space semantic map, wherein the global parking space semantic map is constructed according to the parking environment, and the bird's-eye view image is input into a parking space detection model for parking space detection to obtain first parking space semantic information in an image coordinate system, wherein the parking space detection model is trained based on a sample bird's-eye view image, and then the first parking space semantic information is converted into a global coordinate system based on the relative position relationship between the vehicle and the parking space in the bird's-eye view image to obtain second parking space semantic information in a global coordinate system, wherein the global coordinate system is constructed based on the parking environment as a reference. Finally, the second parking space semantic information is matched with the global parking space semantic map to determine the position of the vehicle in the parking environment. Combined with semantic vision technology, it effectively combines semantic understanding and efficient computing, which helps to accurately identify parking space semantic information in complex parking environments and changing lighting conditions. It uses bird's-eye view information to comprehensively capture the parking space situation around the vehicle, and under the global standard of the parking environment, matches the parking space semantic information currently corresponding to the vehicle in the parking environment with the constructed global parking space semantic map in the parking environment, which can effectively determine the position of the vehicle in the parking environment and improve the accuracy of vehicle positioning during parking.

[0105] In this way, through a more accurate, robust and widely applicable parking visual positioning method, it not only promotes the development of automatic parking technology, but also provides a more efficient and safe solution for intelligent transportation and urban management.

[0106] See also Fig.12 , Fig.12is a block diagram of a parking visual positioning system based on parking space semantic information, shown in an exemplary embodiment of the present application. The system can be applied to Figure 1 The implementation environment shown is a schematic diagram. It should be understood that the system may also be applicable to other exemplary implementation environments, and this embodiment does not limit the implementation environment to which the system is applicable.

[0107] like Fig.12 As shown, in an exemplary embodiment, a parking visual positioning system 1200 based on parking space semantic information includes at least an acquisition module 1210, a detection module 1220, a conversion module 1230 and a positioning module 1240, which are described in detail as follows:

[0108] An acquisition module 1210 is used to acquire a bird's-eye view image of the vehicle in the parking environment and a global parking space semantic map in the parking environment, where the global parking space semantic map is constructed according to the parking environment;

[0109] A detection module 1220 is used to input the bird's-eye view image into a parking space detection model to perform parking space detection and obtain first parking space semantic information in the image coordinate system. The parking space detection model is trained based on the sample bird's-eye view image;

[0110] The conversion module 1230 is used to perform global coordinate conversion on the first parking space semantic information according to the relative position relationship between the vehicle and the parking space in the bird's-eye view image, so as to obtain the second parking space semantic information in the global coordinate system, where the global coordinate system is constructed with the parking environment as a reference;

[0111] The positioning module 1240 is used to match the second parking space semantic information with the global parking space semantic map to determine the position of the vehicle in the parking environment.

[0112] It should be noted that the parking visual positioning system based on parking space semantic information provided in the above embodiment and the parking visual positioning method based on parking space semantic information provided in the above embodiment belong to the same concept, and the contents of the operations performed by each module have been described in detail in the method embodiment and will not be repeated here.

[0113] See also Fig.13 , Fig.13 It is a structural schematic diagram of a vehicle-mounted terminal provided in one embodiment of the present application. Fig.13 The structure diagram of the computer system of the vehicle terminal suitable for implementing the embodiment of the present application is shown. It should be noted that: Fig.13 The computer system 1300 of the vehicle-mounted terminal shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0114] like Fig.13As shown, the computer system 1300 includes a central processing unit (CPU) 1301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1302 or the program loaded from the storage part 1308 to the random access memory (RAM) 1303, such as executing the method in the above embodiment. In RAM 1303, various programs and data required for system operation are also stored. CPU 1301, ROM 1302 and RAM 1303 are connected to each other through bus 1304. Input / output (I / O) interface 1305 is also connected to bus 1304.

[0115] The following components are connected to the I / O interface 1305: an input section 1306 including a keyboard, a mouse, etc.; an output section 1307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the I / O interface 1305 as needed. A removable medium 1311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1310 as needed so that a computer program read therefrom is installed into the storage section 1308 as needed.

[0116] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1309, and / or installed from a removable medium 1311. When the computer program is executed by a central processing unit (CPU) 1301, various functions defined in the system of the present application are executed.

[0117] The flowchart and block diagram in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0118] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0119] The above embodiments are merely illustrative of the principles and effects of the present application, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A parking visual positioning method based on parking space semantic information, characterized in that: The method comprises: Acquire a bird's-eye view image of the vehicle in a parking environment and a global parking space semantic map, wherein the global parking space semantic map is constructed according to the parking environment; Inputting the bird's-eye view image into a parking space detection model to perform parking space detection, and obtaining first parking space semantic information in an image coordinate system, wherein the parking space detection model is trained based on a sample bird's-eye view image; According to the relative position relationship between the vehicle and the parking space in the bird's-eye view image, performing global coordinate transformation on the first parking space semantic information to obtain second parking space semantic information in a global coordinate system, wherein the global coordinate system is constructed with reference to the parking environment; The second parking space semantic information is matched with the global parking space semantic map to determine the position of the vehicle in the parking environment.

2. The parking visual positioning method based on parking space semantic information according to claim 1 is characterized in that: The method for generating the bird's-eye view image includes: Calibrate the vehicle camera’s external parameters and capture multiple surround view images of the vehicle at different viewing angles; According to the camera extrinsic parameters, each of the surround view images is converted into a bird's-eye view to obtain a plurality of bird's-eye view sub-images at different viewing angles; According to a preset bird's-eye view image size, a plurality of the bird's-eye view sub-images are spliced ​​to generate the bird's-eye view image.

3. The parking visual positioning method based on parking space semantic information according to claim 2 is characterized in that: The step of calibrating the camera extrinsic parameters of the vehicle camera includes: Obtain the preset in-camera distortion coefficient of the vehicle camera, and obtain multiple calibration images at different viewing angles based on the checkerboard calibration plate; For each of the calibration images, dedistortion processing is performed according to the distortion coefficients involved in the camera, and the dedistorted calibration image is gray-scaled to obtain a gray-scale image; Identifying corner points of the checkerboard in each of the grayscale images to obtain multiple sets of coordinates of the checkerboard corner points; The camera extrinsic parameters are obtained by calculating the coordinates of multiple sets of the chessboard corner points and the calibration point coordinates of the preset chessboard corner points in the calibration plate.

4. The parking visual positioning method based on parking space semantic information according to claim 1, characterized in that: The performing global coordinate transformation on the first parking space semantic information according to the relative position relationship between the vehicle and the parking space in the bird's-eye view image to obtain the second parking space semantic information in the global coordinate system includes: According to the relative position relationship between the vehicle and the parking space in the bird's-eye view image, the first parking space semantic information is transformed to obtain third parking space semantic information in the vehicle body coordinate system; The third parking space semantic information is calculated according to the visual inertial fusion parameters and the vehicle's forward distance to obtain the second parking space semantic information. The visual inertial fusion parameters include a rotation matrix and a translation vector, which are obtained by calibrating the relative position and posture relationship between the vehicle camera and the vehicle inertial sensor. The vehicle's forward distance is the vehicle's forward distance relative to the origin of the global coordinate system, which is obtained by monitoring the wheel speed sensor.

5. The parking visual positioning method based on parking space semantic information according to claim 4 is characterized in that: The calibration method of the rotation matrix includes: Collecting synchronized camera data and inertial sensor data, wherein the camera data includes first image data at different timestamps, and the inertial sensor data includes first vehicle motion data at different timestamps; Calculating a first relative rotation matrix between image data according to the first image data at different timestamps, and calculating a second relative rotation matrix between vehicle motion data according to the first vehicle motion data at different timestamps; Constructing a rotation matrix equation according to the first relative rotation matrix and the second relative rotation matrix; Convert the rotation matrix equation into a quaternion equation, and construct an overdetermined equation based on the quaternion equation to solve the singular vector corresponding to the minimum singular value; Perform matrix transformation on the singular vector to obtain the rotation matrix.

6. The parking visual positioning method based on parking space semantic information according to claim 4 is characterized in that: The calibration method of the translation vector includes: selecting a time sliding window comprising a plurality of frames, wherein the time sliding window comprises second image data at different timestamps and second vehicle motion data synchronized with the second image data; Initializing the visual inertial fusion parameters and the vehicle inertial sensor bias, wherein the vehicle inertial sensor bias includes an accelerometer bias and a gyroscope bias; Constructing a linear optimization cost function according to the second image data, the second vehicle motion data, the initialized visual inertial fusion parameters, and the vehicle inertial sensor bias, wherein the linear optimization cost function includes a corner reprojection error equation, an accelerometer measurement error equation, a gyroscope measurement error equation, and a bias random walk noise equation; With the goal of minimizing the linear optimization cost function, the linear optimization cost function is solved to obtain the translation vector.

7. The parking visual positioning method based on parking space semantic information according to any one of claims 1 to 6, characterized in that: The training method of the parking space detection model includes: Acquire an image training set, wherein the image training set includes various sample bird's-eye view images in different parking environments; Annotate the parking space information on each sample bird's-eye view image, wherein the parking space information includes parking space corner point position coordinates, parking space entrance line coordinates, parking space status and parking space type, wherein the parking space corner point position coordinates and the parking space entrance line coordinates are coordinates on the image coordinate system; Constructing an initial model and a loss function of the initial model, wherein the initial model is a target detection algorithm model under a convolutional neural network architecture, and the loss function includes a positioning loss function and a classification loss function; The initial model is trained using the labeled image training set to achieve convergence of the initial model, and when the sum of the positioning loss and the classification loss reaches a preset loss value, the parking space detection model is obtained.

8. The parking visual positioning method based on parking space semantic information according to claim 7 is characterized in that: The method for constructing the global parking space semantic map includes: In the parking environment, a vehicle surrounding image set is acquired by driving the vehicle to generate an environment bird's-eye view image set; Inputting each environment bird's-eye view image in the environment bird's-eye view image set into the parking space detection model to obtain a plurality of fourth parking space semantic information in the image coordinate system; Performing map modeling on the parking environment, and transforming the plurality of fourth parking space semantic information into a global coordinate system to obtain global initial parking space semantic information; Calculating the similarity between two parking space semantic information in the global initial parking space semantic information, and optimizing the global initial parking space semantic information according to the similarity to obtain global target parking space semantic information; The parking spaces are drawn on the map according to the global target parking space semantic information to generate the global parking space semantic map.

9. A parking visual positioning system based on parking space semantic information, characterized in that: The system comprises: An acquisition module, used to acquire a bird's-eye view image of the vehicle in the parking environment and a global parking space semantic map, wherein the global parking space semantic map is constructed according to the parking environment; A detection module, used for inputting the bird's-eye view image into a parking space detection model to perform parking space detection and obtain first parking space semantic information in an image coordinate system, wherein the parking space detection model is trained based on a sample bird's-eye view image; a conversion module, configured to perform global coordinate conversion on the first parking space semantic information according to the relative position relationship between the vehicle and the parking space in the bird's-eye view image, so as to obtain second parking space semantic information in a global coordinate system, wherein the global coordinate system is constructed with reference to the parking environment; A positioning module is used to match the second parking space semantic information with the global parking space semantic map to determine the position of the vehicle in the parking environment.

10. A vehicle-mounted terminal, characterized in that: include: one or more processors; A storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the vehicle-mounted terminal implements the parking visual positioning method based on parking space semantic information as described in any one of claims 1 to 8.

Images