Visual mapping positioning method and cloud platform and storage medium thereof

Abstract

The application relates to a visual mapping positioning method and a cloud platform and a storage medium thereof, which comprises the following steps: receiving and analyzing a data packet uploaded by a vehicle end to obtain environment images, inertial navigation data and wheel speed data at multiple sampling moments, obtaining second pose information according to the environment images, the inertial navigation data and the wheel speed data, obtaining 3D space position information corresponding to feature points of the environment images at each sampling moment according to the second pose information, and obtaining map data of the data packet according to all key frame images and the 3D space position information corresponding to the feature points of the key frame images; waiting for a preset time, if a new data packet is received within the preset time, the new data packet is continuously processed, if no new data packet is received within the preset time, mapping is ended, map data of all data packets is generated into a map file, and the map file is sent to the vehicle end to be used for vehicle positioning by the vehicle end. Through the application, the problems of cost, software algorithm complexity, operation efficiency and the like of the visual mapping positioning function can be solved.

Description

Technical Field

[0001] This application relates to the field of automatic parking technology, specifically to a visual mapping and positioning method and its cloud platform and storage medium. Background Technology

[0002] In intelligent driving applications, a crucial technical approach involves using various sensors (such as image sensors, LiDAR, and inertial navigation components) to collect scene data (e.g., images, point clouds). Simultaneous localization and mapping (SLAM) techniques are then used to map the scene, generating a digital description. Real-time localization is then performed based on the mapping results, providing scene positioning and environmental perception support for subsequent processes such as path planning and driving control in the scene. Mapping and localization are particularly important in current intelligent parking applications, which often operate in indoor parking lots. In such relatively closed and isolated environments, positioning methods like satellite navigation and wireless navigation often fail under unstable network communication conditions. Therefore, using the vehicle's own sensors and corresponding mapping and localization algorithms to calculate flight paths is often the preferred method.

[0003] Currently, the mapping and positioning functions are mainly implemented in the following ways: (1) Mapping and localization scheme based on LiDAR / LiDAR scanner. This scheme provides direct 3D spatial data (point cloud) of the scene. However, the scene expressed by this data format is generally large in volume, which will put a heavy burden on the computing platform. Moreover, since the data itself is mostly stored and processed in the form of point lists, unlike images, there is not much additional organizational structure to utilize. Therefore, most data processing, including the operation efficiency of mapping and localization algorithms, will be relatively slow. This also requires more powerful computing power to make up for the deficiency. Therefore, for slightly lower-end computing platforms, using LiDAR for mapping and localization may have significant operational efficiency problems. In addition, LiDAR equipment is a professional device and its price is relatively expensive. For low-end models with limited costs, developing intelligent driving functions based on mapping and localization on such equipment will deviate from the market positioning of the product itself.

[0004] (2) Mapping and localization scheme based on vehicle-mounted image sensors. This scheme cannot directly acquire 3D spatial structure information of the scene like LiDAR, so it needs to rely on software algorithm technology such as motion-to-structure and cluster optimization, or auxiliary sensors (such as inertial navigation elements), or even limited externally available maps and positioning data (such as high-precision map data, or positioning data from satellite navigation or wireless navigation). The development complexity of such software algorithms is much higher than that of LiDAR. Moreover, since the process of recovering 3D spatial structure information from 2D images involves a state estimation process, estimation errors will inevitably be introduced. Moreover, due to the limitations of the 2D image acquisition process, the estimation results will be larger than those of LiDAR. Therefore, the accuracy of the mapping and localization scheme based on image sensors will be far lower than that of the LiDAR-based scheme. Or conversely, if we want the mapping and localization scheme based on image sensors to approach the accuracy of the LiDAR-based scheme, more complex software algorithms and more powerful computing platforms are required. Summary of the Invention

[0005] The purpose of this application is to propose a visual mapping and localization method, its cloud platform, and storage medium to solve the problem that it is difficult to simultaneously balance the cost, software algorithm complexity, and operating efficiency of visual mapping and localization functions.

[0006] To achieve the above objectives, this application proposes a visual mapping and localization method, the method comprising: The system receives and parses data packets uploaded from the vehicle to obtain environmental images, inertial navigation data, and wheel speed data at multiple sampling times. Feature points are extracted from the environmental images at the multiple sampling times sequentially, and it is determined whether the environmental image is a keyframe image based on the extracted feature points; if it is a keyframe image, the first pose information corresponding to each sampling time is obtained based on its feature points; if it is not a keyframe image, it is discarded. The second pose information is obtained by Kalman filtering and fusing the first pose information, inertial navigation data and wheel speed data corresponding to each keyframe image. The 3D spatial position information corresponding to the feature points of the environmental image at each sampling time is obtained based on the second pose information. The map data of the data packet is obtained based on all keyframe images and the 3D spatial position information corresponding to their feature points. Waiting for a preset time, if a new data packet uploaded by the vehicle is received within the preset time, the new data packet is processed to obtain the map data of the new data packet. If no new data packet uploaded by the vehicle is received within the preset time, the mapping ends, the map data of all data packets is used to generate a map file, and the map file is sent to the vehicle for vehicle positioning.

[0007] Optionally, the step of sequentially extracting feature points from the environmental images at the plurality of sampling times, and determining whether the environmental image is a keyframe image based on the extracted feature points, includes: SIFT feature points are extracted from the environment image. SIFT bag-of-words image representation information is generated based on the SIFT feature points. The SIFT bag-of-words image representation information of the environment image is compared with the SIFT bag-of-words image representation information of keyframe images in the keyframe database. Based on the comparison result, it is determined whether the environment image is a keyframe image.

[0008] Optionally, if the image is a keyframe image, then the first pose information corresponding to each sampling time is obtained based on its feature points, including: If it is a keyframe image, then based on its SIFT bag-of-words image representation information, it is matched with the SIFT bag-of-words image representation information of other keyframe images to determine all keyframe images that have a co-view relationship with it. Based on its SIFT feature points, it is matched with the SIFT feature points of all keyframe images that have a co-view relationship with it to determine SIFT feature point pairs. Based on the SIFT feature point pairs, the vehicle pose is estimated to obtain the first pose information. And, it is added to the keyframe database.

[0009] Optionally, the step of sequentially extracting feature points from the environmental images at the plurality of sampling times further includes: Extract ORB feature points from the environmental image, and generate ORB bag-of-words image representation information based on the ORB feature points; The step of obtaining the map data of the data packet based on the 3D spatial location information corresponding to all keyframe images and their feature points includes: All keyframe images and their corresponding 3D spatial location information, ORB feature points, and ORB bag-of-words image representation information are output as map data for the data package.

[0010] Optionally, the map file is used for vehicle positioning on the vehicle side, including: After receiving the map file, the vehicle parses the map file to obtain multiple keyframe images and their corresponding 3D spatial location information, ORB feature points, and ORB bag-of-words image representation information. The vehicle-side acquires the current environmental image, inertial navigation data, and wheel speed data according to a preset sampling period, extracts the ORB feature points of the current environmental image, and generates the corresponding ORB bag-of-words image representation information based on the ORB feature points. The vehicle matches the ORB bag-of-words image representation information corresponding to the current environment image with the ORB bag-of-words image representation information of the multi-frame keyframe images to determine all keyframe images that have a co-view relationship with the current environment image. It then matches the ORB feature points of the current environment image with the ORB feature points of all keyframe images that have a co-view relationship with it to determine ORB feature point pairs. Finally, it estimates the vehicle pose based on the ORB feature point pairs and the 3D spatial position information to obtain the third pose information. The fourth pose information is obtained by fusing the third pose information, inertial navigation data and wheel speed data corresponding to each keyframe image with Kalman filtering, and the fourth pose information is output as the current pose of the vehicle.

[0011] Optionally, before performing the step of sequentially extracting feature points from the environmental images at the plurality of sampling times, the method further includes: The unique identifier of the data packet is obtained by parsing the data packet. Based on the unique identifier, it is determined whether a data packet is lost. If so, a data retransmission request is sent to the vehicle to notify the vehicle to retransmit the lost data packet. The unique identifier of the data packet corresponds one-to-one with the order in which the data packets are sent by the vehicle. The environmental images, inertial navigation data, and wheel speed data at the multiple sampling times are verified, including verifying the integrity of the environmental images and the continuity of the inertial navigation data and wheel speed data at the multiple sampling times.

[0012] This application also proposes a cloud platform, which includes: The data receiving unit is used to receive and parse the data packets uploaded by the vehicle to obtain environmental images, inertial navigation data and wheel speed data at multiple sampling times; The image processing unit is used to extract feature points from the environmental images at the multiple sampling times in sequence, and determine whether the environmental image is a keyframe image based on the extracted feature points; if it is a keyframe image, the first pose information corresponding to each sampling time is obtained based on its feature points; if it is not a keyframe image, it is discarded. The fusion computing unit is used to perform Kalman filtering fusion on the first pose information, inertial navigation data, and wheel speed data corresponding to each keyframe image to obtain the second pose information, and to obtain the 3D spatial position information corresponding to the feature points of the environmental image at each sampling time based on the second pose information, and to obtain the map data of the data packet based on all keyframe images and the 3D spatial position information corresponding to their feature points; and The map generation unit waits for a preset time. If a new data packet is received from the vehicle within the preset time, the unit continues to process the new data packet to obtain map data for the new data packet. If no new data packet is received from the vehicle within the preset time, the mapping ends, map files are generated from the map data of all data packets, and the map files are sent to the vehicle for vehicle positioning.

[0013] Optionally, the image processing unit is configured to: SIFT feature points are extracted from the environment image, SIFT bag-of-words image representation information is generated based on the SIFT feature points, and the SIFT bag-of-words image representation information of the environment image is compared with the SIFT bag-of-words image representation information of keyframe images in the keyframe database. Based on the comparison result, it is determined whether the environment image is a keyframe image. If it is a keyframe image, then based on its SIFT bag-of-words image representation information, it is matched with the SIFT bag-of-words image representation information of other keyframe images to determine all keyframe images that have a co-view relationship with it. Based on its SIFT feature points, it is matched with the SIFT feature points of all keyframe images that have a co-view relationship with it to determine SIFT feature point pairs. Based on the SIFT feature point pairs, the vehicle pose is estimated to obtain the first pose information. And, it is added to the keyframe database.

[0014] Optionally, the image processing unit is further configured to extract ORB feature points from the environmental image and generate ORB bag-of-words image representation information based on the ORB feature points; The fusion computing unit is used to output the 3D spatial location information, ORB feature points, and ORB bag-of-words image representation information corresponding to all keyframe images and their SIFT feature points as map data of the data packet.

[0015] This application also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the visual mapping and localization method described above.

[0016] The method, cloud platform, and computer storage medium proposed in this application have at least the following beneficial effects: The system filters multiple frames of environmental images acquired by the vehicle, extracts feature points from these images, discards useless environmental images, and processes only keyframe images that are essential for mapping, reducing data processing volume. Then, it obtains initial first pose information based on the extracted feature points, and further fuses this first pose information with inertial navigation data and wheel speed data acquired by the vehicle to obtain second pose information, improving the accuracy of the pose information. Finally, it recovers the 3D spatial position information corresponding to the feature points based on the second pose information. The resulting map file contains keyframe images, their feature points, and 3D spatial position information. During subsequent vehicle-side positioning, the real-time position of the vehicle can be determined based on the matching results between the real-time environmental images acquired by the vehicle and the keyframe images in the map file, as well as the 3D spatial position information. The software algorithm of this embodiment is simple, requires less data processing, and can be implemented using only simple image sensors, inertial navigation elements, and wheel speedometers. Therefore, it can solve the problem of simultaneously balancing the cost, software algorithm complexity, and operating efficiency of visual mapping and positioning functions.

[0017] Other features and advantages of the embodiments of this application will be set forth in the following description. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a visual mapping and localization method in one embodiment of this application.

[0020] Figure 2 This is a flowchart illustrating the data transmission process between the cloud platform and the vehicle in one embodiment of this application.

[0021] Figure 3 This is a flowchart illustrating the vehicle-side positioning process in one embodiment of this application.

[0022] Figure 4 This is a schematic diagram of the structure of a cloud platform according to another embodiment of this application. Detailed Implementation

[0023] The various exemplary embodiments, features, and aspects of this application will be described in detail below with reference to the accompanying drawings. Furthermore, numerous specific details are set forth in the following detailed embodiments to better illustrate this application. Those skilled in the art will understand that this application can be practiced without certain specific details. In some instances, means well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0024] One embodiment of this application proposes a visual mapping and localization method, see reference. Figure 1 and 2 The method in this embodiment includes the following steps: Step S100: Receive and parse the data packets uploaded by the vehicle to obtain environmental images, inertial navigation data and wheel speed data at multiple sampling times.

[0025] Specifically, when mapping and positioning are required, the vehicle-mounted system triggers the cloud platform to begin mapping. The vehicle connects to the cloud platform and notifies it to open the data upload channel. The vehicle then begins acquiring environmental images, inertial navigation data, and wheel speed data and storing them in a cache. The vehicle is equipped with an image sensor, inertial navigation element, and wheel speedometer. These are standard features in most vehicle models. The image sensor is used to collect environmental images and sampling timestamps (usually video streams) around the vehicle. The inertial navigation element is used to sample vehicle inertial navigation data according to a preset sampling period. This inertial navigation data includes yaw angle, pitch angle, roll angle, lateral acceleration, and so on. The wheel speed meter is used to sample vehicle wheel speed and sampling timestamps according to a preset sampling period. When the cache is full, that is, when the environmental images, inertial navigation data, and wheel speed data in the cache reach the set upper limit of the cache capacity, the vehicle end encapsulates all the environmental images, inertial navigation data, and wheel speed data in the cache to generate a data packet, and then uploads the data packet to the cloud platform. During the mapping process, the vehicle end continuously acquires environmental images, inertial navigation data, and wheel speed data, and generates corresponding data packets and uploads them to the server when the cache is full. Therefore, the number of data packets depends on the amount of data required for mapping and the cache size.

[0026] Specifically, the cloud platform receives and parses the data packets uploaded by the vehicle to obtain environmental images, inertial navigation data, and wheel speed data at multiple sampling times. It should be noted that when the cloud platform processes the first data packet, the vehicle simultaneously acquires the data of the second data packet. The method in this embodiment sends the data in segments to improve operational efficiency. If the vehicle generates a unique data packet and sends it to the cloud platform for processing after collecting all the data, the cloud platform will have to wait for a long time, resulting in low operational efficiency.

[0027] Step S200: Extract feature points from the environmental images at the multiple sampling times in sequence, and determine whether the environmental image is a keyframe image based on the extracted feature points; if it is a keyframe image, obtain the first pose information corresponding to each sampling time based on its feature points; if it is not a keyframe image, discard it.

[0028] Specifically, the method in this embodiment filters environmental images collected at multiple sampling times from the vehicle end, extracts feature points from these images, and uses any algorithm for feature point extraction without restriction. The environmental image at the first sampling time is assumed to be a keyframe image. Based on the extracted features, the environmental image at the second sampling time is compared with this keyframe image to determine the differences between the two images. If the differences reach a certain level, the environmental image at the second sampling time is also a keyframe image; otherwise, it is not. This process continues, with subsequent environmental images at each sampling time being compared with all previously determined keyframe images. Assuming there are n keyframe images, the comparison will yield n image differences. When the differences among the n images reach a certain level, it is determined to be a keyframe image; otherwise, it is not.

[0029] It should be noted that, due to the large amount of video stream data, the similarity between adjacent frames is high when the vehicle speed is low. Therefore, the method in this embodiment filters key frame images. If it is not a key frame image, the environmental image that is not useful for mapping is discarded. Only the key frame images that have a substantial role in mapping are processed to reduce the amount of data processing.

[0030] The step of obtaining the first pose information corresponding to each sampling moment based on its feature points can be implemented using the SFM (Structure from Motion) algorithm and the PnP (Perspective-n-Point) algorithm. The SFM algorithm is a 3D reconstruction algorithm that can recover the camera pose and reconstruct 3D coordinate points from two or more scenes (images). PnP (Perspective-n-Point) is a method for solving the motion of 3D to 2D point pairs. It requires knowing n 3D spatial points and their projected positions to estimate the camera pose (i.e., the first pose information). Therefore, the SFM algorithm can be used in the initial stage, followed by the PnP algorithm.

[0031] Step S300: Perform Kalman filtering fusion on the first pose information, inertial navigation data and wheel speed data corresponding to each keyframe image to obtain the second pose information, and obtain the 3D spatial position information corresponding to the feature points of the environmental image at each sampling time based on the second pose information, and obtain the map data of the data packet based on all keyframe images and the 3D spatial position information corresponding to their feature points.

[0032] Specifically, Kalman filtering is an algorithm that uses the state equations of a linear system to optimally estimate the system state using system input and output observation data, and can be used to achieve multi-source data fusion. The first pose information is the vehicle pose estimated based on the vehicle's environmental image, and the second pose information can be understood as the vehicle pose that has been corrected by considering inertial navigation data and wheel speed data. The vehicle pose includes the vehicle position and the vehicle heading angle.

[0033] Specifically, obtaining the 3D spatial position information corresponding to the feature points of the environmental image at each sampling time based on the second pose information can be achieved by triangulating the feature points based on the second pose information to recover the 3D spatial position of the feature points. It should be noted that feature point triangulation is a fundamental problem in VSLAM. It recovers the 3D coordinates of feature points based on their projections under multiple cameras. When a feature point is observed by a camera, an observation "ray" originating from the center of the camera can be obtained in 3D space based on the camera pose and observation vector. Multiple camera pose observations will generate multiple observation rays. Ideally, these observation rays intersect at a point in space, and the intersection of all observation rays is the 3D spatial position of the feature point. In this embodiment, the same camera in different pose states is equivalent to multiple cameras because the camera position changes in different pose states.

[0034] In addition, the coordinates of the 3D spatial position of the feature points are determined based on the vehicle spatial coordinate system, the origin of which is the vehicle position corresponding to the first sampling time. Furthermore, the 3D spatial location information corresponding to the feature points of each keyframe image can be bundled for optimization to reduce the overall error.

[0035] Step S400: Wait for a preset time. If a new data packet uploaded by the vehicle is received within the preset time, continue to process the new data packet to obtain the map data of the new data packet. If no new data packet uploaded by the vehicle is received within the preset time, end the mapping, generate a map file from the map data of all data packets, and send the map file to the vehicle for vehicle positioning.

[0036] Specifically, after receiving the map file, the vehicle parses the map file to obtain multiple keyframe images and their corresponding 3D spatial location information. The vehicle acquires the current environmental image, inertial navigation data, and wheel speed data according to a preset sampling period, and extracts the ORB feature points of the current environmental image. The vehicle matches the feature points of the current environmental image with the feature points of all keyframe images that have a co-view relationship with it to determine feature point pairs. Based on these feature point pairs and the 3D spatial location information contained in the map file, the vehicle's pose is estimated to obtain third pose information. Kalman filtering is performed on the third pose information, inertial navigation data, and wheel speed data corresponding to each keyframe image to obtain fourth pose information. This fourth pose information is output as the vehicle's current pose, including the vehicle's current position and current heading angle, thus achieving localization.

[0037] Based on the above description of the method in this embodiment, it can be seen that the software algorithm of this embodiment is simple, filters video stream images, processes only key frame images, and has a small amount of data processing; this embodiment can be implemented based on simple image sensors, inertial navigation elements, wheel speed meters, etc., and most intelligent vehicles are equipped with image sensors, inertial navigation elements, and wheel speed meters, requiring no additional hardware costs, thus having great application value and being suitable for promotion; in this embodiment, cloud platform data processing and vehicle-side sampling data can be performed simultaneously (see...). Figure 2 This improves operational efficiency; therefore, it can solve the problem of simultaneously balancing the cost of visual mapping and positioning functions, the complexity of software algorithms, and operational efficiency.

[0038] In this embodiment, optionally, the step of sequentially extracting feature points from the environmental images at the plurality of sampling times and determining whether the environmental image is a keyframe image based on the extracted feature points includes: SIFT feature points are extracted from the environment image. SIFT bag-of-words image representation information is generated based on the SIFT feature points. The SIFT bag-of-words image representation information of the environment image is compared with the SIFT bag-of-words image representation information of keyframe images in the keyframe database. Based on the comparison result, it is determined whether the environment image is a keyframe image.

[0039] Specifically, SIFT feature points have high accuracy; therefore, image construction is based on SIFT feature points. When extracting SIFT feature points, all SIFT feature points of a frame of environmental image can be obtained. Since each SIFT feature point represents a specific feature, comparing the SIFT feature points of two frames one by one is inefficient. Therefore, this embodiment uses generated SIFT bag-of-words image representations to represent the features of a frame of environmental image. By comparing the SIFT bag-of-words image representations of two frames of environmental image, the differences between the two frames can be quickly determined, further improving efficiency.

[0040] In addition to comparing the SIFT bag-of-words image representations of two environmental images to determine the differences between them, another example is to compare the timestamps (sampling times) of the two environmental images. If the difference between the timestamps is greater than a preset time difference value, then the difference between the two environmental images can be considered to have reached a certain level, and it can be determined as a keyframe image.

[0041] In this embodiment, optionally, if the image is a keyframe image, then the first pose information corresponding to each sampling time is obtained based on its feature points, including: If it is a keyframe image, then based on its SIFT bag-of-words image representation information, it is matched with the SIFT bag-of-words image representation information of other keyframe images to determine all keyframe images that have a co-view relationship with it. Based on its SIFT feature points, it is matched with the SIFT feature points of all keyframe images that have a co-view relationship with it to determine SIFT feature point pairs. Based on the SIFT feature point pairs, the vehicle pose is estimated to obtain the first pose information. And, it is added to the keyframe database.

[0042] Specifically, the co-viewing relationship refers to the fact that the common parts between two frames of images meet certain conditions. For example, if the similarity of the ORB bag-of-words image representation information of the two frames of images is greater than a certain preset value, then the two frames of images have a co-viewing relationship.

[0043] In this embodiment, optionally, the step of sequentially extracting feature points from the environmental images at the plurality of sampling times further includes: Extract ORB feature points from the environmental image, and generate ORB bag-of-words image representation information based on the ORB feature points; The step of obtaining the map data of the data packet based on the 3D spatial location information corresponding to all keyframe images and their feature points includes: All keyframe images and their corresponding 3D spatial location information, ORB feature points, and ORB bag-of-words image representation information are output as map data for the data package.

[0044] Specifically, in this embodiment, SIFT feature points are used for mapping, while ORB feature points are used during the vehicle-side localization stage. That is, the vehicle uses ORB feature points for matching and localization based on map file data. Therefore, the complete map data in the data packet includes both SIFT and ORB feature points from the keyframes. This embodiment uses both types of feature points simultaneously to leverage the speed of ORB feature point extraction, which facilitates rapid localization, and the high accuracy of SIFT feature points, which improves mapping accuracy. This is a key innovation of this embodiment.

[0045] In the method of this embodiment, optionally, as follows: Figure 3 As shown, the map file is used for vehicle positioning on the vehicle side, including: Step S500: After receiving the map file, the vehicle parses the map file to obtain multiple keyframe images and their corresponding 3D spatial location information, ORB feature points, and ORB bag-of-words image representation information.

[0046] Step S600: The vehicle acquires the current environmental image, inertial navigation data and wheel speed data of the vehicle according to the preset sampling period, extracts the ORB feature points of the current environmental image, and generates the corresponding ORB bag-of-words image representation information based on the ORB feature points. Specifically, ORB feature points can be extracted quickly, thus enabling rapid localization based on image ORB feature points. When extracting ORB feature points, all ORB feature points of a single frame of an environmental image can be obtained. Since each ORB feature point represents a specific feature, comparing the ORB feature points of two environmental images one by one is inefficient. Therefore, this embodiment uses generated ORB bag-of-words image representation information to represent the features of a single frame of an environmental image.

[0047] Step S700: The vehicle matches the ORB bag-of-words image representation information corresponding to the current environment image with the ORB bag-of-words image representation information of the multi-frame keyframe images to determine all keyframe images that have a co-view relationship with the current environment image. Based on the ORB feature points of the current environment image and the ORB feature points of all keyframe images that have a co-view relationship with it, ORB feature point pairs are determined. Based on the ORB feature point pairs and the 3D spatial position information, the vehicle pose is estimated to obtain the third pose information. Specifically, by comparing the ORB bag-of-words image representation information of two environmental images, it is possible to quickly determine whether there is a co-view relationship between the two environmental images, thereby improving the operating efficiency. The co-view relationship refers to the fact that the common parts between the two images meet certain conditions. During the vehicle-side positioning stage, for example, if the similarity of the ORB bag-of-words image representation information of the two images is greater than a certain preset value, then the two images have a co-view relationship.

[0048] Step S800: Perform Kalman filtering fusion on the third pose information, inertial navigation data and wheel speed data corresponding to each keyframe image to obtain the fourth pose information, and output the fourth pose information as the current pose of the vehicle. Specifically, the fourth pose information includes the vehicle's current position and current heading angle.

[0049] Optionally, in this embodiment of the method, before performing the step of sequentially extracting feature points from the environmental images at the plurality of sampling times, the method further includes: The unique identifier of each data packet is obtained by parsing the data packet. Based on the unique identifier, it is determined whether a data packet has been lost. If so, a data retransmission request is sent to the vehicle to notify the vehicle to retransmit the lost data packet. The unique identifier of each data packet corresponds one-to-one with the order in which the data packets are sent by the vehicle. Specifically, assuming the vehicle sends 5 data packets to the cloud platform, with unique identifiers b1, b2, b3, b4, and b5 respectively; if the cloud platform receives data packet b1 followed by data packet b3, it indicates that data packet b2 has been lost. To achieve this, the vehicle needs to assign a unique identifier to each data packet.

[0050] The environmental images, inertial navigation data, and wheel speed data at multiple sampling times are verified, including verifying the integrity of the environmental images and the continuity of the inertial navigation data and wheel speed data at multiple sampling times. Specifically, for inertial navigation data and wheel speed data, the continuity of the data packets can be verified by checking whether the timestamps of the first and last frames of two consecutive received data packets correspond. For environmental images (video stream data) acquired by the image sensor, since the timestamps of the video stream data in the two consecutive received data packets are only defined relative to their respective video stream data files, it is not possible to align the video stream data in the two consecutive data packets by checking the timestamps. Therefore, the integrity of the video stream data in the data packets is verified by comparing the signal-to-noise ratio (PSNR) of the first and last image frames of the two consecutive received video stream data packets. For example, if the first and last image frames are exactly the same, then PSNR = 0 dB. A signal-to-noise ratio threshold, such as 30 dB, can be set as a criterion condition. If it is less than 30 dB, it is determined to be complete; if it is greater than or equal to 30 dB, it is determined to be incomplete.

[0051] In this embodiment, optionally, to enhance the applicability of the mapping function to various complex environments (such as lighting and dynamic objects) within an indoor parking lot, the image sensor's configuration parameters (dynamic white balance, dynamic exposure compensation EV, ISO sensitivity) can be used to perform necessary smoothing filtering, color correction, illumination normalization, and dynamic object recognition and segmentation on the environmental image. To achieve this, the data packet uploaded from the vehicle to the cloud platform also contains the image sensor's configuration parameters, which are then parsed by the cloud platform to obtain.

[0052] Corresponding to the mapping process described in the above embodiments, another embodiment of this application proposes a cloud platform that can be used to implement the mapping process described in the above embodiments. (See attached document for details.) Figure 4 The cloud platform in this embodiment includes: Data receiving unit 1 is used to receive and parse data packets uploaded by the vehicle to obtain environmental images, inertial navigation data and wheel speed data at multiple sampling times; Image processing unit 2 is used to extract feature points from the environmental images at the multiple sampling times in sequence, and determine whether the environmental image is a keyframe image based on the extracted feature points; if it is a keyframe image, the first pose information corresponding to each sampling time is obtained based on its feature points; if it is not a keyframe image, it is discarded. The fusion computing unit 3 is used to perform Kalman filtering fusion on the first pose information, inertial navigation data, and wheel speed data corresponding to each keyframe image to obtain the second pose information, and to obtain the 3D spatial position information corresponding to the feature points of the environmental image at each sampling time based on the second pose information, and to obtain the map data of the data packet based on all keyframe images and the 3D spatial position information corresponding to their feature points; and Map generation unit 4 is used to wait for a preset time. If a new data packet uploaded by the vehicle is received within the preset time, the new data packet is processed to obtain the map data of the new data packet. If no new data packet uploaded by the vehicle is received within the preset time, the mapping ends, the map data of all data packets is used to generate a map file, and the map file is sent to the vehicle for vehicle positioning.

[0053] In this embodiment, optionally, the image processing unit 2 is used for: SIFT feature points are extracted from the environment image, SIFT bag-of-words image representation information is generated based on the SIFT feature points, and the SIFT bag-of-words image representation information of the environment image is compared with the SIFT bag-of-words image representation information of keyframe images in the keyframe database. Based on the comparison result, it is determined whether the environment image is a keyframe image. If it is a keyframe image, then based on its SIFT bag-of-words image representation information, it is matched with the SIFT bag-of-words image representation information of other keyframe images to determine all keyframe images that have a co-view relationship with it. Based on its SIFT feature points, it is matched with the SIFT feature points of all keyframe images that have a co-view relationship with it to determine SIFT feature point pairs. Based on the SIFT feature point pairs, the vehicle pose is estimated to obtain the first pose information. And, it is added to the keyframe database.

[0054] In this embodiment, optionally, the image processing unit 3 is further configured to extract ORB feature points from the environmental image and generate ORB bag-of-words image representation information based on the ORB feature points; The fusion computing unit is used to output the 3D spatial location information, ORB feature points, and ORB bag-of-words image representation information corresponding to all keyframe images and their SIFT feature points as map data of the data packet.

[0055] The cloud platform of the embodiments described above is merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the cloud platform solution of the embodiments, depending on actual needs.

[0056] It should be noted that the cloud platform in the above embodiments corresponds to the cloud platform mapping process in the above embodiments. Therefore, the parts of the cloud platform in the above embodiments that are not described in detail can be obtained by referring to the content of the method in the above embodiments. That is, the specific steps recorded in the method in the above embodiments can be understood as the functions that the cloud platform in the above embodiments can achieve, and will not be described again here.

[0057] Furthermore, if the cloud platform of the above embodiments is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Therefore, as another embodiment, this application also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the visual mapping and localization method described in the above embodiments.

[0058] Another embodiment of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the visual mapping and localization method as described in the above embodiments.

[0059] Specifically, the computer-readable storage medium may include any entity or recording medium capable of carrying the computer program instructions, such as a USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media.

[0060] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technological improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A visual mapping and localization method, characterized in that, The method includes: The system receives and parses data packets uploaded from the vehicle to obtain environmental images, inertial navigation data, and wheel speed data at multiple sampling times. Feature points are extracted from the environmental images at the multiple sampling times sequentially, and it is determined whether the environmental image is a keyframe image based on the extracted feature points; if it is a keyframe image, the first pose information corresponding to each sampling time is obtained based on its feature points; if it is not a keyframe image, it is discarded; the first pose information is the camera pose. The second pose information is obtained by Kalman filtering and fusing the first pose information, inertial navigation data and wheel speed data corresponding to each keyframe image. The 3D spatial position information corresponding to the feature points of the environmental image at each sampling time is obtained based on the second pose information. The map data of the data packet is obtained based on all keyframe images and the 3D spatial position information corresponding to their feature points. The second pose information is the vehicle pose. Wait for a preset time. If a new data packet is received from the vehicle within the preset time, continue to process the new data packet to obtain the map data of the new data packet. If no new data packet is received from the vehicle within the preset time, end the mapping, generate a map file from the map data of all data packets, and send the map file to the vehicle for vehicle positioning. The step of sequentially extracting feature points from the environmental images at the multiple sampling times and determining whether the environmental image is a keyframe image based on the extracted feature points includes: extracting SIFT feature points and ORB feature points from the environmental images; generating SIFT bag-of-words image representation information based on the SIFT feature points; comparing the SIFT bag-of-words image representation information of the environmental images with the SIFT bag-of-words image representation information of keyframe images in the keyframe database; and determining whether the environmental image is a keyframe image based on the comparison result; and generating ORB bag-of-words image representation information based on the ORB feature points. The step of obtaining the map data of the data packet based on the 3D spatial location information corresponding to all keyframe images and their feature points includes: outputting the 3D spatial location information, ORB feature points, and ORB bag-of-words image representation information corresponding to all keyframe images and their SIFT feature points as the map data of the data packet. The map file is used for vehicle positioning on the vehicle side, including: The vehicle acquires the current environmental image, inertial navigation data, and wheel speed data according to a preset sampling period, extracts ORB feature points from the current environmental image, and generates corresponding ORB bag-of-words image representation information based on these ORB feature points. The vehicle matches the ORB bag-of-words image representation information corresponding to the current environmental image with the ORB bag-of-words image representation information of multiple keyframe images in the map file to determine all keyframe images that have a co-view relationship with the current environmental image. It then matches the ORB feature points of the current environmental image with the ORB feature points of all keyframe images that have a co-view relationship with it to determine ORB feature point pairs, and estimates the vehicle pose based on the ORB feature point pairs and the 3D spatial position information to obtain third pose information. The vehicle performs Kalman filtering fusion based on the third pose information, inertial navigation data, and wheel speed data corresponding to each keyframe image to obtain fourth pose information, and outputs the fourth pose information as the current pose of the vehicle.

2. The visual mapping and localization method according to claim 1, characterized in that, If the image is a keyframe, then the first pose information corresponding to each sampling time is obtained based on its feature points, including: If it is a keyframe image, then based on its SIFT bag-of-words image representation information, it is matched with the SIFT bag-of-words image representation information of other keyframe images to determine all keyframe images that have a co-view relationship with it. Based on its SIFT feature points, it is matched with the SIFT feature points of all keyframe images that have a co-view relationship with it to determine SIFT feature point pairs. Based on the SIFT feature point pairs, the vehicle pose is estimated to obtain the first pose information. And, it is added to the keyframe database.

3. The visual mapping and localization method according to any one of claims 1 to 2, characterized in that, Before performing the step of sequentially extracting feature points from the environmental images at the multiple sampling times, the method further includes: The unique identifier of the data packet is obtained by parsing the data packet. Based on the unique identifier, it is determined whether a data packet is lost. If so, a data retransmission request is sent to the vehicle to notify the vehicle to retransmit the lost data packet. The unique identifier of the data packet corresponds one-to-one with the order in which the data packets are sent by the vehicle. The environmental images, inertial navigation data, and wheel speed data at the multiple sampling times are verified, including verifying the integrity of the environmental images and the continuity of the inertial navigation data and wheel speed data at the multiple sampling times.

4. A cloud platform, characterized in that, The cloud platform includes: The data receiving unit is used to receive and parse the data packets uploaded by the vehicle to obtain environmental images, inertial navigation data and wheel speed data at multiple sampling times; The image processing unit is used to extract feature points from the environmental images at the multiple sampling times in sequence, and determine whether the environmental image is a keyframe image based on the extracted feature points; if it is a keyframe image, the first pose information corresponding to each sampling time is obtained based on its feature points; if it is not a keyframe image, it is discarded; the first pose information is the camera pose. The fusion computing unit is used to perform Kalman filtering fusion on the first pose information, inertial navigation data and wheel speed data corresponding to each keyframe image to obtain the second pose information, and to obtain the 3D spatial position information corresponding to the feature points of the environmental image at each sampling time based on the second pose information, and to obtain the map data of the data packet based on all keyframe images and the 3D spatial position information corresponding to their feature points; the second pose information is the vehicle pose. The map generation unit waits for a preset time. If a new data packet is received from the vehicle within the preset time, the new data packet is processed to obtain the map data of the new data packet. If no new data packet is received from the vehicle within the preset time, the mapping is terminated, the map data of all data packets are used to generate a map file, and the map file is sent to the vehicle for vehicle positioning. in: The image processing unit is specifically used to extract SIFT feature points and ORB feature points from the environment image, generate SIFT bag-of-words image representation information based on the SIFT feature points, compare the SIFT bag-of-words image representation information of the environment image with the SIFT bag-of-words image representation information of keyframe images in the keyframe database, and determine whether the environment image is a keyframe image based on the comparison result; and generate ORB bag-of-words image representation information based on the ORB feature points. The fusion computing unit is specifically used to output the 3D spatial location information, ORB feature points, and ORB bag-of-words image representation information corresponding to all keyframe images and their SIFT feature points as map data of the data packet; The map file is used for vehicle positioning on the vehicle side, including: The vehicle acquires the current environmental image, inertial navigation data, and wheel speed data according to a preset sampling period, extracts ORB feature points from the current environmental image, and generates corresponding ORB bag-of-words image representation information based on these ORB feature points. The vehicle matches the ORB bag-of-words image representation information corresponding to the current environmental image with the ORB bag-of-words image representation information of multiple keyframe images in the map file to determine all keyframe images that have a co-view relationship with the current environmental image. It then matches the ORB feature points of the current environmental image with the ORB feature points of all keyframe images that have a co-view relationship with it to determine ORB feature point pairs, and estimates the vehicle pose based on the ORB feature point pairs and the 3D spatial position information to obtain third pose information. The vehicle performs Kalman filtering fusion based on the third pose information, inertial navigation data, and wheel speed data corresponding to each keyframe image to obtain fourth pose information, and outputs the fourth pose information as the current pose of the vehicle.

5. The cloud platform according to claim 4, characterized in that, The image processing unit is used for: If it is a keyframe image, then based on its SIFT bag-of-words image representation information, it is matched with the SIFT bag-of-words image representation information of other keyframe images to determine all keyframe images that have a co-view relationship with it. Based on its SIFT feature points, it is matched with the SIFT feature points of all keyframe images that have a co-view relationship with it to determine SIFT feature point pairs. Based on the SIFT feature point pairs, the vehicle pose is estimated to obtain the first pose information. And, it is added to the keyframe database.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the visual mapping and localization method as described in any one of claims 1 to 3.

Images