Image processing method, device, equipment and computer readable storage medium

Abstract

The application discloses an image processing method, device and equipment and a computer readable storage medium. The method comprises: acquiring a 3D point cloud corresponding to crowd-sourced trajectory information of a target scene; generating a base map corresponding to the target scene based on the first 3D point cloud and the first driving trajectory; determining a target 3D point cloud in the second 3D point cloud based on the base map and the second driving trajectory; and generating a target map corresponding to the target scene based on the target 3D point cloud and the base map. The application performs crowd-sourced fusion mapping of the target scene by using vehicle posture information and a BEV map, and can perform crowd-sourced fusion mapping by using different vehicles without loading a laser radar on the vehicle, thereby reducing the long data collection period during fusion mapping and the cost of the vehicle, improving the timeliness of fusion mapping and the timeliness of updating. Meanwhile, the base map idea is used to improve the alignment and fusion success rate of crowd-sourced data during crowd-sourced fusion mapping.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Currently, parking lot crowdsourcing mapping utilizes data on different parking spaces corresponding to multiple vehicles traveling on different routes, and uses data fragments in the cloud to create, update, or replace maps.

[0003] Existing crowdsourced fusion mapping solutions require vehicles to have as many identical sensors as possible, and the collected data to contain the same elements for fusion mapping. Due to vehicle models or user selection, this approach results in limited data available for crowdsourced mapping, and each vehicle model needs to maintain its own map. While using different vehicle models equipped with LiDAR allows for fusion mapping, LiDAR is expensive, and it's difficult to equip all vehicles with it. This leads to longer data collection cycles when using LiDAR for fusion mapping, resulting in poor map update capabilities.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide an image processing method, apparatus, device, and computer-readable storage medium, which aims to solve the technical problem that existing crowdsourced fusion mapping of different vehicle models equipped with LiDAR is difficult to update in a timely manner.

[0006] To achieve the above objectives, this application provides an image processing method, comprising:

[0007] Obtain 3D point cloud corresponding to crowdsourced trajectory information of target scene, wherein the crowdsourced trajectory information includes multiple point cloud bird's-eye view BEV images corresponding to each driving trajectory of the vehicle in the target scene and vehicle posture information corresponding to each BEV image, and the 3D point cloud includes a first 3D point cloud corresponding to a first driving trajectory and a second 3D point cloud corresponding to a second driving trajectory.

[0008] Based on the first 3D point cloud and the first driving trajectory, a base map corresponding to the target scene is generated;

[0009] Based on the base map and the second driving trajectory, the target 3D point cloud is determined in the second 3D point cloud;

[0010] Based on the target 3D point cloud and the base map, a target map corresponding to the target scene is generated.

[0011] Further, the first 3D point cloud includes a first semantic point corresponding to the first driving trajectory; the step of generating a base map corresponding to the target scene based on the first 3D point cloud includes:

[0012] Preprocess the first 3D point cloud based on the first semantic point to obtain the first target 3D point cloud;

[0013] Based on the first semantic points of the first target 3D point cloud, the first driving trajectory of each of the first target 3D point clouds is aligned to obtain the base map.

[0014] Further, the step of preprocessing the first 3D point cloud based on the first semantic point to obtain the first target 3D point cloud includes:

[0015] Based on the preset size information corresponding to the preset semantic point, the first semantic point and the height difference between adjacent 3D point clouds in the first 3D point cloud, the first 3D point cloud is filtered to obtain the filtered first 3D point cloud.

[0016] Based on the shooting time of the BEV image corresponding to the first filtered 3D point cloud, the first filtered 3D point cloud is further filtered to obtain the first target 3D point cloud.

[0017] Further, the step of determining the target 3D point cloud in the second 3D point cloud based on the base map and the second driving trajectory includes:

[0018] The second driving trajectory is segmented to obtain sub-driving trajectories, and the sub-3D point cloud corresponding to each sub-driving trajectory is obtained from the second 3D point cloud;

[0019] Based on the sub-trajectory, each sub-3D point cloud is aligned with the base map to obtain the confidence level corresponding to each sub-3D point cloud;

[0020] Based on the confidence level, the target 3D point cloud is determined in the sub-3D point cloud.

[0021] Further, the step of aligning each of the sub-3D point clouds with the base map based on the sub-driving trajectory to obtain the confidence level corresponding to each of the sub-3D point clouds includes:

[0022] Based on the sub-driving trajectory, each sub-3D point cloud is aligned with the base map to determine the distance between each sub-driving trajectory and the corresponding driving trajectory in the base map;

[0023] The confidence level is determined based on the distance.

[0024] Further, the step of determining the target 3D point cloud in the sub-3D point cloud based on the confidence level includes:

[0025] In the sub-3D point cloud, a target sub-3D point cloud with a confidence level greater than a preset confidence level is obtained, and the target sub-3D point cloud is used as the target 3D point cloud.

[0026] Furthermore, the step of generating a target map corresponding to the target scene based on the target 3D point cloud and the base map includes:

[0027] Based on the second semantic points of the target 3D point cloud, a clustering operation is performed on the target 3D point cloud to obtain the target cluster center;

[0028] Based on the size information corresponding to the base image, the second semantic points corresponding to the target cluster center are fused to obtain a fused image;

[0029] Based on the fused image and the base map, a target map corresponding to the target scene is generated.

[0030] Further, the step of performing clustering operations on the target 3D point cloud based on the second semantic points of the target 3D point cloud to obtain target cluster centers includes:

[0031] Based on the second semantic point of the target 3D point cloud, a clustering operation is performed on the target 3D point cloud to obtain the cluster center corresponding to the target 3D point cloud;

[0032] Based on the number of trajectories of the second driving trajectory corresponding to the cluster center in the target 3D point cloud, and the number of times the cluster center appears in the target 3D point cloud, the maturity information corresponding to each cluster center is determined.

[0033] Cluster centers whose maturity information is greater than a preset maturity level are selected as the target cluster centers.

[0034] Further, the step of fusing the second semantic point corresponding to the target cluster center based on the size information corresponding to the base image to obtain the fused image includes:

[0035] Based on the size information corresponding to the base image, the second semantic point corresponding to the target cluster center is fused to obtain the first image;

[0036] Determine whether there are overlapping semantic points in the first image;

[0037] If there are overlapping semantic points with overlapping spatial locations, then the target semantic point corresponding to the spatial location of the overlapping semantic point is determined based on the maturity information corresponding to the overlapping semantic point.

[0038] The fused image is determined based on the first image and the target semantic points.

[0039] Furthermore, the step of obtaining the 3D point cloud corresponding to the crowdsourced trajectory information of the target scene includes:

[0040] Obtain the crowdsourcing trajectory information;

[0041] Deep learning is used to identify the BEV map to obtain semantic points corresponding to each driving trajectory;

[0042] The 3D point cloud is determined based on the semantic points corresponding to each driving trajectory and the vehicle posture information.

[0043] Furthermore, the step of obtaining the crowdsourcing trajectory information includes:

[0044] The vehicle attitude information uploaded by the vehicle is obtained, wherein the vehicle is equipped with a GNSS sensor, an IMU sensor and a wheel speed meter, and the vehicle determines the vehicle attitude information based on the monitoring results of the GNSS sensor, the monitoring results of the IMU sensor and the monitoring results of the wheel speed meter;

[0045] The uploaded BEV image is obtained, wherein the vehicle is equipped with a front camera, a left camera, and a right camera, and the vehicle determines the BEV image based on the images captured by the front camera, the left camera, and the right camera.

[0046] Furthermore, to achieve the above objectives, this application also provides an image processing apparatus, the image processing apparatus comprising:

[0047] The acquisition module is used to acquire 3D point clouds corresponding to crowdsourced trajectory information of the target scene. The crowdsourced trajectory information includes multiple point cloud bird's-eye view BEV images corresponding to each driving trajectory of the vehicle in the target scene and vehicle posture information corresponding to each BEV image. The 3D point cloud includes a first 3D point cloud corresponding to a first driving trajectory and a second 3D point cloud corresponding to a second driving trajectory.

[0048] The base map generation module is used to generate a base map corresponding to the target scene based on the first 3D point cloud and the first driving trajectory.

[0049] The determination module is used to determine the target 3D point cloud in the second 3D point cloud based on the base map and the second driving trajectory;

[0050] The base map generation module is used to generate a target map corresponding to the target scene based on the target 3D point cloud and the base map.

[0051] In addition, to achieve the above objectives, this application also provides an image processing apparatus, which includes: a memory, a processor, and an image processing program stored in the memory and executable on the processor. When the image processing program is executed by the processor, it implements the steps of the image processing method described above.

[0052] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing an image processing program, which, when executed by a processor, implements the steps of the image processing method described above.

[0053] This application acquires 3D point clouds corresponding to crowdsourced trajectory information of a target scene. The crowdsourced trajectory information includes multiple point cloud bird's-eye view BEV maps corresponding to various driving trajectories of vehicles in the target scene, and vehicle attitude information corresponding to each BEV map. The 3D point clouds include a first 3D point cloud corresponding to a first driving trajectory and a second 3D point cloud corresponding to a second driving trajectory. Then, based on the first 3D point cloud and the first driving trajectory, a base map corresponding to the target scene is generated. Next, based on the base map and the second driving trajectory, a target 3D point cloud is determined in the second 3D point cloud. Finally, based on the target 3D point cloud and the base map, a target map corresponding to the target scene is generated. This allows for crowdsourced fusion mapping of the target scene using vehicle attitude information and BEV maps. It eliminates the need to equip vehicles with LiDAR, enabling crowdsourced fusion mapping from different vehicles. This reduces the long data collection cycle and vehicle costs during fusion mapping, and improves the timeliness and update speed of fusion mapping. Simultaneously, the use of the base map concept improves the alignment and fusion success rate of crowdsourced data during crowdsourced fusion mapping. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the image processing device structure in the hardware operating environment involved in the embodiments of this application;

[0055] Figure 2 This is a flowchart illustrating the first embodiment of the image processing method of this application;

[0056] Figure 3 This is a schematic diagram of a possible scenario of a map in one embodiment of the image processing method of this application;

[0057] Figure 4 This is a schematic diagram of the functional modules of an embodiment of the image processing apparatus of this application.

[0058] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0059] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0060] like Figure 1 As shown, Figure 1 This is a schematic diagram of the image processing device structure in the hardware operating environment involved in the embodiments of this application.

[0061] The image processing device in this application embodiment can be a server. For example... Figure 1 As shown, the image processing device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0062] Optionally, the image processing device may also include a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, a WiFi module, and so on. These sensors may include, for example, light sensors, motion sensors, and other sensors. Of course, the image processing device may also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, which will not be elaborated upon here.

[0063] Those skilled in the art will understand that Figure 1 The terminal structure shown does not constitute a limitation on the image processing device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0064] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an image processing program.

[0065] exist Figure 1 In the image processing device shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client (user terminal) and communicate data with the client; and the processor 1001 can be used to call the image processing program stored in the memory 1005.

[0066] In this embodiment, the image processing device includes: a memory 1005, a processor 1001, and an image processing program stored in the memory 1005 and executable on the processor 1001. When the processor 1001 calls the image processing program stored in the memory 1005, it executes the steps of the image processing methods in the following embodiments.

[0067] This application also provides an image processing method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the image processing method of this application.

[0068] The image processing method includes:

[0069] Step S101: Obtain the 3D point cloud corresponding to the crowdsourced trajectory information of the target scene. The crowdsourced trajectory information includes multiple point cloud bird's-eye view BEV images corresponding to each driving trajectory of the vehicle in the target scene and vehicle posture information corresponding to each BEV image. The 3D point cloud includes a first 3D point cloud corresponding to the first driving trajectory and a second 3D point cloud corresponding to the second driving trajectory.

[0070] The vehicle can be multiple vehicles of different models, and the target scene can be any parking lot. When the vehicle enters the target scene, the vehicle's position and pose information is acquired in real time. At the same time, the vehicle's surround view camera collects environmental images of the surrounding environment, and determines the BEV (Bird's EyeView) map based on the environmental images. The BEV map, along with the corresponding vehicle position and pose information and the shooting time, is then uploaded to the cloud server.

[0071] When the BEV map in the crowdsourced trajectory information corresponding to the target scene meets the preset conditions, the server obtains the 3D point cloud corresponding to the crowdsourced trajectory information of the target scene. Specifically, when the shooting time interval between each shooting moment of the BEV map in the crowdsourced trajectory information reaches the preset time interval, and the driving trajectory corresponding to the BEV map of each vehicle can cover the target scene, the driving trajectories can be aligned according to the overlapping area of ​​each driving trajectory. The aligned driving trajectory is compared with the target scene. When the distance between the edge of the aligned driving trajectory and the edge of the target scene (in the same proportion) is less than the first preset distance, and the distance between two adjacent driving trajectories is less than the second preset distance, it is determined that the BEV map corresponding to the target scene meets the preset conditions. The first preset distance can be set according to the length of one parking space plus the width of the road, and the second preset distance can be set according to the length of two parking spaces plus the width of two roads.

[0072] Specifically, in one possible implementation, step S101 includes:

[0073] Step S1011: Obtain the crowdsourcing trajectory information;

[0074] Step S1012: Perform deep learning recognition on the BEV map to obtain semantic points corresponding to each driving trajectory;

[0075] Step S1013: Determine the 3D point cloud based on the semantic points corresponding to each driving trajectory and the vehicle posture information.

[0076] When the BEV map in the crowdsourced trajectory information corresponding to the target scenario meets the preset conditions, the crowdsourced trajectory information is obtained, namely the BEV map uploaded by the vehicle and the vehicle posture information corresponding to the BEV map. Then, deep learning recognition is performed on each BEV map to obtain the semantic points corresponding to each driving trajectory. Specifically, existing deep learning algorithms are used to perform deep learning recognition on each BEV map to obtain the semantic points corresponding to each driving trajectory. The semantic points include lane lines, arrows, stop lines, speed bumps, parking space lines, parking lot gates, building walls, cones, parking space codes, and traffic sign types, etc.

[0077] Next, based on the semantic points corresponding to each driving trajectory and the vehicle posture information, the 3D point cloud is determined. Specifically, the semantic points corresponding to each driving trajectory are associated with the corresponding vehicle pose information to reconstruct the 2D points detected by the BEV map into a 3D point cloud of the road plane, thereby accurately obtaining the 3D point cloud and ensuring that the 3D point cloud can accurately obtain the target map.

[0078] The first semantic point corresponding to the first driving trajectory may include lane lines, arrows, stop lines, speed bumps, and parking space lines, while the second semantic point corresponding to the second driving trajectory may include lane lines, arrows, stop lines, speed bumps, parking space lines, parking lot gates, building walls, traffic cones, parking space codes, and traffic sign types, etc.

[0079] Specifically, in one possible implementation, step S1011 includes:

[0080] Step a, obtain the vehicle attitude information uploaded by the vehicle, wherein the vehicle is equipped with a GNSS sensor, an IMU sensor and a wheel speed meter, and the vehicle determines the vehicle attitude information based on the monitoring results of the GNSS sensor, the monitoring results of the IMU sensor and the monitoring results of the wheel speed meter;

[0081] Step b: Obtain the uploaded BEV image, wherein the vehicle is equipped with a front camera, a left camera, and a right camera, and the vehicle determines the BEV image based on the images captured by the front camera, the left camera, and the right camera.

[0082] The vehicle uses essential GNSS sensors, IMU sensors, and wheel speedometers to measure and monitor data. Based on the monitoring results from the GNSS sensors, IMU sensors, and wheel speedometers, the vehicle determines its position and orientation information using existing pose algorithms.

[0083] Simultaneously, the vehicle is equipped with surround-view cameras: a front camera, a left camera, and a right camera. By acquiring images from these three cameras, the vehicle processes them using existing image algorithms such as cropping, stitching, and affine transformation to obtain the current BEV (Balance of Vehicle) image. Through the vehicle's cameras and essential sensors, vehicle pose information and the BEV image can be obtained. Fusion mapping using these two images allows any vehicle in the target scene to participate in the mapping process, eliminating the need for LiDAR on the vehicle, thus reducing costs while improving the timeliness and update speed of the fusion mapping.

[0084] Step S102: Based on the first 3D point cloud and the first driving trajectory, generate a base map corresponding to the target scene;

[0085] Upon obtaining the first 3D point cloud, a base map corresponding to the target scene is generated based on the first 3D point cloud and the first driving trajectory. Specifically, according to the first semantic points of the first 3D point cloud, an alignment and stitching operation is performed on the first driving trajectory to stitch together all the first driving trajectories, thereby obtaining the base map. Figure 3 The base map includes the road network structure of the target scene.

[0086] Step S103: Based on the base map and the second driving trajectory, determine the target 3D point cloud in the second 3D point cloud;

[0087] After obtaining the base map, the target 3D point cloud is determined in the second 3D point cloud based on the base map and the second driving trajectory. Specifically, the second driving trajectory is first aligned and stitched according to the second semantic points of the second 3D point cloud. The second 3D point cloud is then filtered according to the distance between the second driving trajectory and the driving trajectory in the base map to obtain the target 3D point cloud.

[0088] Step S104: Based on the target 3D point cloud and the base map, generate a target map corresponding to the target scene.

[0089] When the target 3D point cloud is obtained, a target map corresponding to the target scene is generated based on the target 3D point cloud and the base map. Specifically, the target 3D point clouds are fused to obtain a fused image, and the fused image is bound to the base map. That is, according to the nearest association principle, the elements of the fused image are bound to the road network structure of the base map to obtain the target map.

[0090] By acquiring 3D point clouds corresponding to crowdsourced trajectory information of the target scene, wherein the crowdsourced trajectory information includes multiple point cloud bird's-eye view BEV maps corresponding to each driving trajectory of the vehicle in the target scene and vehicle attitude information corresponding to each BEV map, the 3D point clouds include a first 3D point cloud corresponding to a first driving trajectory and a second 3D point cloud corresponding to a second driving trajectory; then, based on the first 3D point cloud and the first driving trajectory, a base map corresponding to the target scene is generated; then, based on the base map and the second driving trajectory, a target 3D point cloud is determined in the second 3D point cloud; finally, based on the target 3D point cloud and the base map, a target map corresponding to the target scene is generated. Crowdsourced fusion mapping of the target scene can be performed using vehicle attitude information and BEV maps, eliminating the need to equip vehicles with LiDAR. This reduces the long data collection cycle and vehicle costs during fusion mapping, and improves the timeliness and update timeliness of fusion mapping. Simultaneously, utilizing the base map concept improves the alignment and fusion success rate of crowdsourced data during crowdsourced fusion mapping.

[0091] Based on the first embodiment, a second embodiment of the image processing method of this application is proposed, wherein the first 3D point cloud includes the first semantic point corresponding to the first driving trajectory; step S102 includes:

[0092] Step S201: Preprocess the first 3D point cloud based on the first semantic point to obtain the first target 3D point cloud;

[0093] Step S202: Based on the first semantic points of the first target 3D point cloud, perform an alignment operation on the first driving trajectory of each of the first target 3D point clouds to obtain the base map.

[0094] When the first 3D point cloud is obtained, the first semantic point corresponding to the first driving trajectory is obtained. The first semantic point may include lane lines, arrows, stop lines, speed bumps, parking lines, etc. Then, the first 3D point cloud is preprocessed based on the first semantic point to obtain the first target 3D point cloud. Specifically, the first target 3D point cloud is obtained by filtering the first 3D point cloud through the first semantic point.

[0095] Next, based on the first semantic points of the first target 3D point cloud, the first driving trajectories of each of the first target 3D point clouds are aligned to obtain a base map. Specifically, the overlapping portions of the first driving trajectories of the first target 3D point clouds are determined based on the first semantic points, and the first driving trajectories are aligned and stitched together based on these overlapping portions to obtain the base map. Figure 3 The base map includes the road network structure of the target scene, where existing algorithms can be used to align and stitch together the various first driving trajectories.

[0096] Specifically, in one possible implementation, step S201 includes:

[0097] Step c: Based on the preset size information corresponding to the preset semantic point, the first semantic point, and the height difference between adjacent 3D point clouds in the first 3D point cloud, the first 3D point cloud is filtered to obtain the filtered first 3D point cloud.

[0098] Step d: Based on the shooting time of the BEV image corresponding to the first filtered 3D point cloud, the first filtered 3D point cloud is filtered to obtain the first target 3D point cloud.

[0099] When the first 3D point cloud is obtained, preset size information corresponding to the target scene is acquired. Based on the preset size information, the first semantic point, and the height difference between adjacent 3D point clouds in the first 3D point cloud, the first 3D point cloud is filtered to obtain a filtered first 3D point cloud. Specifically, according to the preset size information (size range of lane lines, arrows, stop lines, speed bumps, and parking spaces) corresponding to lane lines, arrows, stop lines, speed bumps, and parking spaces in the target scene, semantic points that satisfy the corresponding preset size information are determined. 3D point clouds to be filtered that satisfy the corresponding preset size information are then acquired from the first 3D point cloud. The height difference between adjacent 3D point clouds in the first 3D point cloud is used to identify 3D point clouds with height differences greater than a preset height difference. These 3D point clouds are then deleted from the 3D point cloud to be filtered, resulting in the filtered first 3D point cloud.

[0100] Next, based on the shooting time of the BEV image corresponding to the first filtered 3D point cloud, the first filtered 3D point cloud is further filtered to obtain the first target 3D point cloud. Specifically, it is determined whether there is an overlapping driving trajectory corresponding to the first filtered 3D point cloud. If so, the 3D point cloud corresponding to the overlapping driving trajectory (overlapping 3D point cloud) with the 3D point cloud whose shooting time is closest to the current time (latest 3D point cloud) is determined. Other 3D point clouds except the latest 3D point cloud are deleted from the overlapping 3D point cloud in the first filtered 3D point cloud to obtain the first target 3D point cloud.

[0101] By filtering the first 3D point cloud, problematic and redundant 3D point clouds can be removed, thereby improving the efficiency of crowdsourced fusion mapping.

[0102] The first 3D point cloud is preprocessed based on the first semantic point to obtain a first target 3D point cloud. Then, based on the first semantic point of the first target 3D point cloud, the first driving trajectory of each first target 3D point cloud is aligned to obtain the base map. By filtering the first 3D point cloud, the number of 3D point clouds participating in the base map construction is reduced, thereby improving the efficiency of crowdsourced fusion mapping. By utilizing the base map concept, the alignment and fusion success rate of crowdsourced data during crowdsourced fusion mapping is improved.

[0103] Based on the first embodiment, a third embodiment of the image processing method of this application is proposed, wherein step S103 includes:

[0104] Step S301: The second driving trajectory is segmented to obtain sub-driving trajectories, and the sub-3D point cloud corresponding to each sub-driving trajectory is obtained in the second 3D point cloud;

[0105] Step S302: Based on the sub-driving trajectory, align each of the sub-3D point clouds with the base map to obtain the confidence level corresponding to each of the sub-3D point clouds;

[0106] Step S303: Based on the confidence level, determine the target 3D point cloud in the sub-3D point cloud.

[0107] The second semantic points corresponding to the second 3D point cloud include lane lines, arrows, stop lines, speed bumps, parking space lines, parking lot gates, building walls, cones, parking space codes, and traffic sign types, etc.

[0108] After obtaining the base map, the second driving trajectory is segmented to obtain sub-driving trajectories. Sub-3D point clouds corresponding to each sub-driving trajectory are obtained from the second 3D point cloud to filter the sub-3D point clouds.

[0109] Next, based on the sub-trajectory, each sub-3D point cloud is aligned with the base map to obtain the confidence level corresponding to each sub-3D point cloud. Specifically, in one possible implementation, step S302 includes:

[0110] Step S3021: Based on the sub-driving trajectory, align each of the sub-3D point clouds with the base map to determine the distance between each of the sub-driving trajectories and the corresponding driving trajectory of the base map;

[0111] Step S3022: Determine the confidence level based on the distance.

[0112] After obtaining the sub-3D point clouds, each sub-3D point cloud is aligned with the base map based on the sub-driving trajectory. Specifically, the overlapping portion between the second driving trajectory of the sub-3D point cloud and the driving trajectory of the base map is determined based on the second semantic point. Each second driving trajectory is aligned based on the overlapping portion with the driving trajectory of the base map. Then, based on the aligned second driving trajectory, the distance between each sub-driving trajectory and the corresponding driving trajectory of the base map is determined, and the confidence level is determined based on the distance. The larger the distance, the lower the confidence level. Thus, the confidence level of the sub-3D point cloud can be accurately obtained based on the sub-driving trajectory. By improving the accuracy of filtering the sub-3D point clouds, the interference of erroneous 3D point clouds on crowdsourced fusion mapping is reduced, thereby improving the accuracy of crowdsourced fusion mapping.

[0113] Next, based on the confidence level, the target 3D point cloud is determined from the sub-3D point cloud, that is, the target 3D point cloud is obtained by filtering the sub-3D point cloud through the confidence level. Specifically, in one possible implementation, this step S303 includes:

[0114] Step e: In the sub-3D point cloud, obtain the target sub-3D point cloud with a confidence level greater than the preset confidence level, and use the target sub-3D point cloud as the target 3D point cloud.

[0115] When the confidence level is obtained, it is determined whether each confidence level is greater than the preset confidence level, and the target sub-3D point cloud with a confidence level greater than the preset confidence level is obtained. The target sub-3D point cloud is used as the target 3D point cloud. The sub-3D point cloud is filtered by confidence level to reduce the interference of erroneous 3D point clouds on crowdsourced fusion mapping, and improve the accuracy and output quality of crowdsourced fusion mapping.

[0116] It should be noted that before step S301, the following steps are also included:

[0117] Based on the second semantic points and the second driving trajectory of the second 3D point cloud, each second 3D point cloud is aligned to the base map to obtain an aligned base map. Driving trajectories that overlap with the second driving trajectory are deleted from the aligned base map to retain semantic points such as parking gates, building walls, traffic cones, parking space codes, and traffic sign types from the second semantic points. An updated base map is obtained and used as the base map. Then, step S301 is executed to enrich the map content by adding semantic points such as parking gates, building walls, traffic cones, parking space codes, and traffic sign types from the second semantic points to the map, thereby further improving the accuracy and quality of crowdsourced fusion mapping.

[0118] The second driving trajectory is segmented to obtain sub-driving trajectories, and sub-3D point clouds corresponding to each sub-driving trajectory are obtained in the second 3D point cloud. Then, based on the sub-driving trajectories, each sub-3D point cloud is aligned with the base map to obtain the confidence level corresponding to each sub-3D point cloud. Then, based on the confidence level, the target 3D point cloud is determined in the sub-3D point cloud. By improving the accuracy of sub-3D point cloud screening, the interference of erroneous 3D point clouds on crowdsourced fusion mapping is reduced, thereby improving the accuracy of crowdsourced fusion mapping.

[0119] Based on the first embodiment, a fourth embodiment of the image processing method of this application is proposed, wherein step S104 includes:

[0120] Step S401: Based on the second semantic points of the target 3D point cloud, perform a clustering operation on the target 3D point cloud to obtain the target cluster center;

[0121] Step S402: Based on the size information corresponding to the base image, fuse the second semantic points corresponding to the target cluster center to obtain a fused image;

[0122] Step S403: Based on the fused image and the base map, generate a target map corresponding to the target scene.

[0123] After obtaining the target 3D point cloud, clustering is performed on the target 3D point cloud based on the second semantic points to obtain target cluster centers. The target 3D point cloud can be clustered using existing clustering algorithms according to the type information corresponding to the second semantic points. This type information includes point clustering, line clustering, and polygon clustering. Point clustering includes clusters such as jitter signals, sign centers, and gate centers. Line clustering includes clusters such as pillars, dashed line segments, and arrow segments. Polygon clustering includes clusters such as parking spaces and no-parking zones.

[0124] Specifically, in one possible implementation, step S401 includes:

[0125] Step S4011: Based on the second semantic point of the target 3D point cloud, perform a clustering operation on the target 3D point cloud to obtain the cluster center corresponding to the target 3D point cloud;

[0126] Step S4012: Based on the number of trajectories of the second driving trajectory corresponding to the cluster center in the target 3D point cloud and the number of times the cluster center appears in the target 3D point cloud, determine the maturity information corresponding to each cluster center.

[0127] Step S4013: Select the cluster centers whose maturity information is greater than the preset maturity as the target cluster centers.

[0128] After obtaining the target 3D point cloud, based on the second semantic points of the target 3D point cloud, the existing clustering algorithm is used to perform clustering operations on the target 3D point cloud to obtain the cluster centers corresponding to the target 3D point cloud.

[0129] Next, the number of trajectories of the cluster centers in the second driving trajectory corresponding to the target 3D point cloud is obtained, along with the frequency of occurrence of the cluster centers in the target 3D point cloud. Based on this number of trajectories and frequency of occurrence, the maturity information corresponding to each cluster center is determined. The number of trajectories is the number of times the second driving trajectory corresponding to the target 3D point cloud passes through the cluster center. For example, for a cluster center of a certain arrow, if 10 driving trajectories in the second driving trajectory corresponding to the target 3D point cloud pass through the arrow, the number of trajectories is 10. If 8 out of the 10 driving trajectories detect the arrow, the frequency of occurrence is 8, and the maturity information is 0.8 (8 / 10). If 2 out of the 10 driving trajectories detect the arrow, the frequency of occurrence is 2, and the maturity information is 0.2 (2 / 10).

[0130] Then, cluster centers with maturity information greater than a preset maturity level are selected as target cluster centers. The preset maturity level can be set reasonably, for example, it can be 0.5. For example, for a cluster center of a certain arrow, if 10 driving trajectories in the second driving trajectory corresponding to the target 3D point cloud pass through the arrow, and if 2 of the 10 driving trajectories detect the arrow, then its occurrence count is 2, and the maturity information is 0.2, which is less than 0.5. This indicates that the confidence of the arrow is not high, and it needs to be discarded during mapping. By filtering cluster centers based on maturity information, the interference of erroneous 3D point clouds on crowdsourced fusion mapping is reduced, and the accuracy of crowdsourced fusion mapping is improved.

[0131] When the target cluster center is obtained, based on the size information corresponding to the base image, the second semantic point corresponding to the target cluster center is fused to obtain a fused image; specifically, in one possible implementation, step S402 includes:

[0132] Step S4021: Based on the size information corresponding to the base image, fuse the second semantic point corresponding to the target cluster center to obtain the first image;

[0133] Step S4022: Determine whether there are overlapping semantic points with overlapping spatial locations in the first image;

[0134] Step S4023: If there are overlapping semantic points with overlapping spatial locations, then based on the maturity information corresponding to the overlapping semantic points, determine the target semantic point of the spatial location corresponding to the overlapping semantic points.

[0135] Step S4024: Determine the fused image based on the first image and the target semantic points.

[0136] When the target cluster center is obtained, the second semantic point corresponding to the target cluster center is fused based on the size information corresponding to the base map to obtain the first image. A base map tray area can be set according to the size information corresponding to the base map, and the second semantic point corresponding to the target cluster center is fused in the base map tray area to obtain the first image.

[0137] Upon acquiring the first image, it is determined whether there are overlapping semantic points with overlapping spatial locations within the first image. For example, if a portion of two adjacent parking spaces is identified as another parking space, then the semantic points corresponding to the two adjacent parking spaces and the other parking space are overlapping semantic points with overlapping spatial locations. If overlapping semantic points with overlapping spatial locations exist, then based on the maturity information corresponding to the overlapping semantic points, a target semantic point for the spatial location corresponding to the overlapping semantic points is determined. Specifically, the semantic point with the highest maturity information among the overlapping semantic points is selected as the target semantic point for the spatial location corresponding to the overlapping semantic point.

[0138] Next, based on the first image and the target semantic points, a fused image is determined. Specifically, other semantic points besides the target semantic points are removed from the overlapping semantic points in the first image to obtain the fused image. This removes the semantic points that are misperceived, reduces the interference of erroneous 3D point clouds on crowdsourced fusion mapping, and improves the accuracy of crowdsourced fusion mapping.

[0139] Then, based on the fused image and the base map, a target map corresponding to the target scene is generated. Specifically, the 3D point clouds of each target are fused to obtain a fused image, and the fused image is bound to the base map. That is, according to the nearest association principle, the elements of the fused image are bound to the road network structure of the base map to obtain the target map.

[0140] By clustering the target 3D point cloud based on the second semantic points, a target cluster center is obtained. Then, based on the size information corresponding to the base map, the second semantic points corresponding to the target cluster center are fused to obtain a fused image. Subsequently, based on the fused image and the base map, a target map corresponding to the target scene is generated. The fused image can be accurately obtained through the target cluster center, and thus the target map can be accurately generated, further improving the accuracy of crowdsourced fusion mapping.

[0141] In addition, this application also proposes an image processing apparatus, referring to Figure 4 The image processing apparatus includes:

[0142] The acquisition module 10 is used to acquire 3D point clouds corresponding to the crowdsourced trajectory information of the target scene. The crowdsourced trajectory information includes multiple point cloud bird's-eye view BEV images corresponding to each driving trajectory of the vehicle in the target scene and vehicle posture information corresponding to each BEV image. The 3D point cloud includes a first 3D point cloud corresponding to the first driving trajectory and a second 3D point cloud corresponding to the second driving trajectory.

[0143] The base map generation module 20 is used to generate a base map corresponding to the target scene based on the first 3D point cloud and the first driving trajectory.

[0144] The determination module 30 is used to determine the target 3D point cloud in the second 3D point cloud based on the base map and the second driving trajectory;

[0145] The base map generation module 40 is used to generate a target map corresponding to the target scene based on the target 3D point cloud and the base map.

[0146] The methods executed by the above-mentioned program units can be referred to in the various embodiments of the image processing method of this application, and will not be repeated here.

[0147] Furthermore, this application also proposes a computer-readable storage medium storing an image processing program, which, when executed by a processor, implements the steps of the image processing method described above.

[0148] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0149] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0151] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An image processing method, characterized in that, include: Obtain 3D point cloud corresponding to crowdsourced trajectory information of target scene, wherein the crowdsourced trajectory information includes multiple point cloud bird's-eye view BEV images corresponding to each driving trajectory of the vehicle in the target scene and vehicle posture information corresponding to each BEV image, and the 3D point cloud includes a first 3D point cloud corresponding to a first driving trajectory and a second 3D point cloud corresponding to a second driving trajectory. Based on the first 3D point cloud and the first driving trajectory, a base map corresponding to the target scene is generated; Based on the base map and the second driving trajectory, the target 3D point cloud is determined in the second 3D point cloud; Based on the target 3D point cloud and the base map, a target map corresponding to the target scene is generated; The step of generating a target map corresponding to the target scene based on the target 3D point cloud and the base map includes: Based on the type information corresponding to the second semantic points of the target 3D point cloud, a clustering operation is performed on the target 3D point cloud to obtain target cluster centers. Specifically, based on the second semantic points of the target 3D point cloud, a clustering operation is performed to obtain cluster centers corresponding to the target 3D point cloud. Based on the number of trajectories of the second driving trajectory corresponding to the cluster centers in the target 3D point cloud, and the number of times the cluster centers appear in the target 3D point cloud, maturity information corresponding to each cluster center is determined. Maturity information = number of appearances / number of trajectories. Cluster centers with maturity information greater than a preset maturity value are selected as the target cluster centers. Based on the size information corresponding to the base image, the second semantic points corresponding to the target cluster centers are fused to obtain a fused image. Specifically, based on the size information corresponding to the base image, the second semantic points corresponding to the target cluster centers are fused to obtain a first image. It is then determined whether there are overlapping semantic points with overlapping spatial locations in the first image. If there are overlapping semantic points with overlapping spatial locations, the target semantic point corresponding to the overlapping semantic point is determined based on the maturity information corresponding to the overlapping semantic point. Finally, the fused image is determined based on the first image and the target semantic point. Based on the fused image and the base map, a target map corresponding to the target scene is generated.

2. The image processing method as described in claim 1, characterized in that, The first 3D point cloud includes a first semantic point corresponding to the first driving trajectory; the step of generating a base map corresponding to the target scene based on the first 3D point cloud includes: Preprocess the first 3D point cloud based on the first semantic point to obtain the first target 3D point cloud; Based on the first semantic points of the first target 3D point cloud, the first driving trajectory of each of the first target 3D point clouds is aligned to obtain the base map.

3. The image processing method as described in claim 2, characterized in that, The step of preprocessing the first 3D point cloud based on the first semantic point to obtain the first target 3D point cloud includes: Based on the preset size information corresponding to the preset semantic point, the first semantic point and the height difference between adjacent 3D point clouds in the first 3D point cloud, the first 3D point cloud is filtered to obtain the filtered first 3D point cloud. Based on the shooting time of the BEV image corresponding to the first filtered 3D point cloud, the first filtered 3D point cloud is further filtered to obtain the first target 3D point cloud.

4. The image processing method as described in claim 1, characterized in that, The step of determining the target 3D point cloud in the second 3D point cloud based on the base map and the second driving trajectory includes: The second driving trajectory is segmented to obtain sub-driving trajectories, and the sub-3D point cloud corresponding to each sub-driving trajectory is obtained from the second 3D point cloud; Based on the sub-trajectory, each sub-3D point cloud is aligned with the base map to obtain the confidence level corresponding to each sub-3D point cloud; Based on the confidence level, the target 3D point cloud is determined in the sub-3D point cloud.

5. The image processing method as described in claim 4, characterized in that, The step of aligning each of the sub-3D point clouds with the base map based on the sub-driving trajectory to obtain the confidence level corresponding to each sub-3D point cloud includes: Based on the sub-driving trajectory, each sub-3D point cloud is aligned with the base map to determine the distance between each sub-driving trajectory and the corresponding driving trajectory in the base map; The confidence level is determined based on the distance.

6. The image processing method as described in claim 4, characterized in that, The step of determining the target 3D point cloud in the sub-3D point cloud based on the confidence level includes: In the sub-3D point cloud, a target sub-3D point cloud with a confidence level greater than a preset confidence level is obtained, and the target sub-3D point cloud is used as the target 3D point cloud.

7. The image processing method according to any one of claims 1 to 6, characterized in that, The steps for obtaining the 3D point cloud corresponding to the crowdsourced trajectory information of the target scene include: Obtain the crowdsourcing trajectory information; Deep learning is used to identify the BEV map to obtain semantic points corresponding to each driving trajectory; The 3D point cloud is determined based on the semantic points corresponding to each driving trajectory and the vehicle posture information.

8. The image processing method as described in claim 7, characterized in that, The step of obtaining the crowdsourcing trajectory information includes: The vehicle attitude information uploaded by the vehicle is obtained, wherein the vehicle is equipped with a GNSS sensor, an IMU sensor and a wheel speed meter, and the vehicle determines the vehicle attitude information based on the monitoring results of the GNSS sensor, the monitoring results of the IMU sensor and the monitoring results of the wheel speed meter; The uploaded BEV image is obtained, wherein the vehicle is equipped with a front camera, a left camera, and a right camera, and the vehicle determines the BEV image based on the images captured by the front camera, the left camera, and the right camera.

9. An image processing apparatus, characterized in that, The image processing device includes: The acquisition module is used to acquire 3D point clouds corresponding to crowdsourced trajectory information of the target scene. The crowdsourced trajectory information includes multiple point cloud bird's-eye view BEV images corresponding to each driving trajectory of the vehicle in the target scene and vehicle posture information corresponding to each BEV image. The 3D point cloud includes a first 3D point cloud corresponding to a first driving trajectory and a second 3D point cloud corresponding to a second driving trajectory. The base map generation module is used to generate a base map corresponding to the target scene based on the first 3D point cloud and the first driving trajectory. The determination module is used to determine the target 3D point cloud in the second 3D point cloud based on the base map and the second driving trajectory; The base map generation module is used to generate a target map corresponding to the target scene based on the target 3D point cloud and the base map; The base map generation module is further configured to: Based on the type information corresponding to the second semantic points of the target 3D point cloud, a clustering operation is performed on the target 3D point cloud to obtain target cluster centers. Specifically, based on the second semantic points of the target 3D point cloud, a clustering operation is performed to obtain cluster centers corresponding to the target 3D point cloud. Based on the number of trajectories of the second driving trajectory corresponding to the cluster centers in the target 3D point cloud, and the number of times the cluster centers appear in the target 3D point cloud, maturity information corresponding to each cluster center is determined. Maturity information = number of appearances / number of trajectories. Cluster centers with maturity information greater than a preset maturity value are selected as the target cluster centers. Based on the size information corresponding to the base image, the second semantic points corresponding to the target cluster centers are fused to obtain a fused image. Specifically, based on the size information corresponding to the base image, the second semantic points corresponding to the target cluster centers are fused to obtain a first image. It is then determined whether there are overlapping semantic points with overlapping spatial locations in the first image. If there are overlapping semantic points with overlapping spatial locations, the target semantic point corresponding to the overlapping semantic point is determined based on the maturity information corresponding to the overlapping semantic point. Finally, the fused image is determined based on the first image and the target semantic point. Based on the fused image and the base map, a target map corresponding to the target scene is generated.

10. An image processing device, characterized in that, The image processing device includes: a memory, a processor, and an image processing program stored in the memory and executable on the processor, wherein the image processing program, when executed by the processor, implements the steps of the image processing method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image processing program, which, when executed by a processor, implements the steps of the image processing method as described in any one of claims 1 to 8.

Images