A Visual SLAM Method for Semantic Segmentation

Abstract

The present invention relates to a visual SLAM method for semantic segmentation, comprising: obtaining pixel points of a target object in a target environment image by using a semantic model; converting the pixel points of the target object into 3D semantic point cloud data in a vehicle coordinate system; predicting the driving environment state of the vehicle at the current moment according to the change of vehicle posture data in a preset historical time period, determining the target increment of the vehicle posture data at the current moment in the vehicle coordinate system, combining the vehicle posture data in the map coordinate system at the previous moment, determining the predicted vehicle posture data in the map coordinate system and the predicted 3D semantic point cloud data in the map coordinate system, matching with a three-dimensional semantic map, obtaining the target 3D semantic point cloud data at the current moment; comparing with the predicted vehicle posture data in the map coordinate system, determining the target vehicle posture data at the current moment in the map coordinate system. Thus, it is possible to combine environmental image information in a complex road environment to improve the accuracy and reliability of the posture data of the vehicle in the global coordinates.

Description

Technical Field

[0001] The present invention relates to the field of vehicle autonomous driving, and specifically designs a visual SLAM method for semantic segmentation. Background Art

[0002] With the development of society, people are paying more and more attention to autonomous driving technology. Autonomous driving technology is a technology that perceives road scenes through the on-board sensor system, obtains positioning information of roads, vehicle positions and obstacles based on the perceived road scenes, controls the steering and speed of the vehicle based on the information obtained, automatically plans the driving route and controls the vehicle to reach the predetermined target.

[0003] Assume that the vehicle position is located by using a wheel speed meter, and the plane position of the target object in the road scene is collected by a camera, and intelligent driving is performed based on the obtained vehicle position and the plane position of the target object in the collected road scene. However, when the vehicle performs intelligent driving based on the above information, if it is located in a complex road environment (uneven road environment, such as up and down ramps), the vehicle cannot accurately obtain the current driving state information of the vehicle and the positioning information of the target object in the environment, and ultimately cannot accurately control the vehicle to reach the predetermined target, and positioning failure may cause serious traffic accidents.

[0004] Therefore, it is extremely important to consider environmental information in the actual application of autonomous driving. Based on this, a semantic segmentation visual SLAM (Simultaneous Localization and Mapping) method is proposed. Summary of the invention

[0005] The present invention aims at the technical problems existing in the prior art and can combine environmental image information in a complex road environment to improve the accuracy and reliability of vehicle posture data in global coordinates.

[0006] The technical solution of the present invention to solve the above technical problems is as follows: a visual SLAM method for semantic segmentation, comprising:

[0007] S101, acquiring a target environment image in a vehicle coordinate system within a preset time window according to a vehicle-mounted image acquisition device, and acquiring vehicle position data in a local coordinate system according to a vehicle-mounted positioning device;

[0008] S102, obtaining a target object and its pixel points in a target environment image using a pre-trained semantic model;

[0009] S103, converting the pixel points of the target object into 3D semantic point cloud data in the vehicle coordinate system according to the vehicle posture data in the local coordinate system;

[0010] S104, predicting the current driving environment state of the vehicle according to the change of the vehicle posture data in the vehicle coordinate system within a preset historical time period;

[0011] S105, determining predicted vehicle posture data at the current moment in the map coordinate system according to a target increment of the vehicle posture data at the current moment in the vehicle coordinate system determined based on the driving environment state relative to the vehicle posture data at the previous moment and the vehicle posture data at the previous moment in the map coordinate system;

[0012] S106, determining predicted 3D semantic point cloud data in the map coordinate system according to the predicted vehicle posture data in the map coordinate system, and matching it with the constructed three-dimensional semantic map to obtain target 3D semantic point cloud data at the current moment;

[0013] S107, determining the target position and posture data of the vehicle at the current moment in the map coordinate system according to the predicted position and posture data of the vehicle in the map coordinate system, the difference between the predicted 3D semantic point cloud data and the matched target 3D semantic point cloud data.

[0014] The beneficial effects of the present invention are:

[0015] The environmental images of multiple sub-time nodes acquired within the preset time window by the on-board image acquisition device are transformed by inverse perspective and superimposed to form a target environmental image, providing rich environmental perception information; by analyzing the incremental change curve of historical vehicle posture data within the preset historical time period, combined with the predicted current driving environment state, the predicted vehicle posture increment compared to the previous moment is adjusted to make it better adapt to the complex and changeable driving environment, thereby determining more accurate vehicle predicted posture data in the map coordinate system and its corresponding 3D semantic point cloud data; by matching the predicted 3D semantic point cloud data with the three-dimensional semantic map, the accuracy of the predicted posture is evaluated, and the final vehicle target posture data is obtained, which further improves the accuracy of vehicle posture data estimation and realizes real-time, high-precision vehicle posture optimization and positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a visual SLAM method for semantic segmentation according to an embodiment of the present invention;

[0017] Figure 2 The present invention is a flowchart of an embodiment of predicting the driving environment state of a vehicle at a current moment according to changes in vehicle posture data in a vehicle coordinate system within a preset historical time period. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0019] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0020] In the description of the present application, the term "for example" is used to mean "used as an example, illustration or description". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid unnecessary details to obscure the description of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0021] Example 1

[0022] Figure 1 It is a flowchart of a visual SLAM method for semantic segmentation according to an embodiment of the present invention.

[0023] like Figure 1 As shown, a visual SLAM method for semantic segmentation includes the following steps:

[0024] S101, obtaining a target environment image in a vehicle coordinate system within a preset time window according to a vehicle-mounted image acquisition device, and obtaining vehicle position data in a local coordinate system according to a vehicle-mounted positioning device.

[0025] Among them, the vehicle coordinate system is a three-dimensional coordinate system established with a fixed point of the vehicle as the coordinate origin, which is used to describe the state of the vehicle and its surrounding environment. The coordinate origin changes as the vehicle moves; while the local coordinate system is defined based on the initial position and direction of the vehicle, that is, the vehicle coordinate system at the starting moment. As the vehicle moves, the vehicle coordinate system moves with it, while the local coordinate system remains unchanged and is always fixed at the starting position.

[0026] Among them, the vehicle posture data includes vehicle position information and vehicle attitude (direction) information. The vehicle position information includes the vehicle's position information on the X-axis, Y-axis, and Z-axis. The vehicle attitude information includes the vehicle's pitch angle, heading angle, and roll angle.

[0027] In some embodiments, the vehicle-mounted image acquisition device includes one or more cameras, and the vehicle-mounted image acquisition device acquires the target environment image in the vehicle coordinate system within a preset time window, specifically including:

[0028] A1. Stitching environmental images collected by the vehicle-mounted image acquisition device within a preset time window to obtain a first environmental image.

[0029] Specifically, through image stitching technology, all environmental images collected by the on-board image acquisition device are combined into a first environmental image to cover a 360-degree field of view around the vehicle, thereby providing a complete view of the vehicle's surroundings and providing rich image information for subsequent processing.

[0030] A2. Perform inverse perspective transformation on the first environment image to obtain a target environment image.

[0031] Specifically, the inverse perspective transformation is an image correction technology that maps the first environment image to the ground plane through pre-calibrated camera parameters and ground assumptions (i.e., the ground is a strict plane) to form a bird's-eye view, i.e., the target environment image. The pixels in the target environment image directly correspond to the physical positions on the actual ground, which is convenient for subsequent processing and analysis. The inverse perspective transformation eliminates the perspective distortion in the image, so that the size and position of the objects in the image are consistent with the objects on the actual ground, which is convenient for subsequent semantic segmentation and target detection tasks. This is not elaborated in the present invention.

[0032] In some embodiments, the vehicle-mounted positioning device includes an inertial measurement module and a wheel speed meter, and obtaining vehicle posture data in a local coordinate system according to the vehicle-mounted positioning device specifically includes:

[0033] B1. Obtain inertial measurement data from the inertial measurement module and wheel speed meter data transmitted by the wheel speed meter.

[0034] Specifically, the inertial measurement module can be rigidly connected to the vehicle body. The inertial measurement unit (IMU) is a device that measures the three-axis attitude angle (or angular velocity) and acceleration of an object. The inertial measurement module contains three single-axis accelerometers and three single-axis gyroscopes. The accelerometer detects the acceleration signal of the object in the independent three axes of the carrier coordinate system, while the gyroscope detects the angular velocity signal of the carrier relative to the navigation coordinate system, measures the angular velocity and acceleration of the object in three-dimensional space, and uses this to calculate the attitude of the object. The wheel speed meter is installed on the wheel to measure the number of rotations of the vehicle, and then determines the vehicle's movement distance based on the number of rotations and the preset conversion coefficient.

[0035] B2. Acquire the first vehicle posture data based on the inertial measurement data and the wheel speed meter data and a dead reckoning algorithm.

[0036] B3. Construct a visual odometer model based on the target environment image, inertial measurement data and wheel speed meter data, and obtain the second vehicle posture data through the model.

[0037] B4. Use a spherical interpolation algorithm and an extended Kalman filter algorithm to fuse the first vehicle posture data and the second vehicle posture data to generate vehicle posture data.

[0038] The spherical interpolation algorithm is used to process data involving angles or directions to ensure the accuracy and consistency of the pose data during the fusion process; the extended Kalman filter is an optimal state estimation method for nonlinear systems, which fuses the data of the IMU and wheel speed meter (usually regarded as the state equation of the system) with the data of the visual odometer (usually regarded as the observation equation of the system) to obtain a more accurate vehicle pose estimation, which can effectively combine information data from different sources and gradually reduce errors through iterative calculations, thereby improving the accuracy of pose estimation. This invention does not go into details.

[0039] S102, using the pre-trained semantic model to perform semantic segmentation on the target environment image to obtain the target object and its pixel points.

[0040] The semantic model is an image classification model that outputs a classification map with the same resolution as the input image. The classification value of each pixel represents the category identifier of the target object to which the pixel belongs in the input image. The target objects include but are not limited to trees, roads, lane lines, and vehicles.

[0041] S103, converting the pixel points of the target object into 3D semantic point cloud data in the vehicle coordinate system according to the vehicle posture data in the local coordinate system.

[0042] In some embodiments, the target environment image includes multiple sub-target environment images captured at consecutive time points within a preset time window. For example, the target environment image includes but is not limited to three sub-target environment images captured at three consecutive time points obtained by capturing three times within a preset time window of 3 seconds.

[0043] Furthermore, the step S103 specifically includes:

[0044] C1. Through the preset size factor and camera parameters, the pixel points of the target object of each sub-target environment image are converted from the image coordinate system to the 2D semantic point cloud data in the vehicle coordinate system.

[0045] Among them, the coordinate information of the pixel point of the target object in the image coordinate system is determined according to the camera parameters (focal length, optical center position and distortion coefficient), and the size factor is pre-set according to the camera parameters and the vehicle coordinate system, and is used to convert the pixel unit in the image into the physical unit in the real world. The determination of the size factor usually depends on the calibration process of the camera. This content usually involves the use of camera intrinsic parameters, and can refer to the existing related technologies, which will not be elaborated in the present invention.

[0046] C2. According to the vehicle posture data in the local coordinate system, the 2D semantic point cloud data in the vehicle coordinate system is converted into 3D semantic point cloud data in the local coordinate system.

[0047] This content usually involves the transformation matrix operation implementation of the rotation and translation of the coordinate system, which is not elaborated in the present invention.

[0048] Among them, 3D semantic point cloud data refers to a set of pixel points of a target object with data on the x, y, and z coordinate axes, which is used to describe the geometric information of the target object in three-dimensional space.

[0049] C3. Superimpose the 3D semantic point cloud data in the local coordinate system corresponding to all sub-target environment images to obtain the 3D semantic point cloud data of the target environment image in the local coordinate system.

[0050] Specifically, for multiple sub-target environment images collected at consecutive time points, the 3D semantic point cloud data in their corresponding local coordinate systems are superimposed. This process is similar to accumulating data in time, and can generate a 3D semantic point cloud data containing richer information and higher data reliability. For example, suppose there are 3 sub-target environment images, each of which generates a 3D semantic point cloud data in a local coordinate system. These data are superimposed to form a larger 3D semantic point cloud data set.

[0051] C4. Based on the vehicle posture data in the local coordinate system, the 3D semantic point cloud data of the target environment image in the local coordinate system is inversely transformed to the vehicle coordinate system.

[0052] The inverse transformation in this content is opposite to step C2, and involves the implementation of the inverse transformation matrix operation of the reverse rotation and translation of the coordinate system.

[0053] S104, predicting the driving environment state of the vehicle at the current moment according to the change of the vehicle posture data in the vehicle coordinate system within the preset historical time period closest to the current moment.

[0054] In some embodiments, the driving environment state includes a stable state and a fluctuating state, such as Figure 2 As shown, step S104 specifically includes:

[0055] S201, obtaining incremental data between vehicle posture data at all times within a preset historical time period, including position increment (displacement) and direction increment (heading angle change), and generating curve graphs of position increment-time and direction increment-time.

[0056] Specifically, the vehicle posture data within a preset historical time period includes position (X, Y, Z) and heading angle θ. For the vehicle posture data at adjacent moments, the position increments ΔX, ΔY, ΔZ and the direction increment Δθ are calculated.

[0057] Position increment-time curve diagram: With time as the horizontal axis and position increments ΔX, ΔY, and ΔZ as the vertical axes, three curves are drawn respectively.

[0058] Direction increment-time curve graph: Draw a curve with time as the horizontal axis and direction increment Δθ as the vertical axis.

[0059] S202, for the position increment and the direction increment, respectively calculate the corresponding increment standard deviations, if the position increment standard deviation is less than the position increment threshold and the direction standard deviation is less than the direction increment threshold, then determine that the driving environment state is a stable state; otherwise, it is a fluctuating state.

[0060] Specifically, the incremental standard deviation includes σ_ΔX, σ_ΔY, σ_ΔZ, and σ_Δθ. The average of σ_ΔX, σ_ΔY, and σ_ΔZ can be taken as the position increment standard deviation. The incremental standard deviation reflects the degree of discreteness of the data within the preset historical time period, that is, the degree of change of the increment.

[0061] S105, determining the predicted vehicle posture data at the current moment in the map coordinate system according to the target increment of the vehicle posture data at the current moment in the vehicle coordinate system determined based on the driving environment state relative to the vehicle posture data at the previous moment, and the vehicle posture data at the previous moment in the map coordinate system.

[0062] Specifically, the target increment of the vehicle posture data at the current moment in the determined vehicle coordinate system relative to the vehicle posture data at the previous moment is accumulated to the vehicle posture data at the previous moment in the map coordinate system to obtain the vehicle predicted posture data at the current moment in the map coordinate system.

[0063] In some embodiments, the target increment of the vehicle posture data at the current moment in the vehicle coordinate system relative to the vehicle posture data at the previous moment is determined specifically as follows:

[0064] S301, when the driving environment state is stable, the first increment of the vehicle posture data at the current moment in the vehicle coordinate system predicted by the on-board positioning device (IMU, wheel speed meter) combined with the vehicle dynamics model relative to the vehicle posture data at the previous moment is determined as the target increment.

[0065] S302: When the driving environment state is a fluctuating state, the first increment is adjusted according to the 3D semantic point cloud data of the target environment image at the current moment in the vehicle coordinate system to obtain a target increment.

[0066] Specifically, step S302 includes:

[0067] E1. Determine the target object and its three-dimensional coordinates from the 3D semantic point cloud data of the target environment image at the current moment in the vehicle coordinate system.

[0068] Among them, based on all target objects in the driving environment, a target object set is pre-constructed, and the target object set represents the target objects in the driving environment that can affect the normal driving state of the vehicle and cause fluctuations, such as roadblocks and traffic cones.

[0069] Furthermore, according to the target object set, all target objects in the target environment image are matched therewith to determine the target object.

[0070] E2. Determine the relative distance D between the target object and the vehicle based on the three-dimensional coordinates of the target object in the vehicle coordinate system.

[0071] E3. Determine a first threshold value d1 and a second threshold value d2 based on a preset safety distance. The first threshold value is smaller than the second threshold value, and the second threshold value is smaller than the safety distance, to indicate different degrees of danger during driving. Obtain the amplitude factor k according to the following formula:

[0072]

[0073] Among them, kmax is set to a maximum amplitude factor greater than 1, which represents the maximum adjustment amplitude under extreme conditions and is set based on historical experience.

[0074] E4. Determine the target increment according to the amplitude factor: Δ=α1*Δ1+α2*Δ1*k, where Δ represents the target increment, Δ1 represents the first increment, k represents the amplitude factor, α1 and α2 represent the weight values ​​of the increment of the predicted vehicle posture relative to the previous moment affected by the on-board positioning device and the safety distance, respectively, and α1+α2=1.

[0075] S106, determining the predicted 3D semantic point cloud data in the map coordinate system according to the predicted vehicle posture data at the current moment in the map coordinate system, and matching it with the constructed three-dimensional semantic map to obtain the target 3D semantic point cloud data at the current moment.

[0076] In some embodiments, step S106 specifically includes:

[0077] D1. According to the predicted vehicle posture data at the current moment in the map coordinate system, the 3D semantic point cloud data in the vehicle coordinate system is converted to the map coordinate system to form predicted 3D semantic point cloud data.

[0078] This content involves the process of coordinate transformation, which converts the 3D semantic point cloud data in the vehicle coordinate system to the global map coordinate system based on the vehicle's predicted posture data. In this way, the point cloud data can be matched with the 3D semantic map.

[0079] D2. Match the predicted 3D semantic point cloud data at the current moment in the map coordinate system with the three-dimensional semantic map, and retrieve the best matching result through a preset matching algorithm, thereby obtaining the target 3D semantic point cloud data at the current moment in the map coordinate system.

[0080] Specifically, the constructed three-dimensional semantic map is the three-dimensional semantic map obtained at the previous moment, including the geometric coordinate information of the environment and the corresponding semantic information; the preset matching algorithm includes but is not limited to the nearest neighbor search and the ICP algorithm. By matching the predicted 3D semantic point cloud data with the three-dimensional semantic map through the preset matching algorithm, the deviation of the pose data and the point cloud data caused by sensor noise or model prediction error can be corrected to obtain more accurate results.

[0081] S107, determining the target position data of the vehicle at the current moment in the map coordinate system according to the predicted position data of the vehicle at the current moment in the map coordinate system, the difference between the predicted 3D semantic point cloud data and the matched target 3D semantic point cloud data.

[0082] Specifically, by comparing the difference between the predicted 3D semantic point cloud data and the matched target 3D semantic point cloud data, the accuracy of the vehicle's predicted pose data can be evaluated. If there is a significant difference, an optimization algorithm (such as least squares method, Kalman filter) can be used to correct the vehicle's predicted pose data to obtain more accurate vehicle target pose data. This process is an iterative optimization process that aims to improve the accuracy and robustness of vehicle pose data estimation.

[0083] Therefore, by combining the image information of the vehicle's vicinity obtained by the on-board image acquisition device and the predicted vehicle posture data information obtained by the on-board positioning device, the pixel points of the target object in the environmental image are processed into 3D semantic point cloud data in the vehicle coordinate system according to the vehicle posture data in the local coordinate system. Even if the vehicle is located in an uneven road environment, the vehicle's current three-dimensional positioning information and the three-dimensional positioning information of the target object in the environment can be accurately obtained. Combined with the environmental information, high-precision vehicle posture data and 3D semantic point cloud data in the map coordinate system can be obtained.

[0084] Specifically, the environmental images of multiple sub-time nodes acquired within a preset time window by the on-board image acquisition device are transformed by inverse perspective and superimposed to form a target environmental image, providing rich environmental perception information; by analyzing the incremental change curve of historical vehicle posture data within a preset historical time period and combining it with the predicted current driving environment state, the predicted vehicle posture increment compared to the previous moment is adjusted to make it better adapt to the complex and changeable driving environment, thereby determining more accurate vehicle predicted posture data in the map coordinate system and its corresponding 3D semantic point cloud data; by matching the predicted 3D semantic point cloud data with the three-dimensional semantic map, the accuracy of the predicted posture is evaluated, and the final vehicle target posture data is obtained, which further improves the accuracy of vehicle posture data estimation and realizes real-time, high-precision vehicle posture optimization and positioning accuracy.

[0085] Example 2

[0086] In actual scenarios, the target objects (trees, buildings, car logos, etc.) in the vehicle's driving environment are usually extremely rich. Autonomous driving vehicles usually do not need to process 360-degree panoramic environmental information around the vehicle, especially in high-speed driving or urban congestion scenarios. Although the panoramic information is comprehensive, it also contains many redundant objects that are not related to the current driving direction. Processing panoramic images not only increases the burden on the model and image processing, but may also introduce unnecessary noise and interference.

[0087] In some embodiments, in step A1, the environment images collected by the vehicle-mounted image acquisition device within a preset time window are stitched together to obtain a first environment image, and the process further includes:

[0088] S401, based on the target increment of the vehicle posture data at the current moment in the vehicle coordinate system determined in step S105 relative to the vehicle posture data at the previous moment, obtain the vehicle predicted posture data at the current moment in the vehicle coordinate system, and obtain the vehicle's predicted driving direction (determined based on the heading angle in the vehicle predicted posture data).

[0089] S402, determining a region of interest (ROI) according to the predicted driving direction of the vehicle, where the region is located within a preset angle range on both sides of the driving direction of the vehicle.

[0090] The preset angle ranges on both sides of the vehicle's driving direction can be adjusted according to the actual environment. For example, the preset angle range can be set to 60°.

[0091] S403 , based on the first environment image and according to the predicted driving direction of the vehicle, extracting an image within the ROI from the first environment image as a new first environment image to replace the original first environment image.

[0092] This step may be implemented through an image mask operation based on the region of interest, which will not be elaborated in the present invention.

[0093] Therefore, the target environment image is processed through image mask technology, and only the local image around the vehicle's driving direction is semantically segmented, which significantly reduces the amount of image data that needs to be processed, thereby reducing the computational burden of the model and improving real-time performance; since unnecessary image area processing is reduced, the system can generate and process 3D semantic point cloud data more quickly, providing more timely support for the subsequent accurate determination of the vehicle's real-time posture data; ROI extraction can focus more on the key information of the vehicle's driving direction, improving the subsequent detection efficiency and accuracy of targets (target objects that potentially affect vehicle driving).

[0094] Example 3

[0095] In Example 2, how to process the environment image around the vehicle is a key issue. Although the traditional panoramic image processing method is comprehensive, in complex environments such as high-speed driving or urban congestion, the panoramic image contains a lot of redundant information, which is inefficient to process and may introduce noise and interference. This embodiment further improves Example 2 to improve the processing efficiency of semantic segmentation images and improve the accuracy and effectiveness of obtaining 3D semantic point cloud data in the vehicle coordinate system.

[0096] In some embodiments, after step S402, the following steps may also be included:

[0097] S501, in a vehicle-mounted image acquisition device, a camera that is close to the predicted driving direction of the vehicle and installed on the front side of the vehicle is determined as an active recognition camera for acquiring environmental images within a ROI area in the predicted driving direction of the vehicle.

[0098] Among them, the active recognition camera is set as a surround-view camera, and the ROI area is set to a fan-shaped area centered on the vehicle and covering a preset angle range on both sides of the vehicle's predicted driving direction.

[0099] S502, adjusting relevant parameters (viewing angle, focal length, resolution) of the active recognition camera according to the determined ROI area, to ensure that the active recognition camera can clearly and accurately capture the environmental image within the ROI area when performing acquisition work with the adjusted parameters.

[0100] S503, determining the environment image collected by the active recognition camera within the preset time window according to the adjusted relevant parameters as the first environment image, and executing step A2 to obtain the target environment image.

[0101] Therefore, by determining the active recognition camera, the environmental image is directly collected for the ROI area determined according to the predicted driving direction of the vehicle. Compared with Example 2, there is no need for subsequent panoramic image stitching and ROI image processing steps, which greatly reduces the amount of calculation and improves the real-time performance of data processing. At the same time, due to the direct processing of the ROI image, the computational burden of the semantic segmentation model is significantly reduced, and it can respond to environmental changes more quickly, thereby improving processing efficiency. By adjusting the viewing angle, focal length, resolution and other parameters of the active recognition camera, it is ensured that the collected environmental image can completely cover the image in the ROI area and ensure its clarity and accuracy, thereby improving the accuracy of subsequent semantic segmentation and target object extraction to generate semantic point cloud data. In addition, the direct collection of the ROI image reduces the noise and interference that may be introduced by panoramic image stitching and ROI image extraction, further improving the image data quality.

[0102] Example 4

[0103] The accuracy and comprehensiveness of environmental perception are crucial to ensure the accuracy of vehicle posture determination. However, in practical applications, due to the occlusion of dynamic objects in complex road environments, active recognition cameras may not be able to fully capture all important static object information in the driving direction, thereby reducing the reliability and accuracy of subsequent semantic point cloud data matching in the map. In addition, traditional semantic segmentation models usually only output the category information of the target object, but cannot distinguish between static and dynamic objects, which limits the ability to detect and process occlusion situations.

[0104] In some embodiments, in a vehicle-mounted image acquisition device, cameras installed on one or more sides of the vehicle are calibrated as follow-up cameras to capture and actively identify environmental areas outside the camera coverage area, so as to supplement the environmental image in specific scenarios.

[0105] In some embodiments, the semantic model used for semantic segmentation in step S102 is also used to output attribute labels of the target object, and the attribute labels include static objects and dynamic objects.

[0106] Specifically, the pre-trained semantic model is obtained by the following steps:

[0107] F1: Based on the original semantic model of target object recognition, the pixels in the sample data used for training are labeled with attribute labels;

[0108] F2. For each sample data, the training sample data is used as input into the semantic model to be trained, and the prediction result of each pixel in the training sample is output. The prediction result includes the target object corresponding to the pixel and its attribute label;

[0109] F3. Adjust the model parameters in the semantic model with the optimization goal of minimizing the difference between the prediction result of each pixel in the training sample and the annotation information of each pixel.

[0110] In some embodiments, in step S102, semantic segmentation is performed on the target environment image to obtain the target object and its pixel points, further comprising:

[0111] S601, obtaining a target object and its attribute label in a semantically segmented image of a target environment image, and comparing the target object whose attribute label is a static object in the target environment image at a current moment with that in a previous moment.

[0112] S602, judging whether the occlusion condition is met based on the comparison result, if so, triggering the follow-up camera to supplement the environment image at the current moment to obtain a supplementary environment image, and obtaining the target object and its pixel points with the attribute label of a static object in the supplementary environment image, and merging it with the target object and its pixel points of the target environment image.

[0113] The occlusion condition is set as follows: if no new static object appears at the current moment or the number of static objects decreases compared to the previous moment, it is determined that there may be a situation in which a static object is occluded by a dynamic object in the target environment image at the current moment.

[0114] S603, until it is detected that the target object with the attribute label of a static object in the target environment image at a certain moment and the previous moment does not meet the occlusion condition, the follow-up camera is turned off and the original active recognition camera is restored to work alone to collect the environment image.

[0115] Therefore, a follow-up camera is introduced to supplement the environmental image of the active recognition camera. Although the active recognition camera can collect environmental images within the ROI area in the predicted driving direction of the vehicle, it is still inevitable that dynamic objects will occlude static objects in complex environments. The introduction of the follow-up camera can supplement the collection of environmental images outside the coverage area of ​​the active recognition camera, especially when it is detected that static objects may be occluded by dynamic objects. The follow-up camera can capture static objects collected from other perspectives, thereby enriching the environmental perception information and improving the accuracy of subsequent 3D semantic point cloud data generation and map matching in the vehicle coordinate system, as well as the accuracy of vehicle posture data.

[0116] In addition, the follow-up camera is only triggered when an occlusion condition is detected and remains off at other times. The active recognition camera and the follow-up camera work together to ensure the accuracy of semantic segmentation and subsequent vehicle posture and point cloud data through key and rich environmental images, while also optimizing resource utilization, reducing unnecessary computing and energy consumption, and improving the efficiency of vehicle positioning, three-dimensional semantic map matching, and visual odometry model updates.

[0117] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0118] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0122] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0123] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A visual SLAM method for semantic segmentation, characterized in that: include: S101, obtaining a target environment image in a vehicle coordinate system within a preset time window according to a vehicle-mounted image acquisition device, and obtaining vehicle posture data in a local coordinate system according to a vehicle-mounted positioning device; the vehicle coordinate system is a three-dimensional coordinate system established with a certain fixed point of the vehicle as a coordinate origin, and the vehicle coordinate system changes with the movement of the vehicle; the local coordinate system is set to the vehicle coordinate system at the starting time when the vehicle starts to travel, and remains unchanged; S102, obtaining a target object and its pixel points in a target environment image using a pre-trained semantic model; S103, converting the pixel points of the target object into 3D semantic point cloud data in the vehicle coordinate system according to the vehicle posture data in the local coordinate system; S104, predicting the current driving environment state of the vehicle according to the change of the vehicle posture data in the vehicle coordinate system within a preset historical time period; S105, determining predicted vehicle posture data at the current moment in the map coordinate system according to a target increment of the vehicle posture data at the current moment in the vehicle coordinate system determined based on the driving environment state relative to the vehicle posture data at the previous moment and the vehicle posture data at the previous moment in the map coordinate system; S106, determining predicted 3D semantic point cloud data in the map coordinate system according to the predicted vehicle posture data in the map coordinate system, and matching it with the constructed three-dimensional semantic map to obtain target 3D semantic point cloud data at the current moment; S107, determining the target position and posture data of the vehicle at the current moment in the map coordinate system according to the predicted position and posture data of the vehicle in the map coordinate system, the difference between the predicted 3D semantic point cloud data and the matched target 3D semantic point cloud data; The driving environment state includes a stable state and a fluctuating state, and S104 specifically includes: S201, obtaining incremental data between vehicle posture data at all times within a preset historical time period, including position increment and direction increment, and generating a curve graph of position increment-time and direction increment-time; S202, based on the position increment-time and direction increment-time curves, respectively obtain the corresponding position increment standard deviation and direction increment standard deviation, if the position increment standard deviation is less than the position increment threshold and the direction standard deviation is less than the direction increment threshold, then determine that the driving environment state is a stable state; otherwise, it is a fluctuating state; The S105 includes: Accumulating the target increment of the vehicle posture data at the current moment in the determined vehicle coordinate system relative to the vehicle posture data at the previous moment to the vehicle posture data at the previous moment in the map coordinate system to obtain the vehicle predicted posture data at the current moment in the map coordinate system; The target increment of the vehicle posture data at the current moment in the determined vehicle coordinate system relative to the vehicle posture data at the previous moment is determined as: S301, when the driving environment state is a stable state, determining a first increment of the vehicle posture data at the current moment in the vehicle coordinate system predicted by the vehicle-mounted positioning device in combination with the vehicle dynamics model relative to the vehicle posture data at the previous moment as a target increment; S302: When the driving environment state is a fluctuating state, the first increment is adjusted according to the 3D semantic point cloud data of the target environment image at the current moment in the vehicle coordinate system to obtain a target increment.

2. The visual SLAM method for semantic segmentation according to claim 1, characterized in that The vehicle position data includes vehicle position information and vehicle attitude information. The vehicle position information includes the position information of the vehicle on the X-axis, Y-axis, and Z-axis. The vehicle attitude information includes the pitch angle, heading angle, and roll angle of the vehicle.

3. The visual SLAM method for semantic segmentation according to claim 1, characterized in that The vehicle-mounted image acquisition device includes one or more cameras, and the step of acquiring the target environment image in the vehicle coordinate system within a preset time window according to the vehicle-mounted image acquisition device specifically includes: A1. Stitching the environment images collected by the vehicle-mounted image acquisition device within a preset time window to obtain a first environment image; A2. Perform inverse perspective transformation on the first environment image to obtain a target environment image.

4. The visual SLAM method for semantic segmentation according to claim 1, characterized in that The S302 further includes: E1. Determine a target object and its three-dimensional coordinates from the 3D semantic point cloud data of the target environment image at the current moment in the vehicle coordinate system; the target object is determined according to the target object of the target environment image and a pre-constructed target object set, and the target object set is used to characterize the target object that can affect the normal driving state of the vehicle and cause fluctuations in the driving environment; E2. Determine the relative distance between the target object and the vehicle according to the three-dimensional coordinates of the target object in the vehicle coordinate system; E3. Determine a first threshold d1 and a second threshold d2 based on a preset safety distance, the first threshold is smaller than the second threshold, the second threshold is smaller than the safety distance, and determine an amplitude factor k according to a relative distance between the target object and the vehicle; E4. Determine the target increment according to the amplitude factor: Δ=α1*Δ1+α2*Δ1*k, where Δ represents the target increment, Δ1 represents the first increment, k represents the amplitude factor, α1 and α2 represent the weight values ​​of the increment of the predicted vehicle posture relative to the previous moment affected by the on-board positioning device and the safety distance, respectively, and α1+α2=1.

5. The visual SLAM method for semantic segmentation according to claim 3, characterized in that: In step A1, the environment images collected by the vehicle-mounted image acquisition device within a preset time window are stitched together to obtain a first environment image, and the step further includes: S401, determining the predicted vehicle posture data at the current moment in the vehicle coordinate system according to the target increment of the vehicle posture data at the current moment in the vehicle coordinate system determined in step S105 relative to the vehicle posture data at the previous moment, and obtaining the predicted driving direction of the vehicle; S402, determining an ROI region according to the predicted driving direction of the vehicle, the region being located within a preset angle range on both sides of the driving direction of the vehicle; S403 , based on the first environment image and according to the predicted driving direction of the vehicle, extracting an image within the ROI from the first environment image as a new first environment image to replace the original first environment image.

6. The visual SLAM method for semantic segmentation according to claim 5, characterized in that: After S402, the method further includes: S501, in a vehicle-mounted image acquisition device, determining a camera that is close to the predicted driving direction of the vehicle and installed on the front side of the vehicle as an active recognition camera, which is used to acquire an environmental image in a ROI area in the predicted driving direction of the vehicle; Among them, the active recognition camera is set as a surround view camera, and the ROI area is set as a fan-shaped area centered on the vehicle and covering a preset angle range on both sides of the predicted driving direction of the vehicle; S502, adjusting relevant parameters of the active recognition camera according to the determined ROI area, to ensure that the active recognition camera can collect environmental images within the ROI area when performing collection work with the adjusted parameters; S503, determining the environment image collected by the active recognition camera within the preset time window according to the adjusted relevant parameters as the first environment image, and executing step A2 to obtain the target environment image.

7. The visual SLAM method for semantic segmentation according to claim 6, characterized in that: In the vehicle-mounted image acquisition device, the cameras installed on one or more sides of the vehicle are calibrated as follow-up cameras, which are used to collect and actively identify the environment area outside the coverage area of ​​the camera; The semantic model used for semantic segmentation is also used to output attribute labels of target objects, and the attribute labels include static objects and dynamic objects.

8. The visual SLAM method for semantic segmentation according to claim 7, characterized in that: In the step S102, semantic segmentation is performed on the target environment image to obtain the target object and its pixel points, and the step further includes: S601, obtaining a target object and its attribute label in a semantically segmented image of a target environment image, and comparing the target object whose attribute label is a static object in the target environment image at a current moment with that in a previous moment; S602, judging whether the occlusion condition is met according to the comparison result, and if so, triggering the follow-up camera to supplement the environment image at the current moment to obtain a supplementary environment image, and obtaining the target object and its pixel points whose attribute label is a static object in the supplementary environment image, and merging them with the target object and its pixel points in the target environment image; The occlusion condition is set as follows: compared with the target object in the target environment image at the previous moment, no new static objects appear or the number of static objects decreases; S603, until it is detected that the target object with the attribute label of a static object in the target environment image at a certain moment and the previous moment does not meet the occlusion condition, the follow-up camera is turned off and the original active recognition camera is restored to work alone to collect the environment image.

Images