Automobile lane changing auxiliary method, system and equipment and medium

Abstract

The embodiment of the invention provides an automobile lane changing auxiliary method, system and device and a medium, and belongs to the field of advanced driving auxiliary driving. The method comprises the following steps: acquiring original point cloud data of an automobile surrounding environment, and fusing to obtain fused point cloud data; performing road surface point cloud segmentation and obstacle detection tracking based on the fused point cloud data to obtain relative position information and relative speed information of the vehicle and each obstacle; generating an obstacle position-speed feature map, generating a corresponding lane changing auxiliary label based on the obstacle position-speed feature map, and establishing a lane changing auxiliary data set; and based on the obstacle position-speed feature map and the corresponding lane changing auxiliary label, data expansion is carried out to enhance the lane changing auxiliary data set to train a lane changing auxiliary model, and the model is used to predict a lane changing behavior. The feature map and the corresponding label are generated by detecting and tracking the road obstacles, the optimal lane changing behavior suggestion is accurately provided for the driver of the vehicle in real time through the deep learning model, and the safety and comfort of high-speed driving are improved.

Description

Technical Field

[0001] This invention relates to the field of advanced driver assistance technology, and more specifically to a vehicle lane-changing assistance method, system, device, and medium. Background Technology

[0002] In highway scenarios, lane-changing maneuvers require drivers to have accurate and timely perception and judgment of the complex surrounding environment (especially vehicles rapidly approaching from the side and rear). How to provide drivers with optimal lane-changing behavior suggestions in real time and accurately to improve driving safety and comfort is a pressing issue. Current lane-changing assistance systems are insufficient in terms of the timeliness and accuracy of information acquisition, processing, and decision-making, making it difficult to meet the demands of complex road conditions and high real-time requirements in high-speed driving environments.

[0003] Existing lane change assist solutions include camera-based and millimeter-wave radar-based solutions. Camera-based solutions suffer from several drawbacks. Cameras are susceptible to interference from environmental factors such as lighting and weather, and the acquired information lacks depth perception, making it difficult to accurately measure distances to surrounding vehicles and obstacles. In complex traffic scenarios or inclement weather, image quality and recognition accuracy deteriorate significantly, leading to unreliable or malfunctioning lane change assist functions. Millimeter-wave radar-based solutions, on the other hand, are limited by the relatively low resolution of millimeter-wave radar, which has limited processing capabilities in multi-target scenarios. When multiple vehicles are present simultaneously and their movements are complex, millimeter-wave radar may experience target confusion and tracking loss, resulting in lane change assist decision errors. In summary, existing lane change assist technologies have numerous shortcomings in terms of the accuracy and completeness of information acquisition, as well as the real-time performance and reliability of lane change decisions, failing to fully meet the safe and comfortable lane change needs of high-speed vehicles in complex road conditions. Summary of the Invention

[0004] The purpose of this invention is to provide an advanced driver assistance system that detects and tracks road obstacles, generates feature maps and corresponding labels, and uses a deep learning model to accurately and in real-time provide the driver with optimal lane-changing behavior suggestions, thereby improving the safety and comfort of high-speed driving.

[0005] To achieve the above objectives, embodiments of the present invention provide a vehicle lane-changing assistance method, comprising: Collect raw point cloud data of the environment surrounding the vehicle and fuse it to obtain fused point cloud data; Based on the fused point cloud data, road point cloud segmentation and obstacle detection and tracking are performed to obtain the relative position and relative speed information of the vehicle and each obstacle; Based on the relative position information and relative velocity information, an obstacle position-velocity feature map is generated, and based on the obstacle position-velocity feature map, corresponding lane change assist labels are generated and a lane change assist dataset is established. Based on the obstacle position-velocity feature map and the corresponding lane change assist labels, data augmentation is performed to enhance the lane change assist dataset; The pre-built lane change assist model is trained based on the expanded lane change assist dataset, and the lane change assist model is used in real time during the vehicle's driving to predict lane change behavior and output as a lane change suggestion.

[0006] Optionally, based on the fused point cloud data, road surface point cloud segmentation and obstacle detection and tracking are performed to obtain the relative position and relative speed information of the vehicle and each obstacle, including: Based on the fused point cloud data, the point cloud outside the lane range is filtered according to the lane information to obtain multi-lane point cloud data; A ground segmentation algorithm is used to separate ground point clouds and non-ground point clouds in multi-lane point cloud data, and the non-ground point clouds are clustered to form multiple obstacle point cloud clusters. A multi-target tracking algorithm is used to perform cross-frame association and tracking of the relative position and relative speed between the vehicle and each obstacle. The relative position between the vehicle and each obstacle is defined as the longitudinal position, and the relative speed between the vehicle and each obstacle is defined as the longitudinal speed.

[0007] Optionally, based on the relative position information and relative velocity information, an obstacle position-velocity feature map is generated, including: The relative position and relative velocity information are mapped onto a two-dimensional raster image using rasterization projection to generate a position feature map and a velocity feature map. These two feature maps are then combined and encoded into a three-dimensional tensor structure obstacle position-velocity feature map. The rasterization projection formula is as follows: ; ; In the formula, D represents the vertical axis of the raster matrix coordinate system. The horizontal axis represents the raster matrix coordinate system, and int indicates the rounding down operation. and Represents point cloud coordinates, This represents the minimum value along the x-axis in the point cloud data. represents the minimum value in the y-axis direction of the point cloud data, w represents the actual horizontal distance represented by each grid cell in the grid matrix, and d represents the actual vertical distance represented by each grid cell in the grid matrix.

[0008] Optionally, based on the obstacle position-velocity feature map, a corresponding lane change assist label is generated, including: Based on the obstacle position-velocity feature map, a lane change assist label is generated using the vehicle's lane change coefficient relative to the obstacle and a preset safety threshold. The lane change coefficient includes a first lane change coefficient relative to the obstacle in front of the vehicle, a second lane change coefficient relative to the obstacle in front of the vehicle's left lane, a third lane change coefficient relative to the obstacle behind the vehicle's left lane, a fourth lane change coefficient relative to the obstacle in front of the vehicle's right lane, and a fifth lane change coefficient relative to the obstacle behind the vehicle's right lane. The lane change coefficients are calculated according to the formula... The calculation is performed, where v represents the relative speed between the vehicle and the obstacle, d represents the relative position between the vehicle and the obstacle, and m and n represent the corresponding weighting coefficients. When the first lane change coefficient is less than the preset safety threshold, the vehicle maintains the current speed in the current lane and is marked with the first lane change assist label; When the first lane change coefficient is greater than the preset safety threshold, the second lane change coefficient is less than the preset safety threshold, and the sum of the second and third lane change coefficients is less than the sum of the fourth and fifth lane change coefficients, the vehicle changes lanes to the left and is marked with the second lane change assist label. When the first lane change coefficient is greater than the preset safety threshold, the fourth lane change coefficient is less than the preset safety threshold, the fifth lane change coefficient is less than the preset safety threshold, and the sum of the fourth and fifth lane change coefficients is less than the sum of the second and third lane change coefficients, the vehicle changes lanes to the right and is marked as lane change assist third label. When the first lane change coefficient is greater than the preset safety threshold and the second lane change coefficient is greater than the preset safety threshold, or the third lane change coefficient is greater than the preset safety threshold and the fourth lane change coefficient is greater than the preset safety threshold, or the fifth lane change coefficient is greater than the preset safety threshold, the vehicle decelerates in the current lane and is marked as lane change assist fourth label.

[0009] Optionally, based on the obstacle position-velocity feature map and the corresponding lane change assist labels, data augmentation is performed to enhance the lane change assist dataset, including: Based on the obstacle position-velocity feature map and the corresponding lane change assist labels, data augmentation is performed in two ways: original data mirroring and random data generation. The original data mirroring method involves horizontally flipping the obstacle position-velocity feature map along the vertical axis of the grid matrix coordinate system and converting the labels accordingly. The random data generation method involves randomly moving the obstacle position within the corresponding lane and re-labeling the labels.

[0010] Optionally, a pre-built lane change assist model can be trained based on the expanded lane change assist dataset, including: We selected the ResNet-18 convolutional neural network model, removed the original classification layer, added a fully connected layer with an output dimension of 4 at the end of the network, followed by a softmax activation function, and constructed the initial lane-changing auxiliary model. The obstacle position-velocity feature map at the current moment is input into the convolutional layer and residual block of the initial lane change assist model for feature extraction. Finally, after passing through a fully connected layer and softmax activation, a 4-dimensional vector is output, which represents the probability of each lane change assist label predicted by the lane change assist model. The behavior with the highest probability is selected as the lane change suggestion. The output loss function is the classification cross-entropy loss, and the lane change assist model is trained.

[0011] Optionally, the formula for the classification cross-entropy loss function is as follows: ; In the formula, This represents the actual lane change auxiliary label of the i-th element. Let represent the predicted probability of the i-th element, and n represent the lane change auxiliary label category.

[0012] Secondly, the present invention also provides a vehicle lane change assist system, comprising: The acquisition unit is used to acquire raw point cloud data of the environment around the vehicle and fuse it to obtain fused point cloud data. The processing unit is used to perform road surface point cloud segmentation and obstacle detection and tracking based on the fused point cloud data, and to obtain the relative position information and relative speed information of the vehicle and each obstacle; The generation unit is used to generate an obstacle position-velocity feature map based on the relative position information and relative velocity information, and to generate corresponding lane change assistance labels and establish a lane change assistance dataset based on the obstacle position-velocity feature map. An expansion unit is used to expand the data based on the obstacle position-velocity feature map and the corresponding lane change assist labels to enhance the lane change assist dataset; The lane change assist unit is used to train a pre-built lane change assist model based on the expanded lane change assist dataset, and to use the lane change assist model in real time to predict lane change behavior during the vehicle's driving process, and output it as a lane change suggestion.

[0013] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described vehicle lane change assist method.

[0014] Fourthly, the present invention also provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described vehicle lane-changing assistance method.

[0015] The above technical solution uses multiple lidars to acquire fused point cloud data, detects and tracks road obstacles, generates feature maps and corresponding labels, and provides the driver with the best lane-changing behavior suggestions (maintain, left lane change, right lane change, and deceleration in the current lane) in an accurate and real-time manner through a deep learning model, thereby improving the safety and comfort of high-speed driving.

[0016] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a vehicle lane-changing assistance method provided in an embodiment of this application; Figure 2 This is a schematic diagram of a grid matrix provided in an embodiment of this application; Figure 3 (a) is a raster image corresponding to a raster matrix provided in an embodiment of this application; Figure 3 (b) is a location feature map provided in an embodiment of this application; Figure 3 (c) is a velocity feature map provided in an embodiment of this application; Figure 4 (a) is an obstacle position-velocity feature map provided in an embodiment of this application; Figure 4 (b) is a schematic diagram of a lane change assist label corresponding to an obstacle position-velocity feature map provided in an embodiment of this application; Figure 5 (a) is a position-velocity feature map after mirroring the original data provided in an embodiment of this application; Figure 5 (b) is a method provided in the embodiments of this application that is related to Figure 5 (a) and the corresponding label illustration; Figure 6 (a) is a position-velocity feature map generated from random data according to an embodiment of this application; Figure 6 (b) is a method provided in the embodiments of this application that is related to Figure 6 (a) A diagram of the corresponding label; Figure 7 This is a schematic diagram of the structure of a vehicle lane-changing assistance method system provided in an embodiment of this application; Figure 8This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] Various embodiments of this disclosure will be described more fully in the following detailed description. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0019] In the following, the terms “comprising” or “may include”, which may be used in various embodiments of this disclosure, indicate the presence of the disclosed functions or operations and do not limit the addition of one or more functions or operations. Furthermore, as used in various embodiments of this disclosure, the terms “comprising,” “having,” and their cognates are intended only to indicate a specific feature, number, step, operation, or combination of the foregoing and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, or combinations of the foregoing, or the possibility of adding one or more features, numbers, steps, operations, or combinations of the foregoing.

[0020] In various embodiments of this disclosure, the expression "or" or "at least one of A and / or B" includes any combination or all combinations of the words listed simultaneously. For example, the expression "A or B" or "at least one of A and / or B" may include A, may include B, or may include both A and B.

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] See Figure 1 The diagram shows a flowchart of a lane-changing assistance method for vehicles in a specific embodiment, including the following execution steps: Step 100: Collect raw point cloud data of the environment around the car and fuse it to obtain fused point cloud data.

[0023] In one specific implementation, the lidar is installed around the vehicle body (such as at the front, below the side mirrors, and at the rear) to achieve 360° full-coverage perception. The point cloud data collected by each lidar includes three-dimensional spatial coordinate information (x, y, z). The coordinate system of the main lidar (usually the primary forward-facing sensor) is defined as follows: the positive x-axis represents the vehicle's forward direction, the positive y-axis represents the left side of the forward direction, and the positive z-axis represents vertically upward. Through joint calibration, the rotation matrix of each subordinate lidar coordinate system relative to the main lidar coordinate system is obtained. Translation matrix The point cloud coordinates collected by each subordinate lidar are uniformly transformed to the main lidar coordinate system according to the following formula: ; ; ; In the formula, This indicates the rotation angle of the subordinate lidar around the z-axis. This indicates the position of the subordinate lidar in the coordinate system of the main lidar. This represents the point cloud in the main lidar coordinate system. This represents the point cloud in the lidar coordinate system.

[0024] Step 101: Based on the fused point cloud data, perform road surface point cloud segmentation and obstacle detection and tracking to obtain the relative position information and relative speed information of the vehicle and each obstacle.

[0025] Specifically, when executing step 101, the following steps can be performed: S1010: Based on the fused point cloud data, the point cloud outside the lane range is filtered according to the lane information to obtain multi-lane point cloud data.

[0026] S1011: The ground segmentation algorithm is used to separate the ground point cloud and non-ground point cloud in the multi-lane point cloud data, and the non-ground point cloud is clustered to form multiple obstacle point cloud clusters.

[0027] S1012: Employs a multi-target tracking algorithm to perform cross-frame correlation and tracking of the relative position and relative speed between the vehicle and each obstacle.

[0028] The relative position of the vehicle to each obstacle is defined as the longitudinal position, and the relative speed of the vehicle to each obstacle is defined as the longitudinal speed.

[0029] In one specific implementation, the fused point cloud data obtained in step one is used to filter out point clouds outside the lane range based on lane information, resulting in multi-lane point cloud data. A region growing method based on elevation difference is employed, selecting points with lower elevations and denser neighborhoods as seed points, and calculating their elevations. For the neighboring points of the seed point, if their elevation difference If the value is less than a threshold, it is included in the ground point cloud region and used as a new seed point to continue growing. This process is repeated until no new points can be grown; the remaining points are non-ground points, and this non-ground point cloud is retained as a potential obstacle point cloud. For the retained non-ground point cloud, clustering is performed using the K-Means algorithm: K initial cluster centers are randomly selected. The distance from each point to each cluster center is calculated, and the point is assigned to the nearest cluster. The center of each cluster is recalculated, and the above process is repeated until the cluster centers no longer change. Different point cloud clusters are formed, and each point cloud cluster corresponds to a potential obstacle (such as a vehicle, cone, etc.).

[0030] The system acquires the relative position information (longitudinal position; positive when the vehicle is behind the obstacle, negative when it is in front) between the vehicle and each obstacle. It then performs cross-frame association and tracking of detected obstacles. Using the Hungarian matching algorithm, it calculates the optimal matching scheme between the obstacle in the current frame and the tracked obstacle in the previous frame by minimizing matching metrics such as obstacle distance and point cloud count. This optimal scheme serves as the input to the Kalman filter, initializing and updating the Kalman filter state. Simultaneously, the Kalman filter is used to predict the current state based on the previous state and motion model, and the predicted value is corrected by combining the current observation value. This allows for a more accurate calculation of the relative velocity information (longitudinal velocity; negative when the vehicle's speed is faster than the obstacle's, positive when it is slower) between the vehicle and each obstacle. The calculation formula for the Kalman filter is as follows: ; ; ; ; ; In the formula, This represents the state estimate at the current moment. This represents the state estimate at a historical moment. The covariance matrix represents the estimation error. Represents the state transition matrix. The transpose of the state transition matrix. Represents the control input matrix. Indicates control input, Represents the process noise covariance matrix. Represents the observation matrix. Represents the transpose of the observation matrix. Represents the observation noise covariance matrix. Indicates Kalman gain, This represents the prediction error covariance. Indicates the predicted state. This represents the measurement value at the current time k.

[0031] Step 102: Generate an obstacle position-velocity feature map based on the relative position information and relative velocity information, and generate corresponding lane change assistance labels and establish a lane change assistance dataset based on the obstacle position-velocity feature map.

[0032] Specifically, generating an obstacle position-velocity feature map based on the relative position and relative velocity information includes: mapping the obstacle onto a two-dimensional raster map using rasterization projection to generate a position feature map and a velocity feature map; and combining the position feature map and the velocity feature map to encode a three-dimensional tensor structure obstacle position-velocity feature map; wherein the rasterization projection formula is as follows: ; ; In the formula, D represents the vertical axis of the raster matrix coordinate system. The horizontal axis represents the raster matrix coordinate system, and int indicates the rounding down operation. and Represents point cloud coordinates, This represents the minimum value along the x-axis in the point cloud data. represents the minimum value in the y-axis direction of the point cloud data, w represents the actual horizontal distance represented by each grid cell in the grid matrix, and d represents the actual vertical distance represented by each grid cell in the grid matrix.

[0033] For example, for obstacle point cloud data, rasterization projection is performed according to the rasterization projection formula, and the raster containing obstacles is marked as 1, forming a raster matrix of the point cloud, such as... Figure 2 As shown, different colors represent different lane information. The grid matrix coordinate system has its origin at the lower left corner, with the positive W-axis extending to the right and the positive D-axis extending upwards. In the formula, `int` represents the floor function, rounding the coordinate transformation result to ensure that continuous floating-point coordinates (x, y) are converted to discrete integer grid coordinates (W, D). This represents the minimum value along the x-axis in the point cloud data. represents the minimum value in the y-axis direction of the point cloud data, w represents the actual horizontal distance represented by each grid cell in the grid matrix, and d represents the actual vertical distance represented by each grid cell in the grid matrix.

[0034] In one specific implementation, based on the obtained relative position information between the vehicle and each obstacle, a grid map ( Figure 3 (a)) The corresponding grid cells are marked to generate a location feature map, such as Figure 3 As shown in (b); similarly, the relative speed information between the vehicle and each obstacle is marked in the corresponding grid of the grid map to generate a speed feature map, as shown in (b). Figure 3 As shown in (c), the position feature map and velocity feature map are combined and encoded into a three-dimensional tensor structure W×D×2 to generate an obstacle position-velocity feature map, as shown in (c). Figure 4 As shown in (a).

[0035] Step 103: Based on the obstacle position-velocity feature map and the corresponding lane change assist labels, perform data augmentation to enhance the lane change assist dataset.

[0036] Specifically, based on the obstacle position-velocity feature map and the corresponding lane change assist label, the following steps can be performed: Based on the obstacle position-velocity feature map, a lane change assist label is generated using the lane change coefficient of the vehicle relative to the obstacle and a preset safety threshold, including the following situations.

[0037] The lane-changing coefficient includes a first lane-changing coefficient relative to an obstacle in front of the vehicle, a second lane-changing coefficient relative to an obstacle in front of the vehicle's left lane, a third lane-changing coefficient relative to an obstacle behind the vehicle's left lane, a fourth lane-changing coefficient relative to an obstacle in front of the vehicle's right lane, and a fifth lane-changing coefficient relative to an obstacle behind the vehicle's right lane. The lane-changing coefficients are based on the formula... The calculation is performed, where v represents the relative speed between the vehicle and the obstacle, d represents the relative position between the vehicle and the obstacle, and m and n represent the corresponding weighting coefficients.

[0038] Scenario 1: When the first lane change coefficient is less than the preset safety threshold, the vehicle maintains the current speed in the current lane and is marked as the first lane change assist label.

[0039] Scenario 2: When the first lane change coefficient is greater than the preset safety threshold, the second lane change coefficient is less than the preset safety threshold, and the sum of the second and third lane change coefficients is less than the fourth and fifth lane change coefficients, the vehicle changes lanes to the left and is marked with the second lane change assist label.

[0040] Scenario 3: When the first lane change coefficient is greater than the preset safety threshold, the fourth lane change coefficient is less than the preset safety threshold, the fifth lane change coefficient is less than the preset safety threshold, and the sum of the fourth and fifth lane change coefficients is less than the sum of the second and third lane change coefficients, the vehicle changes lanes to the right and is marked as lane change assist third label.

[0041] Scenario 4: When the first lane change coefficient is greater than the preset safety threshold and the second lane change coefficient is greater than the preset safety threshold, or the third lane change coefficient is greater than the preset safety threshold and the fourth lane change coefficient is greater than the preset safety threshold, or the fifth lane change coefficient is greater than the preset safety threshold, the vehicle decelerates in the current lane and is marked as lane change assist fourth label.

[0042] For example, based on the lane-changing behaviors that this vehicle can perform, corresponding lane-changing assistance labels are generated for the obstacle position-velocity feature map. The lane-changing behavior label categories of this vehicle can be represented as follows: Y is divided into four categories: 0 indicates that the vehicle maintains the current speed in the current lane, 1 indicates that the vehicle changes lanes to the left, 2 indicates that the vehicle changes lanes to the right, and 3 indicates that the vehicle decelerates in the current lane.

[0043] For example, the following lane-changing coefficients are calculated according to the lane-changing coefficient calculation formula: lane-changing coefficient relative to the obstacle in front of the vehicle. The lane-changing coefficient relative to the obstacle in front of the vehicle's left lane. The lane-changing coefficient relative to the obstacle behind the vehicle in the left lane. The lane-changing coefficient relative to the obstacle in front of the vehicle in the right lane. The lane-changing coefficient relative to the obstacle behind the vehicle in the right lane. .

[0044] when When the obstacle ahead in this lane meets the safety threshold, the vehicle maintains the current speed in the current lane, and Y=0.

[0045] when When the vehicle needs to change lanes and the obstacle in the left lane meets the safety threshold better than the obstacle in the right lane, the vehicle changes lanes to the left, Y=1.

[0046] when When the vehicle needs to change lanes and the obstacle in the right lane meets the safety threshold better than the obstacle in the left lane, the vehicle changes lanes to the right, Y=2.

[0047] when When the vehicle needs to change lanes and the obstacles in both the left and right lanes do not meet the safety threshold, the vehicle decelerates in the current lane, Y=3.

[0048] Based on this, lane change assist labels can be generated, and a training dataset for the lane change assist model can be established, such as... Figure 4(a) and (b) are shown.

[0049] In some implementations, data augmentation is performed based on the obstacle position-velocity feature map and the corresponding lane change assist labels to enhance the lane change assist dataset. This includes: data augmentation based on the obstacle position-velocity feature map and the corresponding lane change assist labels through two methods: original data mirroring and random data generation. The original data mirroring method involves horizontally flipping the obstacle position-velocity feature map along the vertical axis of the grid matrix coordinate system and converting the labels accordingly. The random data generation method involves randomly moving the obstacle position within the corresponding lane and relabeling the labels.

[0050] For example, mirroring the original data refers to horizontally flipping the position-velocity feature map along the D-axis. Correspondingly, the lane-changing behavior label Y needs to be transformed: original label Y=1 (left lane change) mirrored to Y=2 (right lane change); original label Y=2 (right lane change) mirrored to Y=1 (left lane change); original labels Y=0 (maintain) and Y=3 (decelerate) remain unchanged after mirroring. The position-velocity feature map and corresponding labels after mirroring the original data are as follows: Figure 5 As shown. Random data generation refers to keeping the number of obstacles and relative speed information of the existing position-velocity feature map unchanged, randomly moving obstacles in the corresponding lanes of the obstacles, changing the relative position information between the vehicle and the obstacles, and obtaining the corresponding labels according to the lane change assist label generation method in step four. The position-velocity feature map and corresponding labels generated by random data generation are shown below. Figure 6 As shown.

[0051] Step 104: Train the pre-built lane change assistance model based on the expanded lane change assistance dataset, and use the lane change assistance model to predict lane change behavior in real time during the vehicle's driving process, and output it as a lane change suggestion.

[0052] Specifically, training a pre-built lane change assist model based on the expanded lane change assist dataset includes the following steps: S1: Select the ResNet-18 convolutional neural network model, remove the original classification layer, add a fully connected layer with an output dimension of 4 at the end of the network, followed by a softmax activation function, to build the initial lane-changing auxiliary model.

[0053] S2: Input the obstacle position-velocity feature map at the current moment into the convolutional layer and residual block of the initial lane change assist model for feature extraction. Finally, after full connection layer and softmax activation, output a 4-dimensional vector, which represents the probability of each lane change assist label predicted by the lane change assist model. Select the behavior with the highest probability as the lane change suggestion. The output loss function is classification cross-entropy loss, and the lane change assist model is trained.

[0054] Specifically, the formula for the classification cross-entropy loss function is as follows: ; In the formula, This represents the actual lane change auxiliary label of the i-th element. Let represent the predicted probability of the i-th element, and n represent the lane change auxiliary label category.

[0055] For example, a 4-dimensional vector can be represented by P = [p0, p1, p2, p3], where p0, p1, p2, and p3 represent the probabilities of the model predicting behavior Y = 0, 1, 2, and 3, respectively, and the behavior with the highest probability is selected as the lane-changing suggestion.

[0056] In this embodiment, the fusion point cloud of multiple lidar sensors can be processed efficiently, key environmental features can be extracted, feature maps and corresponding labels can be generated, and the optimal lane-changing behavior suggestions (maintain, left lane change, right lane change, and deceleration in the current lane) can be accurately and in real time provided to the driver of the vehicle through a deep learning model, thereby improving the safety and comfort of high-speed driving.

[0057] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0058] like Figure 7 As shown, the following are embodiments of the vehicle lane change assist method system provided in this disclosure. The vehicle lane change assist methods in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the vehicle lane change assist method system, please refer to the embodiments of the above vehicle lane change assist methods.

[0059] The acquisition unit is used to acquire raw point cloud data of the environment around the vehicle and fuse it to obtain fused point cloud data. The processing unit is used to perform road surface point cloud segmentation and obstacle detection and tracking based on the fused point cloud data, and to obtain the relative position information and relative speed information of the vehicle and each obstacle; The generation unit is used to generate an obstacle position-velocity feature map based on the relative position information and relative velocity information, and to generate corresponding lane change assistance labels and establish a lane change assistance dataset based on the obstacle position-velocity feature map. An expansion unit is used to expand the data based on the obstacle position-velocity feature map and the corresponding lane change assist labels to enhance the lane change assist dataset; The lane change assist unit is used to train a pre-built lane change assist model based on the expanded lane change assist dataset, and to use the lane change assist model in real time to predict lane change behavior during the vehicle's driving process, and output it as a lane change suggestion.

[0060] Figure 8 This is a schematic diagram of the hardware structure of an electronic device that implements various embodiments of the present invention.

[0061] The lane-changing assistance method for automobiles provided in this application can be applied to electronic devices. Those skilled in the art will understand that the electronic device structure involved in the embodiments of this invention does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the electronic device includes, but is not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0062] Electronic devices may include processors, external memory interfaces, internal memory, universal serial bus (USB) interfaces, charging management modules, power management modules, batteries, wireless communication modules, audio modules, speakers, microphones, sensor modules, buttons, cameras, displays, and SIM card interfaces, etc.

[0063] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0064] A processor may include one or more processing units, such as: a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0065] The processor can serve as the nerve center and command center of an electronic device. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0066] The processor may also include memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or that are used repeatedly. If the processor needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0067] An external storage interface (ESI) can be used to connect external memory cards, such as microSD cards, to expand the storage capacity of electronic devices. The external memory card communicates with the processor through the ESI to perform data storage functions, such as saving music and video files on the external memory card.

[0068] Internal memory can be used to store computer executable program code, which includes instructions. The processor executes various functional applications and data processing of electronic devices by running the instructions stored in internal memory. Internal memory can include a program storage area and a data storage area. Internal memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0069] Wireless communication functionality in electronic devices can be achieved through antennas, wireless communication modules, modem processors, and baseband processors.

[0070] Wireless communication modules can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0071] Electronic devices can implement audio functions through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.

[0072] Electronic devices can achieve shooting functions through ISPs, cameras, video codecs, GPUs, displays, and application processors.

[0073] Electronic devices can achieve display functions through GPUs, displays, and application processors.

[0074] A GPU is a microprocessor for image processing, connected to the display screen and application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. A processor may include one or more GPUs, which execute program instructions to generate or modify display information.

[0075] A display screen is used to display images, videos, etc. A display screen includes a display panel.

[0076] The storage medium provided in this application stores a program product capable of implementing a vehicle lane-changing assistance method.

[0077] The vehicle lane-changing assistance method includes: collecting raw point cloud data of the vehicle's surrounding environment and fusing it to obtain fused point cloud data; based on the fused point cloud data, performing road surface point cloud segmentation and obstacle detection and tracking to obtain the relative position and relative speed information of the vehicle and each obstacle; generating an obstacle position-velocity feature map based on the relative position and relative speed information, and generating corresponding lane-changing assistance labels and establishing a lane-changing assistance dataset based on the obstacle position-velocity feature map; augmenting the data based on the obstacle position-velocity feature map and the corresponding lane-changing assistance labels to enhance the lane-changing assistance dataset; training a pre-constructed lane-changing assistance model based on the augmented lane-changing assistance dataset, and using the lane-changing assistance model in real time during the vehicle's driving process to predict lane-changing behavior and outputting it as a lane-changing suggestion.

[0078] In some possible implementations, the subject matter of this disclosure, namely, the vehicle lane change assist method and system, can be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0079] The storage medium disclosed herein may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0080] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for assisting lane changing for vehicles, characterized in that, include: Collect raw point cloud data of the environment surrounding the vehicle and fuse it to obtain fused point cloud data; Based on the fused point cloud data, road point cloud segmentation and obstacle detection and tracking are performed to obtain the relative position and relative speed information of the vehicle and each obstacle; Based on the relative position information and relative velocity information, an obstacle position-velocity feature map is generated, and based on the obstacle position-velocity feature map, corresponding lane change assist labels are generated and a lane change assist dataset is established. Based on the obstacle position-velocity feature map and the corresponding lane change assist labels, data augmentation is performed to enhance the lane change assist dataset; The pre-built lane change assist model is trained based on the expanded lane change assist dataset, and the lane change assist model is used in real time during the vehicle's driving to predict lane change behavior and output as a lane change suggestion.

2. The vehicle lane-changing assistance method according to claim 1, characterized in that, Based on the fused point cloud data, road surface point cloud segmentation and obstacle detection and tracking are performed to obtain the relative position and relative speed information of the vehicle and each obstacle, including: Based on the fused point cloud data, the point cloud outside the lane range is filtered according to the lane information to obtain multi-lane point cloud data; A ground segmentation algorithm is used to separate ground point clouds and non-ground point clouds in multi-lane point cloud data, and the non-ground point clouds are clustered to form multiple obstacle point cloud clusters. A multi-target tracking algorithm is used to perform cross-frame association and tracking of the relative position and relative speed between the vehicle and each obstacle. The relative position between the vehicle and each obstacle is defined as the longitudinal position, and the relative speed between the vehicle and each obstacle is defined as the longitudinal speed.

3. The vehicle lane-changing assistance method according to claim 1, characterized in that, Based on the relative position information and relative velocity information, an obstacle position-velocity feature map is generated, including: The relative position and relative velocity information are mapped onto a two-dimensional raster image using rasterization projection to generate a position feature map and a velocity feature map. These two feature maps are then combined and encoded into a three-dimensional tensor structure obstacle position-velocity feature map. The rasterization projection formula is as follows: ； ； In the formula, D represents the vertical axis of the raster matrix coordinate system. The horizontal axis represents the raster matrix coordinate system, and int indicates the rounding down operation. and Represents point cloud coordinates, This represents the minimum value along the x-axis in the point cloud data. represents the minimum value in the y-axis direction of the point cloud data, w represents the actual horizontal distance represented by each grid cell in the grid matrix, and d represents the actual vertical distance represented by each grid cell in the grid matrix.

4. The vehicle lane-changing assistance method according to claim 1, characterized in that, Based on the obstacle position-velocity feature map, corresponding lane change assist labels are generated, including: Based on the obstacle position-velocity feature map, a lane change assist label is generated using the vehicle's lane change coefficient relative to the obstacle and a preset safety threshold. The lane change coefficient includes a first lane change coefficient relative to the obstacle in front of the vehicle, a second lane change coefficient relative to the obstacle in front of the vehicle's left lane, a third lane change coefficient relative to the obstacle behind the vehicle's left lane, a fourth lane change coefficient relative to the obstacle in front of the vehicle's right lane, and a fifth lane change coefficient relative to the obstacle behind the vehicle's right lane. The lane change coefficients are calculated according to the formula... The calculation is performed, where v represents the relative speed between the vehicle and the obstacle, d represents the relative position between the vehicle and the obstacle, and m and n represent the corresponding weighting coefficients. When the first lane change coefficient is less than the preset safety threshold, the vehicle maintains the current speed in the current lane and is marked with the first lane change assist label; When the first lane change coefficient is greater than the preset safety threshold, the second lane change coefficient is less than the preset safety threshold, and the sum of the second and third lane change coefficients is less than the sum of the fourth and fifth lane change coefficients, the vehicle changes lanes to the left and is marked with the second lane change assist label. When the first lane change coefficient is greater than the preset safety threshold, the fourth lane change coefficient is less than the preset safety threshold, the fifth lane change coefficient is less than the preset safety threshold, and the sum of the fourth and fifth lane change coefficients is less than the sum of the second and third lane change coefficients, the vehicle changes lanes to the right and is marked as lane change assist third label. When the first lane change coefficient is greater than the preset safety threshold and the second lane change coefficient is greater than the preset safety threshold, or the third lane change coefficient is greater than the preset safety threshold and the fourth lane change coefficient is greater than the preset safety threshold, or the fifth lane change coefficient is greater than the preset safety threshold, the vehicle decelerates in the current lane and is marked as lane change assist fourth label.

5. The vehicle lane-changing assistance method according to claim 3, characterized in that, Based on the obstacle position-velocity feature map and the corresponding lane change assist labels, data augmentation is performed to enhance the lane change assist dataset, including: Based on the obstacle position-velocity feature map and the corresponding lane change assist labels, data augmentation is performed in two ways: original data mirroring and random data generation. The original data mirroring method involves horizontally flipping the obstacle position-velocity feature map along the vertical axis of the grid matrix coordinate system and converting the labels accordingly. The random data generation method involves randomly moving the obstacle position within the corresponding lane and re-labeling the labels.

6. The vehicle lane-changing assistance method according to claim 1, characterized in that, Train a pre-built lane change assist model based on the expanded lane change assist dataset, including: We selected the ResNet-18 convolutional neural network model, removed the original classification layer, added a fully connected layer with an output dimension of 4 at the end of the network, followed by a softmax activation function, and constructed the initial lane-changing auxiliary model. The obstacle position-velocity feature map at the current moment is input into the convolutional layer and residual block of the initial lane change assist model for feature extraction. Finally, after passing through a fully connected layer and softmax activation, a 4-dimensional vector is output, which represents the probability of each lane change assist label predicted by the lane change assist model. The behavior with the highest probability is selected as the lane change suggestion. The output loss function is the classification cross-entropy loss, and the lane change assist model is trained.

7. The vehicle lane-changing assistance method according to claim 6, characterized in that, The formula for the classification cross-entropy loss function is as follows: ； In the formula, This represents the actual lane change auxiliary label of the i-th element. Let represent the predicted probability of the i-th element, and n represent the lane change auxiliary label category.

8. A vehicle lane change assist system, characterized in that, include: The acquisition unit is used to acquire raw point cloud data of the environment around the vehicle and fuse it to obtain fused point cloud data. The processing unit is used to perform road surface point cloud segmentation and obstacle detection and tracking based on the fused point cloud data, and to obtain the relative position information and relative speed information of the vehicle and each obstacle; The generation unit is used to generate an obstacle position-velocity feature map based on the relative position information and relative velocity information, and to generate corresponding lane change assistance labels and establish a lane change assistance dataset based on the obstacle position-velocity feature map. An expansion unit is used to expand the data based on the obstacle position-velocity feature map and the corresponding lane change assist labels to enhance the lane change assist dataset; The lane change assist unit is used to train a pre-built lane change assist model based on the expanded lane change assist dataset, and to use the lane change assist model in real time to predict lane change behavior during the vehicle's driving process, and output it as a lane change suggestion.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the vehicle lane change assist method as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the vehicle lane change assist method as described in any one of claims 1 to 7.

Images