Intelligent connected vehicle environment perception method based on multi-sensor fusion

Abstract

The application relates to the unmanned driving technical field and discloses an intelligent networked vehicle environment sensing method based on multi-sensor fusion, which comprises the following steps: when it is judged that an obstacle appears in the driving direction of a target vehicle, a camera image set in a current time range is acquired; whether the obstacle is a target obstacle is judged according to the camera image set in the current time range, and a judgment result is obtained; when the judgment result is the target obstacle, the motion trajectories of the target vehicle and the target obstacle in the current time range are respectively acquired; the motion trajectories of the target vehicle and the target obstacle in the current time range are respectively input into a preset LSTM model, and the motion trajectories of the target vehicle and the target obstacle in a future time interval are obtained; whether the target vehicle and the target obstacle will collide is predicted according to the motion trajectories in the future time interval, and a prediction result is obtained; and the application is favorable for avoiding frequent occurrence of vehicle emergency obstacle avoidance events.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and more specifically, to a method for environmental perception of intelligent connected vehicles based on multi-sensor fusion. Background Technology

[0002] With the rapid development of intelligent connected and autonomous vehicle technology, environmental perception has become a core capability for achieving safe and autonomous driving. Existing technologies mostly rely on single types of sensors (such as millimeter-wave radar or ultrasonic sensors) to perceive the environment around the vehicle. However, these environmental perception technologies have significant limitations in complex scenarios. For example, they are difficult to meet practical needs in terms of perception range, accuracy, and reliability. Especially under dynamic driving conditions (such as high-speed driving or complex road conditions), autonomous vehicles cannot detect and react to obstacles that are far away or approaching rapidly in a timely manner, which limits their obstacle avoidance capabilities. This limitation not only threatens the safety of the vehicle itself but may also lead to more serious secondary accidents. Currently, existing methods rely on a single sensor, limiting the applicability of environmental perception capabilities in autonomous vehicles. Issues such as short detection range or insufficient resolution mean that vehicles can only react when an obstacle enters the sensor's perception range. However, in dynamic driving environments, obstacles can approach rapidly, leaving autonomous vehicles with extremely limited time for obstacle avoidance. Furthermore, the lack of predictive capabilities for obstacle trajectories means traditional obstacle avoidance schemes often fail to effectively assess dynamic changes in complex scenarios, leading to frequent emergency obstacle avoidance incidents. These incidents not only affect the obstacle avoidance efficiency of autonomous vehicles but may also increase the risk of other traffic accidents (such as rear-end collisions). Moreover, when the obstacle avoidance decision-making strategy of an autonomous vehicle lacks the ability to predict obstacle trajectories, it becomes difficult for the vehicle to adjust its driving strategy in advance, severely impacting its driving efficiency and safety. Summary of the Invention

[0003] To overcome the above-mentioned deficiencies of the prior art, embodiments of the present invention provide an intelligent connected vehicle environmental perception method based on multi-sensor fusion.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] A method for environmental perception of intelligent connected vehicles based on multi-sensor fusion, the method comprising:

[0006] S101: When it is determined that there is an obstacle in the direction of travel of the target vehicle based on the camera image at time TN, the camera image set within the current time range [TN,T] is obtained. The camera image set contains multiple camera images, and T and N are both integers greater than zero.

[0007] S102: Based on the camera image set within the current time range [TN,T], determine whether the obstacle is a target obstacle, and obtain the determination result. The target obstacle is a moving obstacle that conflicts with the driving direction of the target vehicle.

[0008] S103: When the judgment result is a target obstacle, the movement trajectory of the target vehicle and the target obstacle within the current time range [TN,T] is obtained respectively. The movement trajectory includes the spatial coordinates of the target vehicle or the target obstacle at different time points. The judgment result includes one of the target obstacle and non-target obstacle.

[0009] S104: Input the motion trajectories of the target vehicle and the target obstacle within the current time range [TN,T] into the preset LSTM model that performs the state transition prediction task to obtain the motion trajectories of the target vehicle and the target obstacle within the future time interval [T+N,T+2N].

[0010] S105: Based on the motion trajectory within the future time interval [T+N, T+2N], predict whether the target vehicle and the target obstacle will collide, and obtain the prediction result; the prediction result includes either a collision or no collision.

[0011] Furthermore, the camera image is specifically a first camera image or a second camera image;

[0012] Before acquiring the camera image set within the current time range [TN,T], the process includes:

[0013] The camera image acquired at time TN is input into the pre-trained YOLOv8 object detection model to obtain the object detection result. The object detection result includes Q object boxes, the bounding box coordinates and category of each object box, where Q is a natural number.

[0014] Based on the target detection results, determine whether there is at least one target box that is classified as an obstacle. If so, it is determined that there is an obstacle in the direction the target vehicle is traveling, and all corresponding target boxes that are obstacles are output. If not, it is determined that there is no obstacle in the direction the target vehicle is traveling.

[0015] Furthermore, the training method for the YOLOv8 object detection model is as follows:

[0016] Acquire historical obstacle detection training data, and divide the historical obstacle detection training data into an obstacle detection training set and an obstacle detection test set. The historical obstacle detection training data includes multiple camera images and their corresponding annotation data.

[0017] Specifically, the camera image is either a first camera image or a second camera image, and the annotation data includes the target bounding box, the bounding box coordinates, and the category, wherein the category includes obstacles and non-obstacles;

[0018] Construct an initial YOLOv8 network, using camera images from the obstacle detection training set as input and labeled data from the obstacle detection training set as output, and train the YOLOv8 network to obtain the trained YOLOv8 network.

[0019] The trained YOLOv8 network is validated using an obstacle detection test set. A trained YOLOv8 network whose output is greater than or equal to a preset accuracy threshold is used as the trained YOLOv8 object detection model.

[0020] Furthermore, when it is determined that there are no obstacles in the direction of the target vehicle's travel, let T = T + L, and return to step S101, where L is an integer greater than zero.

[0021] Furthermore, the judgment result includes one of the target obstacle and the non-target obstacle;

[0022] Before determining whether the obstacle is a target obstacle, the process includes:

[0023] Based on the output of the YOLOv8 object detection model, obtain all bounding boxes that are obstacles, and mark all bounding boxes that are obstacles as candidate bounding boxes;

[0024] In the camera image set within the current time range [TN,T], randomly extract any two consecutive camera images and obtain the bounding box coordinates of all candidate target boxes in the two consecutive camera images;

[0025] Based on the bounding box coordinates, the center coordinates of candidate target boxes that are the same obstacle in two consecutive camera images are extracted to obtain the center coordinates of the same obstacle in two consecutive different camera images;

[0026] Based on the center coordinates, calculate the lateral and longitudinal displacements of the same obstacle in two consecutive images from different cameras.

[0027] Substitute the lateral and longitudinal displacements into the Euclidean distance formula to calculate the total displacement of each candidate target box.

[0028] The total displacement is compared with a preset total displacement threshold. If the total displacement is greater than or equal to the total displacement threshold, the obstacle represented by the corresponding candidate target box is marked as a moving obstacle; if the total displacement is less than the total displacement threshold, the obstacle represented by the corresponding candidate target box is marked as a stationary obstacle.

[0029] Further, determining whether the obstacle is a target obstacle includes:

[0030] The target vehicle's direction angle is obtained, as well as the lateral and longitudinal displacements of the moving obstacle. The lateral and longitudinal displacements are then substituted into a preset radian calculation formula to obtain the radian of the moving obstacle.

[0031] The formula for calculating the radian is as follows:

[0032] ;

[0033] In the formula: Indicates radians; It is the arctangent function;

[0034] The conversion formula is used to convert radians into degrees to obtain the direction angle of movement of the moving obstacle;

[0035] The angle conversion formula is as follows:

[0036] ;

[0037] In the formula: The angle of motion of the moving obstacle. This is a function used to convert radians to degrees;

[0038] Calculate the angle between the target vehicle and each moving obstacle in the direction of movement, and obtain multiple angle values;

[0039] Each included angle value is compared with a preset included angle threshold. If the included angle value is greater than or equal to the included angle threshold, it is determined that the corresponding moving obstacle conflicts with the driving direction of the target vehicle, and the corresponding moving obstacle is marked as a target obstacle. If the included angle value is less than the included angle threshold, it is determined that the corresponding moving obstacle does not conflict with the driving direction of the target vehicle, and the corresponding moving obstacle is not marked as a target obstacle.

[0040] Furthermore, the training method for the LSTM model performing the state transition prediction task is as follows:

[0041] Acquire historical state transition training data and divide the historical state transition training data into a state transition training set and a state transition training test set. The historical state transition training data contains multiple state transition feature data and their corresponding motion trajectories in the future time interval [T+N, T+2N].

[0042] The state transition feature data refers to the trajectory of the target vehicle or target obstacle during time TN to T.

[0043] An initial LSTM neural network is constructed. The state transition feature data in the state transition training set is used as the input data of the LSTM neural network, and the motion trajectory in the future time interval [T+N, T+2N] in the state transition training set is used as the output data of the LSTM neural network. The LSTM neural network is trained to obtain the trained LSTM neural network.

[0044] Furthermore, the prediction of whether a collision will occur between the target vehicle and the target obstacle includes:

[0045] Based on the trajectory of the target vehicle in the future time interval, all spatial coordinates traversed by the target vehicle in the future time interval are combined into a first set of spatial coordinates;

[0046] Based on the trajectory of the target obstacle within a future time interval, all spatial coordinates traversed by the target obstacle within the future time interval are combined into a second set of spatial coordinates.

[0047] The intersection of the first set of spatial coordinates and the second set of spatial coordinates is calculated to obtain the spatial coordinate intersection.

[0048] Obtain the number of elements in the intersection of spatial coordinates and compare the number of elements with a preset threshold for the number of elements;

[0049] If the number of elements is less than the element number threshold, it is determined that the target vehicle and the target obstacle will not collide in the future time interval, and the prediction result of no collision is output.

[0050] If the number of elements is greater than or equal to the element number threshold, it is determined that the target vehicle and the target obstacle will collide in the future time interval, and the collision will be output as the prediction result.

[0051] An electronic device includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the above-described method for environmental perception of intelligent connected vehicles based on multi-sensor fusion.

[0052] A computer-readable storage medium storing a computer program, which, when executed, implements the above-described method for environmental perception of intelligent connected vehicles based on multi-sensor fusion.

[0053] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0054] This application discloses an environmental perception method for intelligent connected vehicles based on multi-sensor fusion, comprising: when it is determined that an obstacle appears in the direction of travel of the target vehicle, acquiring a set of camera images within the current time range; determining whether the obstacle is a target obstacle based on the set of camera images within the current time range, and obtaining a determination result; when the determination result is a target obstacle, acquiring the motion trajectories of the target vehicle and the target obstacle within the current time range respectively; inputting the motion trajectories of the target vehicle and the target obstacle within the current time range into a preset LSTM model respectively to obtain the motion trajectories of the target vehicle and the target obstacle within a future time interval; and predicting the motion trajectories of the target vehicle within the future time interval based on the motion trajectories within the future time interval. The system predicts whether a collision will occur with the target obstacle. Based on the above, this invention helps to address the shortcomings of existing environmental perception systems in terms of reaction capability, reduces the misjudgment rate of obstacles, compensates for the short reaction time of existing environmental perception systems, improves the reaction speed of autonomous vehicles when avoiding obstacles, and reserves sufficient time for autonomous vehicles to make obstacle avoidance decisions and adjustments. This helps to reduce the frequency of emergency obstacle avoidance by autonomous vehicles, thereby reducing the number of times autonomous vehicles start and stop. Furthermore, it can ensure the driving safety and efficiency of autonomous vehicles while also helping to reduce the probability of rear-end collisions and other traffic accidents. Attached Figure Description

[0055] Figure 1 A flowchart of an intelligent connected vehicle environmental perception method based on multi-sensor fusion provided by the present invention;

[0056] Figure 2 A schematic diagram of the structure of an electronic device provided by the present invention;

[0057] Figure 3 This is a schematic diagram of the structure of a computer-readable storage medium provided by the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Example 1

[0060] Please see Figure 1 As shown in the figure, this embodiment discloses a method for environmental perception of intelligent connected vehicles based on multi-sensor fusion, the method including:

[0061] S101: When it is determined that there is an obstacle in the direction of travel of the target vehicle based on the camera image at time TN, the camera image set within the current time range [TN,T] is obtained. The camera image set contains multiple camera images, and T and N are both integers greater than zero.

[0062] Specifically, the camera image is either a first camera image or a second camera image;

[0063] It should be noted that the target vehicle is an autonomous vehicle with real-time environmental perception. Visual sensors (i.e., high-definition wide-angle industrial cameras) are installed on the outside of the two A-pillars of the autonomous vehicle. The two visual sensors have the same acquisition frequency (i.e., same frequency mode, such as 30 frames / second). One is used to acquire the first camera image of the autonomous vehicle's driving direction, and the other is used to acquire the second camera image of the autonomous vehicle's driving direction.

[0064] Specifically, before acquiring the set of camera images within the current time range [TN,T], the process includes:

[0065] The camera image acquired at time TN is input into the pre-trained YOLOv8 object detection model to obtain the object detection result. The object detection result includes Q object boxes, the bounding box coordinates and category of each object box, where Q is a natural number.

[0066] The training method for the YOLOv8 object detection model is as follows:

[0067] Acquire historical obstacle detection training data, and divide the historical obstacle detection training data into an obstacle detection training set and an obstacle detection test set. The historical obstacle detection training data includes multiple camera images and their corresponding annotation data.

[0068] Specifically, the camera image refers to either a first camera image or a second camera image. The annotation data includes the target bounding box, the bounding box coordinates, and the category, where the category includes obstacles and non-obstacles. It is worth noting that in object detection, the bounding box coordinates describe the position of the obstacle in the image, and are represented using the top-left and bottom-right corner coordinates, such as (…). );

[0069] It should be noted that the camera images in the historical obstacle detection training data were collected by technicians based on experiments or actual situations, and the labeled data in the historical obstacle detection training data were obtained by technicians or AI data labelers through manual annotation.

[0070] Construct an initial YOLOv8 network, using camera images from the obstacle detection training set as input and labeled data from the obstacle detection training set as output, and train the YOLOv8 network to obtain the trained YOLOv8 network.

[0071] The trained YOLOv8 network is validated using an obstacle detection test set. The trained YOLOv8 network with an output greater than or equal to a preset accuracy threshold is used as the trained YOLOv8 object detection model.

[0072] Based on the target detection results, determine whether there is at least one target box that is classified as an obstacle. If so, it is determined that there is an obstacle in the direction the target vehicle is traveling, and all corresponding target boxes that are obstacles are output; otherwise, it is determined that there is no obstacle in the direction the target vehicle is traveling.

[0073] It should be understood that existing single-type sensors (such as millimeter-wave radar or ultrasonic sensors) are prone to mistaking small targets on the road (such as pebbles or twigs) as obstacles due to their perception defects and limitations. To avoid such incidents, this solution uses a visual sensor combined with target detection technology for avoidance. This can effectively avoid the misjudgments caused by traditional millimeter-wave radar or ultrasonic sensors, that is, mistaking objects with no collision risk (such as pebbles or twigs) as obstacles. At the same time, by analyzing the images of obstacles, it is also helpful to lay an important foundation for determining the state of obstacles (including moving and stationary states) in the future.

[0074] In some specific embodiments, when it is determined that there are no obstacles in the direction of the target vehicle's travel, T = T + L is set, and the process returns to step S101, where L is an integer greater than zero.

[0075] It is understandable that when it is determined that there are no obstacles in the direction of the target vehicle's travel, it means that the autonomous vehicle does not have a potential obstacle avoidance need, and it continues to collect the first or second camera image at the next moment to achieve continuous perception of the external environment.

[0076] S102: Based on the camera image set within the current time range [TN,T], determine whether the obstacle is a target obstacle, and obtain the determination result. The target obstacle is a moving obstacle that conflicts with the driving direction of the target vehicle.

[0077] The judgment result includes one of the target obstacle and the non-target obstacle, and the camera image set within the current time range [TN,T] contains multiple first camera images or second camera images;

[0078] Specifically, before determining whether the obstacle is a target obstacle, the process includes:

[0079] Based on the output of the YOLOv8 object detection model, obtain all bounding boxes that are obstacles, and mark all bounding boxes that are obstacles as candidate bounding boxes;

[0080] In the camera image set within the current time range [TN,T], randomly extract any two consecutive camera images and obtain the bounding box coordinates of all candidate target boxes in the two consecutive camera images;

[0081] Based on the bounding box coordinates, the center coordinates of candidate target boxes that are the same obstacle in two consecutive camera images are extracted to obtain the center coordinates of the same obstacle in two consecutive different camera images;

[0082] The formula for extracting the center coordinates is as follows:

[0083] ;

[0084] In the formula: Represents the coordinates of the center point, ( () represents the coordinates of the candidate bounding box in the top-left and bottom-right coordinate notation. It is the horizontal pixel position to the left of the candidate target bounding box. It is the vertical pixel position of the top of the candidate target box. It is the horizontal pixel position to the right of the candidate target bounding box. It is the vertical pixel position at the bottom of the candidate target box;

[0085] Based on the center coordinates, calculate the lateral and longitudinal displacements of the same obstacle in two consecutive images from different cameras.

[0086] The formula for calculating the lateral displacement is as follows: ;

[0087] The formula for calculating the longitudinal displacement is as follows: ;

[0088] In the formula: Indicates the lateral displacement. Indicates the longitudinal displacement. This represents the coordinates of the center point of a candidate bounding box in a camera image. This represents the coordinates of the center point of the candidate bounding box in another camera image;

[0089] Substitute the lateral and longitudinal displacements into the Euclidean distance formula to calculate the total displacement of each candidate target box.

[0090] The Euclidean distance formula is as follows: , This represents the total displacement of the candidate bounding box (i.e., the obstacle) between two consecutive images from different cameras;

[0091] The total displacement is compared with a preset total displacement threshold. If the total displacement is greater than or equal to the total displacement threshold, the obstacle represented by the corresponding candidate target box is marked as a moving obstacle; if the total displacement is less than the total displacement threshold, the obstacle represented by the corresponding candidate target box is marked as a stationary obstacle.

[0092] For example, suppose the camera image is specifically the first camera image, and two obstacles, Z1 and Z2, are found in the first camera image acquired at time TN. The bounding box coordinates of Z1 and Z2 in the first camera image at time TN are (2,2,6,6) and (6,6,10,10), respectively. Since the output of the YOLOv8 object detection model shows how many obstacles (i.e., the bounding boxes of the obstacles) are in the first camera image at time TN, and what their bounding box coordinates are, similarly, by inputting all first camera images within the current time range [TN,T] into the YOLOv8 object detection model, the bounding box coordinates of Z1 and Z2 in other first camera images can also be obtained. Suppose any two consecutive first camera images randomly extracted from the camera image set are X1 and X2, where X1 is the first camera image at time TN and X2 is the first camera image at time T-N+1. Within the first camera image X1, the bounding box coordinates of Z1 are (2,2,6,6), and the bounding box coordinates of Z2 are (6,6,10,10). Within the first camera image X2, the bounding box coordinates of Z1 are (4,4,8,8), and the bounding box coordinates of Z2 are (7,7,11,11). Based on the center point coordinates, the center coordinates of Z1 within the first camera image X1 are (2,6), and the center coordinates of Z1 within the first camera image X2 are (4,8). The center coordinates of Z2 within the first camera image X1 are (6,10), and the center coordinates of Z2 within the first camera image X2 are (7,11). Therefore, after displacement calculation based on the center coordinates, the lateral displacement of Z1 is 4, and the longitudinal displacement is 4, while the lateral displacement of Z2 is 1, and the longitudinal displacement is 1. Finally, substituting the lateral and longitudinal displacements of Z1 (Z2) into the Euclidean distance formula, the total displacement of Z1 is calculated as follows: The total displacement of Z2 is 1. If the total displacement threshold is set to 2, then Z1 can be determined to be a moving obstacle and Z2 to be a stationary obstacle.

[0093] It should be understood that although Z2 also has a certain displacement, this is usually due to the displacement deviation caused by the dynamic shooting of the vision sensor. Therefore, by setting a reasonable threshold for the total displacement, obstacles in motion or at rest can be effectively distinguished.

[0094] In implementation, determining whether the obstacle is a target obstacle includes:

[0095] The target vehicle's direction angle is obtained, as well as the lateral and longitudinal displacements of the moving obstacle. The lateral and longitudinal displacements are then substituted into a preset radian calculation formula to obtain the radian of the moving obstacle.

[0096] It should be noted that the target vehicle (i.e., the driverless vehicle) is also equipped with a direction sensor or GPS sensor to obtain the target vehicle's direction angle. ;

[0097] The formula for calculating the radian is as follows:

[0098] ;

[0099] In the formula: Indicates radians; It is the arctangent function, used to calculate the direction angle of a two-dimensional vector, taking into account the quadrant of the vector;

[0100] The conversion formula is used to convert radians into degrees to obtain the direction angle of movement of the moving obstacle;

[0101] The angle conversion formula is as follows:

[0102] ;

[0103] In the formula: The angle of motion of the moving obstacle. This is a function used to convert radians to degrees, therefore, , A constant unit conversion factor;

[0104] It is understandable that: if If the radian value is positive, then the radian value is directly converted to an angle. If the radian value is negative, it means that the direction angle is in the fourth quadrant (angle range is [-180 degrees, 0 degrees)). By adding 360 degrees, the angle value is adjusted to the positive range [0 degrees, 360 degrees].

[0105] Calculate the angle between the target vehicle and each moving obstacle in the direction of movement, and obtain multiple angle values;

[0106] The formula for calculating the included angle value is as follows: In the formula: The included angle value;

[0107] Each included angle value is compared with a preset included angle threshold. If the included angle value is greater than or equal to the included angle threshold, it is determined that the corresponding moving obstacle conflicts with the driving direction of the target vehicle, and the corresponding moving obstacle is marked as a target obstacle. If the included angle value is less than the included angle threshold, it is determined that the corresponding moving obstacle does not conflict with the driving direction of the target vehicle, and the corresponding moving obstacle is not marked as a target obstacle.

[0108] S103: When the judgment result is a target obstacle, the movement trajectory of the target vehicle and the target obstacle within the current time range [TN,T] is obtained respectively. The movement trajectory includes the spatial coordinates of the target vehicle or the target obstacle at different time points. The judgment result includes one of the target obstacle and non-target obstacle.

[0109] It should be noted that the trajectory of the target vehicle from time TN to T is obtained through real-time location information collected by onboard sensors (such as GPS or IMU). The position of each time period r∈[TN,T] can be represented as: Position = ( Therefore, the spatial coordinates (i.e., trajectory) of the target vehicle during the time intervals TN to T are represented as follows: The spatial coordinates of the target obstacle at different points in time (i.e., the target obstacle itself) are obtained through binocular vision or monocular vision. The specific principle is as follows:

[0110] The YOLOv8 object detection model is used to extract the bounding box coordinates of target obstacles in each camera image. Then, identical target obstacles are matched in consecutive camera images, and the obstacles are identified using IoU (Intersection over Union) or visual features (such as SIFT or ORB). The center point coordinates of the target obstacles in the consecutive camera images are calculated. For details on this part, please refer to the relevant content mentioned above; it will not be repeated here. Afterwards, since the coordinates of the target obstacles are in pixel units, they need to be converted to actual spatial coordinates using a binocular or monocular vision depth estimation algorithm: spatial coordinates = (X, Y, Z). When using binocular vision to convert the spatial coordinates of the target obstacles in the camera coordinate system to the spatial coordinates in the world coordinate system, it is first necessary to acquire image pairs captured by the vision sensor, i.e., the first camera image and the second camera image. After calculating the disparity d of the target obstacle based on the image pair, where... This represents the difference in horizontal coordinates of the same target obstacle in the left and right cameras; based on the camera's baseline length B and focal length. (It is pre-stored in the system database) and can calculate the depth H of the target obstacle, i.e. Finally, based on the depth H, the spatial coordinates of the target obstacle in the camera coordinate system at each time point are converted to spatial coordinates in the world coordinate system, thus obtaining the trajectory of the target obstacle. The conversion formula is as follows: , Z=Z, where, The spatial coordinates of the target obstacle in the camera coordinate system. The coordinates of the camera's principal point (optical center) are the offset from the origin of the pixel coordinate system.

[0111] S104: Input the motion trajectories of the target vehicle and the target obstacle within the current time range [TN,T] into the preset LSTM model that performs the state transition prediction task to obtain the motion trajectories of the target vehicle and the target obstacle within the future time interval [T+N,T+2N].

[0112] In implementation, the training method for the LSTM model performing the state transition prediction task is as follows:

[0113] Acquire historical state transition training data and divide the historical state transition training data into a state transition training set and a state transition training test set. The historical state transition training data contains multiple state transition feature data and their corresponding motion trajectories in the future time interval [T+N, T+2N].

[0114] The state transition feature data refers to the trajectory of the target vehicle or target obstacle during time TN to T.

[0115] It should be noted that the state transition feature data and the motion trajectory within the future time interval [T+N, T+2N] in the historical state transition training data are obtained by technicians in real time based on actual experimental data.

[0116] Construct an initial LSTM neural network, using the state transition feature data from the state transition training set as the input data of the LSTM neural network, and the motion trajectory within the future time interval [T+N, T+2N] from the state transition training set as the output data of the LSTM neural network, and train the LSTM neural network to obtain the trained LSTM neural network.

[0117] The trained LSTM neural network is validated using a state transition training and test set. A trained LSTM neural network whose output is less than or equal to a preset test error threshold is used as a pre-trained LSTM model to perform the state transition prediction task. S105: Based on the motion trajectory within the future time interval [T+N, T+2N], predict whether a collision will occur between the target vehicle and the target obstacle, obtaining a prediction result; the prediction result includes either a collision occurring or no collision occurring.

[0118] In implementation, the prediction of whether a collision will occur between the target vehicle and the target obstacle includes:

[0119] Based on the trajectory of the target vehicle in the future time interval, all spatial coordinates traversed by the target vehicle in the future time interval are combined into a first set of spatial coordinates;

[0120] Based on the trajectory of the target obstacle within a future time interval, all spatial coordinates traversed by the target obstacle within the future time interval are combined into a second set of spatial coordinates.

[0121] The intersection of the first set of spatial coordinates and the second set of spatial coordinates is calculated to obtain the spatial coordinate intersection.

[0122] Obtain the number of elements in the intersection of spatial coordinates and compare the number of elements with a preset threshold for the number of elements;

[0123] If the number of elements is less than the element number threshold, it is determined that the target vehicle and the target obstacle will not collide in the future time interval, and the prediction result of no collision is output.

[0124] If the number of elements is greater than or equal to the element number threshold, it is determined that the target vehicle and the target obstacle will collide in the future time interval, and the collision will be output as the prediction result.

[0125] By predicting the trajectory of obstacles and analyzing collision events between obstacles and autonomous vehicles, this invention helps to address the shortcomings of existing environmental perception systems in terms of reaction capability, reduces the misjudgment rate of obstacles, compensates for the short reaction time of existing environmental perception systems, improves the reaction speed of autonomous vehicles when avoiding obstacles, and reserves sufficient time for autonomous vehicles to make obstacle avoidance decisions and adjustments. This helps to reduce the frequency of emergency obstacle avoidance by autonomous vehicles, thereby reducing the number of times autonomous vehicles start and stop. Furthermore, it can ensure the driving safety and efficiency of autonomous vehicles while also reducing the probability of rear-end collisions and other traffic accidents.

[0126] Example 2

[0127] Please see Figure 2 As shown, this embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements any of the above-described methods for intelligent connected vehicle environmental perception based on multi-sensor fusion.

[0128] Since the electronic device described in this embodiment is the electronic device used to implement the intelligent connected vehicle environmental perception method based on multi-sensor fusion in the embodiments of this application, those skilled in the art can understand the specific implementation method and various variations of the electronic device in this embodiment based on the intelligent connected vehicle environmental perception method based on multi-sensor fusion described in the embodiments of this application. Therefore, how the electronic device implements the method in the embodiments of this application will not be described in detail here. Any electronic device used by those skilled in the art to implement the intelligent connected vehicle environmental perception method based on multi-sensor fusion in the embodiments of this application falls within the scope of protection of this application.

[0129] Example 3

[0130] Please see Figure 3 As shown, this embodiment discloses a computer-readable storage medium, including a memory, a processor, and a computer program stored on the memory and running on the processor. When the processor executes the computer program, it implements any of the above-described methods for intelligent connected vehicle environmental perception based on multi-sensor fusion.

[0131] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters, weights, and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0132] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired or wireless network. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0133] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0134] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0135] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only one method, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0136] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0137] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0138] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0139] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for environmental perception of intelligent connected vehicles based on multi-sensor fusion, characterized in that, The method includes: S101: When it is determined that there is an obstacle in the direction of travel of the target vehicle based on the camera image at time TN, the camera image set within the current time range [TN,T] is obtained. The camera image set contains multiple camera images, and T and N are both integers greater than zero. S102: Based on the camera image set within the current time range [TN,T], determine whether the obstacle is a target obstacle, and obtain the determination result. The target obstacle is a moving obstacle that conflicts with the driving direction of the target vehicle. S103: When the judgment result is a target obstacle, the movement trajectory of the target vehicle and the target obstacle within the current time range [TN,T] is obtained respectively. The movement trajectory includes the spatial coordinates of the target vehicle or the target obstacle at different time points. The judgment result includes one of the target obstacle and non-target obstacle. S104: Input the motion trajectories of the target vehicle and the target obstacle within the current time range [TN,T] into the preset LSTM model that performs the state transition prediction task to obtain the motion trajectories of the target vehicle and the target obstacle within the future time interval [T+N,T+2N]. S105: Based on the trajectory of the target vehicle within the future time interval [T+N, T+2N], predict whether a collision will occur between the target vehicle and the target obstacle, and obtain a prediction result; the prediction result includes either a collision or no collision. The camera image is specifically either a first camera image or a second camera image; Before acquiring the camera image set within the current time range [TN,T], the process includes: The camera image acquired at time TN is input into the pre-trained YOLOv8 object detection model to obtain the object detection result. The object detection result includes Q object boxes, the bounding box coordinates and category of each object box, where Q is a natural number. Based on the target detection results, determine whether there is at least one target box that is classified as an obstacle. If so, it is determined that there is an obstacle in the direction the target vehicle is traveling, and all corresponding target boxes that are obstacles are output; otherwise, it is determined that there is no obstacle in the direction the target vehicle is traveling. The training method for the YOLOv8 object detection model is as follows: Acquire historical obstacle detection training data, and divide the historical obstacle detection training data into an obstacle detection training set and an obstacle detection test set. The historical obstacle detection training data includes multiple camera images and their corresponding annotation data. Specifically, the camera image is either a first camera image or a second camera image, and the annotation data includes the target bounding box, the bounding box coordinates, and the category, wherein the category includes obstacles and non-obstacles; Construct an initial YOLOv8 network, using camera images from the obstacle detection training set as input and labeled data from the obstacle detection training set as output, and train the YOLOv8 network to obtain the trained YOLOv8 network. The trained YOLOv8 network is validated using an obstacle detection test set. The trained YOLOv8 network with an output greater than or equal to a preset accuracy threshold is used as the trained YOLOv8 object detection model. The training method for the LSTM model performing the state transition prediction task is as follows: Acquire historical state transition training data and divide the historical state transition training data into a state transition training set and a state transition training test set. The historical state transition training data contains multiple state transition feature data and their corresponding motion trajectories in the future time interval [T+N, T+2N]. The state transition feature data refers to the trajectory of the target vehicle or target obstacle during time TN to T. An initial LSTM neural network is constructed. The state transition feature data in the state transition training set is used as the input data of the LSTM neural network, and the motion trajectory in the future time interval [T+N, T+2N] in the state transition training set is used as the output data of the LSTM neural network. The LSTM neural network is trained to obtain the trained LSTM neural network.

2. The intelligent connected vehicle environmental perception method based on multi-sensor fusion according to claim 1, characterized in that, When it is determined that there are no obstacles in the direction of the target vehicle's travel, let T = T + L, and return to step S101, where L is an integer greater than zero.

3. The intelligent connected vehicle environmental perception method based on multi-sensor fusion according to claim 2, characterized in that, The judgment result includes one of the target obstacle and the non-target obstacle; Before determining whether the obstacle is a target obstacle, the process includes: Based on the output of the YOLOv8 object detection model, obtain all bounding boxes that are obstacles, and mark all bounding boxes that are obstacles as candidate bounding boxes; In the camera image set within the current time range [TN,T], randomly extract any two consecutive camera images and obtain the bounding box coordinates of all candidate target boxes in the two consecutive camera images; Based on the bounding box coordinates, the center coordinates of candidate target boxes that are the same obstacle in two consecutive camera images are extracted to obtain the center coordinates of the same obstacle in two consecutive different camera images; Based on the center coordinates, calculate the lateral and longitudinal displacements of the same obstacle in two consecutive images from different cameras. Substitute the lateral and longitudinal displacements into the Euclidean distance formula to calculate the total displacement of each candidate target box. The total displacement is compared with a preset total displacement threshold. If the total displacement is greater than or equal to the total displacement threshold, the obstacle represented by the corresponding candidate target box is marked as a moving obstacle; if the total displacement is less than the total displacement threshold, the obstacle represented by the corresponding candidate target box is marked as a stationary obstacle.

4. The intelligent connected vehicle environmental perception method based on multi-sensor fusion according to claim 3, characterized in that, The determination of whether the obstacle is a target obstacle includes: The target vehicle's direction angle is obtained, as well as the lateral and longitudinal displacements of the moving obstacle. The lateral and longitudinal displacements are then substituted into a preset radian calculation formula to obtain the radian of the moving obstacle. The formula for calculating the radian is as follows: ； In the formula: Indicates radians; It is the arctangent function; The conversion formula is used to convert radians into degrees to obtain the direction angle of movement of the moving obstacle; The angle conversion formula is as follows: ； In the formula: The angle of motion of the moving obstacle. This is a function used to convert radians to degrees; Calculate the angle between the target vehicle and each moving obstacle in the direction of movement, and obtain multiple angle values; Each included angle value is compared with a preset included angle threshold. If the included angle value is greater than or equal to the included angle threshold, it is determined that the corresponding moving obstacle conflicts with the driving direction of the target vehicle, and the corresponding moving obstacle is marked as a target obstacle. If the included angle value is less than the included angle threshold, it is determined that the corresponding moving obstacle does not conflict with the driving direction of the target vehicle, and the corresponding moving obstacle is not marked as a target obstacle.

5. The intelligent connected vehicle environmental perception method based on multi-sensor fusion according to claim 4, characterized in that, The prediction of whether a collision will occur between the target vehicle and the target obstacle includes: Based on the trajectory of the target vehicle in the future time interval, all spatial coordinates traversed by the target vehicle in the future time interval are combined into a first set of spatial coordinates; Based on the trajectory of the target obstacle within a future time interval, all spatial coordinates traversed by the target obstacle within the future time interval are combined into a second set of spatial coordinates. The intersection of the first set of spatial coordinates and the second set of spatial coordinates is calculated to obtain the spatial coordinate intersection. Obtain the number of elements in the intersection of spatial coordinates and compare the number of elements with a preset threshold for the number of elements; If the number of elements is less than the element number threshold, it is determined that the target vehicle and the target obstacle will not collide in the future time interval, and the prediction result of no collision is output. If the number of elements is greater than or equal to the element number threshold, it is determined that the target vehicle and the target obstacle will collide in the future time interval, and the collision will be output as the prediction result.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the intelligent connected vehicle environmental perception method based on multi-sensor fusion as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed, implements the intelligent connected vehicle environmental perception method based on multi-sensor fusion as described in any one of claims 1 to 5.

Images