Vehicle driving state determination method and system, vehicle and storage medium

Abstract

The embodiment of the invention provides a vehicle driving state determination method and system, a vehicle and a storage medium. The method comprises the steps that multi-mode information of the vehicle in the driving process is collected; the multi-modal information is identified, an identification result is obtained, and the identification result is used for representing first relative position relations between the multiple static objects and the vehicle at the current moment; according to the recognition result, a target static object is determined from the multiple static objects, and the correlation degree between the target static object and the vehicle is larger than the correlation degree between the vehicle and the static objects except the target static object in the multiple static objects; based on the first relative position relationship between the target stationary object and the vehicle at the current moment, predicting to obtain a second relative position relationship between the target stationary object and the vehicle at the future moment; and determining the driving state of the vehicle based on the second relative position relationship. The technical problem that the accuracy of identifying the driving state of the vehicle is low is solved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of information processing, in particular, relate to a driving state determination method and system of a vehicle, a vehicle and a storage medium. BACKGROUND

[0002] At present, for the recognition of the driving state of a vehicle, an image data collected by a camera is usually recognized to determine the driving state of the vehicle. However, this method has the technical problem of low accuracy in recognizing the driving state of the vehicle.

[0003] At present, there is no good solution to the above problem. SUMMARY

[0004] Embodiments of the present application provide a driving state determination method and system of a vehicle, a vehicle and a storage medium to at least solve the technical problem of low data transmission efficiency.

[0005] According to an aspect of embodiments of the present application, a driving state determination method of a vehicle is provided, which can include: collecting multi-modal information of the vehicle in a driving process, wherein the multi-modal information is used to represent a plurality of stationary objects in a lane where the vehicle is located; identifying the multi-modal information to obtain an identification result, wherein the identification result is used to represent a first relative positional relationship between each of the plurality of stationary objects and the vehicle at a current time; determining a target stationary object from the plurality of stationary objects by using the identification result, wherein an association degree between the target stationary object and the vehicle is greater than an association degree between the vehicle and stationary objects other than the target stationary object in the plurality of stationary objects; predicting a second relative positional relationship between the target stationary object and the vehicle at a future time based on the first relative positional relationship between the target stationary object and the vehicle at the current time; and determining a driving state of the vehicle based on the second relative positional relationship.

[0006] Further, the multi-modal data includes at least one image data and at least one radar data, and identifying the multi-modal information to obtain the identification result includes: identifying the image data to obtain a first identification result, and identifying the radar data to obtain a second identification result, wherein the first identification result and the second identification result are respectively used to represent a relative positional relationship between the stationary object and the vehicle in different coordinate systems.

[0007] Further, the target static object is determined from the plurality of associated static objects of the plurality of historical static objects, including: determining a weighted Euclidean distance between the associated static object and the static object; and determining the target static object from the associated static objects in the plurality of associated static objects that satisfy a distance threshold of the weighted Euclidean distance.

[0008] Further, the target static object is determined from the plurality of associated static objects of the plurality of historical static objects, including: determining a weighted Euclidean distance between the associated static object and the static object; and determining the target static object from the associated static objects in the plurality of associated static objects that satisfy a distance threshold of the weighted Euclidean distance.

[0009] Further, the second relative position relationship between the target static object and the vehicle at the future time is predicted based on the first relative position relationship between the target static object and the vehicle at the current time, including: obtaining running data of the vehicle at the current time, wherein the running data is used to represent the running state of the vehicle; and converting the first relative position relationship into the second relative position relationship by using the running data.

[0010] Further, the driving state of the vehicle is determined based on the second relative position relationship, including: constructing a system dynamic model corresponding to the vehicle by using the running data, and obtaining noise data in a process of collecting the multi-modal information; constructing a state covariance matrix matched with the second relative position relationship based on the system dynamic model and the second relative position relationship, and constructing a noise covariance matrix based on the noise data, wherein the state covariance matrix is used to represent the accuracy of the second relative position relationship, and the noise covariance matrix is used to represent the accuracy of the multi-modal information; converting the state covariance matrix by using the noise covariance matrix to obtain a Kalman gain corresponding to the second relative position relationship; correcting the second relative position relationship by using the Kalman gain to obtain a third relative position relationship; and determining the driving state based on the third relative position relationship.

[0011] Further, the driving state is determined based on the third relative position relationship, including: determining a deviation degree between the vehicle and the target static object based on the third relative position relationship; and determining the driving state of the vehicle based on the deviation degree.

[0012] Further, the driving state of the vehicle is determined based on the offset degree, including: in response to the offset degree satisfying an offset degree threshold, determining that the driving state is a normal driving state; and in response to the offset degree not satisfying the offset degree threshold, determining that the driving state is an abnormal driving state.

[0013] According to another aspect of the embodiments of the present application, a driving state determination apparatus of a vehicle is also provided, which can include: an acquisition unit configured to acquire multi-modal information of the vehicle during driving, wherein the multi-modal information is used to represent a plurality of static objects in a lane in which the vehicle is located; an identification unit configured to identify the multi-modal information to obtain an identification result, wherein the identification result is used to represent a first relative positional relationship between each of the plurality of static objects and the vehicle at a current time; a first determination unit configured to determine a target static object from the plurality of static objects by using the identification result, wherein a correlation degree between the target static object and the vehicle is greater than a correlation degree between the vehicle and each of the plurality of static objects except the target static object; a prediction unit configured to predict a second relative positional relationship between the target static object and the vehicle at a future time based on the first relative positional relationship between the target static object and the vehicle at the current time; and a second determination unit configured to determine the driving state of the vehicle based on the second relative positional relationship.

[0014] According to another aspect of the embodiments of the present application, a vehicle is also provided, which includes: a memory storing an executable program; and a processor configured to run the program, wherein the program is executed to perform the method in the embodiments of the present application.

[0015] According to another aspect of the embodiments of the present application, a computer readable storage medium is also provided, which includes a stored executable program, wherein the executable program is run to control a device in which the computer readable storage medium is located to perform the method in the embodiments of the present application.

[0016] According to another aspect of the embodiments of the present application, a computer program product is also provided, which includes a computer program, and the computer program is executed by a processor to implement the method in the embodiments of the present application.

[0017] According to another aspect of the embodiments of the present application, a computer program product is also provided, which includes a non-volatile computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement the method in the embodiments of the present application.

[0018] According to another aspect of the embodiments of the present application, a computer program is also provided, and the computer program is executed by a processor to implement the method in the embodiments of the present application.

[0019] In this embodiment, multimodal information of the vehicle during its driving process is collected. This multimodal information may include image data and / or radar data of the vehicle during its driving process. The multimodal information can be identified to obtain an identification result. Based on the identification result, a first relative positional relationship between a stationary object and the vehicle at the current moment, as well as the positional information of multiple stationary objects, can be determined. Based on the positional information, at least one target stationary object whose correlation with the vehicle meets the requirements can be identified from the multiple stationary objects. Based on the first relative positional relationship between the target stationary object and the vehicle at the current moment, a second relative positional relationship between the target stationary object and the vehicle at a future moment can be predicted. Based on the second relative positional relationship, the degree of deviation of the vehicle can be determined to determine the driving state of the vehicle. This achieves the goal of accurately determining the driving state of the vehicle using multimodal information, solves the technical problem of low accuracy in identifying the driving state of the vehicle, and realizes the technical effect of improving the accuracy of identifying the driving state of the vehicle. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a flowchart of a method for determining the driving state of a vehicle according to an embodiment of this application;

[0022] Figure 2 This is a flowchart of a lane departure detection method based on information fusion according to an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of a vehicle driving status determination device according to an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] According to an embodiment of this application, a method embodiment for determining the driving state of a vehicle is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0027] This embodiment provides a method for determining the driving state of a vehicle. Figure 1 This is a flowchart of a method for determining the driving state of a vehicle according to an embodiment of this application. Figure 1 As shown, the method may include the following steps:

[0028] Step S102: Collect multimodal information of the vehicle during its driving process, wherein the multimodal information is used to characterize multiple stationary objects in the lane where the vehicle is located.

[0029] In the technical solution provided in step S102 of this application, the aforementioned multimodal information can refer to various types of information from different sensors or multi-source data. In lane departure detection, multimodal information can include, but is not limited to, visual information from cameras (i.e., image data) and radio frequency information from millimeter-wave radar (i.e., radar data). It can be used to determine multiple stationary objects in the lane where the vehicle is located, the positions of these stationary objects, and the relative positional relationship between the stationary objects and the vehicle. The aforementioned stationary objects can be stationary objects in the lane where the vehicle is located, including but not limited to lane lines and stationary obstacles in the lane, such as stationary vehicles, pedestrian crossing signs parked on the side of the road, or disabled vehicles on the emergency stopping lane. It should be noted that this is only an illustrative example, and no specific limitations are made on the type of multimodal information or the type of stationary object.

[0030] Optionally, road images captured by a multi-functional forward-looking camera include visual features such as the color and shape of lane lines and possible traffic signs; millimeter-wave radar provides radio frequency features such as the distance, speed, and angle of vehicles, pedestrians, or other obstacles.

[0031] Optionally, during vehicle operation, various types of sensors, such as multi-functional forward-looking cameras and millimeter-wave radar, can be used to collect environmental information to obtain multimodal information. The multi-functional forward-looking camera can capture visual images of the area ahead, including features such as the shape, position, and color of lane lines, road signs, stationary vehicles, and pedestrians. Millimeter-wave radar can identify information such as the distance, relative speed, and angle of stationary or moving objects ahead.

[0032] Optionally, at this stage, the goal is to acquire information about stationary objects in the lane, including but not limited to: lane lines, which are the most direct stationary objects and whose position and direction can be detected by a camera; road signs, which may be directional, speed limit, or warning signs, whose content and location can be identified by a camera; and stationary vehicles and obstacles, whose relative distance and angle between millimeter-wave radar and the vehicles can be detected.

[0033] Optionally, the acquired multimodal information can be used in subsequent recognition and fusion processes. This information should include not only the positions of stationary objects but also their relative relationships with vehicles, providing a crucial data foundation for understanding the position and state of vehicles relative to the lane.

[0034] For example, when a vehicle is traveling on a highway, a camera may capture images of the vehicle in front and the lane markings, while millimeter-wave radar detects the exact distance to the vehicle in front. Together, this data constitutes multimodal information, which can be used in subsequent steps to identify stationary targets, predict the relative positional relationship between the vehicle and stationary objects, and determine the vehicle's driving status, thereby enabling lane departure detection and warning.

[0035] Step S104: The multimodal information is identified to obtain the identification result, wherein the identification result is used to characterize the position information of multiple stationary objects and the first relative position relationship between each stationary object and the vehicle at the current moment.

[0036] In the technical solution provided by step S104 of this application, the identification result can be used to characterize the sensor's detection and classification results of targets in the environment. In the lane departure detection system, the identification result may include, but is not limited to, the position and shape of the lane line, as well as information on other obstacles on the road. The identification result may also include the first relative positional relationship between the stationary object and the vehicle at the current moment. The first relative positional relationship can be used to characterize the relative positional information between the target (such as lane line, stationary obstacle) observed by the camera and the vehicle, such as distance, direction, etc. For example, the first relative positional relationship may be: the multi-function forward-looking camera detects that the lane line on the right side in front of the vehicle is 5 meters away from the edge of the vehicle.

[0037] Optionally, the multimodal information is identified to obtain an identification result. This identification result can be used to determine the spatial coordinate position of a stationary object, or its position in a radar coordinate system. It can also be used to determine the first relative positional relationship between the stationary object and the vehicle, such as determining the orientation and distance of the stationary object relative to the vehicle. It should be noted that this is only an illustrative example, and no specific limitations are placed on the content of the first relative positional relationship or the content of the position information.

[0038] For example, the camera's image processing algorithm identifies the solid white lane markings in front of the vehicle, while the millimeter-wave radar identifies and classifies a stationary vehicle 120 meters ahead.

[0039] Optionally, a multi-functional forward-looking camera is used to acquire image data. By performing two-dimensional (2D) detection on the image data, target elements in 2D space are obtained. These target elements can be used to determine the appearance, location, and other information of stationary objects. Alternatively, millimeter-wave radar can be used to acquire sparse two-dimensional discrete point clouds. By processing these two-dimensional discrete point clouds, 2.5D target elements are obtained (2D / 2.5D refers to the imaging space).

[0040] Optionally, in step S104, the multimodal information acquired in S102 needs to be processed to identify stationary objects in the lane. This involves applying various computer vision and signal processing algorithms to extract features from images and radar point clouds and classify targets. The identification results can include the location information of the stationary objects and their relative positional relationship with vehicles, which will be used in subsequent data association and fusion processes.

[0041] For example, suppose a vehicle is driving on a city road. A multi-function forward-facing camera and millimeter-wave radar, as the primary sensors, collect the following multi-modal information: The multi-function forward-facing camera captures the road conditions ahead, identifying a clear white solid lane marking approximately 30 meters ahead, and a stop sign approximately 50 meters to the right; the millimeter-wave radar detects the lane marking approximately 30 meters ahead and the stop sign approximately 50 meters to the right, providing more precise distance and angle information, and also detects another stationary vehicle approximately 50 meters ahead. Processing and fusing this information yields the following results: Lane line position: 30 meters ahead, offset to the right by approximately 0.5 meters (this is the combined recognition result of the camera and radar, determined through data association and fusion). Stop sign position: approximately 50 meters to the right, at an angle of 3 degrees. Stationary vehicle position: 50 meters ahead, at an angle of 0 degrees, approximately 2 meters from the center line of the vehicle's lane.

[0042] Optionally, based on the recognition results, the relative positions of the lane lines, stop signs, and stationary vehicles ahead at the current moment can be determined: the lane line is 30 meters away from the vehicle, offset to the right by 0.5 meters. The stop sign is 50 meters away from the vehicle, located to the right of the vehicle, offset at an angle of 3 degrees. The stationary vehicle ahead is 50 meters away from the vehicle, located approximately 2 meters to the right of the center line of the lane line in front of the vehicle.

[0043] Step S106: Using the recognition results, determine the target stationary object from multiple stationary objects, wherein the degree of association between the target stationary object and the vehicle is greater than the degree of association between the vehicle and other stationary objects among the multiple stationary objects excluding the target stationary object.

[0044] In the technical solution provided in step S106 of this application, the degree of association between the target stationary object and the vehicle is greater than the degree of association between the vehicle and other stationary objects among the multiple stationary objects excluding the target stationary object. This degree of association can be used to determine whether the stationary object can be used to determine the vehicle's driving state.

[0045] Optionally, based on the recognition results, a first relative positional relationship between the stationary object and the vehicle can be determined. Based on this first relative positional relationship, the position and orientation of the stationary object relative to the vehicle can be determined. Furthermore, based on the recognition results, it can be used to identify target stationary objects with a high degree of association with the vehicle; for example, the target stationary object can be the target object with the closest relative distance to the vehicle. It should be noted that this is only an illustrative example, and no specific limitations are imposed on the method for determining the target stationary object.

[0046] Alternatively, the degree of correlation can be assessed by calculating the weighted Euclidean distance between the target stationary object and the vehicle, considering parameters such as relative position and speed, or by using more complex machine learning models. The goal here is to identify the stationary objects closest to the vehicle that are most likely to pose a safety hazard and to consider them as the target stationary objects.

[0047] For example, data from cameras and radar is correlated to determine which observation most likely corresponds to which real target. In this case, because the obstacle detected by radar is closest to the vehicle and has the smallest angular deviation, it is correlated with certain targets that the camera might see. Then, the correlation between each stationary object and the vehicle is quantified using weighted Euclidean distance or similar statistical methods. The obstacle detected by radar, due to its very close proximity and near-zero angular deviation, has a significantly higher correlation with the vehicle than lane lines and stop signs further away. Based on these calculations, the object with the highest correlation to the vehicle (a stationary obstacle approximately 30 meters ahead) can be identified as the target stationary object. This means the system will then focus more resources and attention on monitoring and predicting the behavior of this obstacle and its potential impact on vehicle safety.

[0048] Step S108: Based on the first relative positional relationship between the target stationary object and the vehicle at the current moment, predict the second relative positional relationship between the target stationary object and the vehicle at a future moment.

[0049] In the technical solution provided by step S108 of this application, the first relative positional relationship can be the actual state of the stationary target object at the current moment. The second relative positional relationship can be the estimated state of the stationary target object at a future moment, also known as target state estimation.

[0050] Optionally, based on the vehicle's operating state and the current state of the obstacle and the vehicle (i.e., the first relative positional relationship), the state of the stationary target object at the next moment (i.e., the second relative positional relationship) can be predicted.

[0051] Optionally, the relative positional relationship between the vehicle and these stationary objects at a future point in time can be predicted using the identified positional information of stationary objects and the vehicle's current driving state (including speed, acceleration, direction, etc.). This prediction is mainly based on the dynamic characteristics of the vehicle and the static position of the obstacles, and is made through mathematical models (such as Kalman filters) to assess the risk of the vehicle colliding with stationary objects or deviating from its lane.

[0052] For example, suppose that at the current moment, the multi-function forward-looking camera and millimeter-wave radar have identified a stationary, disabled vehicle (a stationary target) approximately 50 meters ahead, and it is known that the relative distance between the disabled vehicle and the center line of the vehicle's lane is 2 meters, with an angle of 0 degrees. The vehicle's current driving status is: speed, 60 kilometers per hour (approximately 16.67 meters per second); acceleration, 0 meters per second² (i.e., the vehicle is traveling at a constant speed); direction, steadily traveling along the lane line.

[0053] Furthermore, a dynamic model can be constructed to describe how vehicle speed, acceleration, and other driving parameters affect the vehicle's future position. For a vehicle traveling at a constant speed, the model can be simplified to a prediction of straight-line movement based on the current speed. Assuming a prediction time window of 3 seconds, the predicted position of the vehicle after 3 seconds can be calculated, meaning the vehicle will travel approximately 50 meters forward (=3 seconds × 16.67 meters / second). Based on the vehicle's predicted future position and the expected position of the stationary target (since the stationary target's position remains unchanged, its expected position is equal to its current position), the future relative positional relationship between the vehicle and the stationary disabled vehicle can be calculated. If the vehicle continues to travel along its current lane and speed, after 3 seconds, the relative distance between the vehicle and the disabled vehicle will decrease from 50 meters to 0 meters, meaning the vehicle is expected to reach the same position as the stationary vehicle ahead.

[0054] Based on the above prediction, the second relative positional relationship between the stationary target and the vehicle at the next future moment is determined as follows: the relative distance between the vehicle and the stationary disabled vehicle ahead will be 0 meters, located approximately 2 meters to the left of the vehicle's lane centerline. This means that the vehicle is driving directly towards the disabled vehicle along its current lane, posing a collision risk. This predictive information provides crucial information for determining the vehicle's driving status in subsequent step S110, i.e., whether it is deviating from the safe driving path.

[0055] Optionally, in practical applications, this prediction can notify the driver or autonomous driving system of potential hazards in advance, prompting them to take evasive or deceleration measures to avoid collisions or lane departures. The accuracy of the prediction depends on real-time monitoring of the vehicle's driving status and accurate identification of the position of stationary objects, and is also affected by environmental factors and vehicle dynamic characteristics.

[0056] Step S110: Determine the vehicle's driving status based on the second relative position relationship.

[0057] In the technical solution provided by step S110 of this application, the driving state may include a normal driving state or an abnormal driving state. If the vehicle is in a deviated state, the driving state of the vehicle can be determined to be an abnormal driving state. If the vehicle is not in a deviated state, the driving state of the vehicle can be determined to be a normal driving state.

[0058] Optionally, a second relative positional relationship between the target stationary object (such as lane lines or obstacles ahead) and the vehicle at future moments can be obtained through multimodal information fusion. Based on the predicted second relative positional relationship, combined with vehicle speed, acceleration, and other driving parameters, it can be determined whether the vehicle's driving state is normal and safe. The assessment of the driving state includes, but is not limited to: lane keeping status (whether the vehicle remains within the current lane and what the safe distance from the lane lines is); hazard warning (whether the vehicle will collide with obstacles ahead or deviate from the lane boundary); and driving trajectory (predicting the vehicle's future path and assessing whether it conforms to lane lines or other road rules).

[0059] For example, consider a vehicle traveling at a constant speed on a two-lane highway. A multi-function forward-facing camera and millimeter-wave radar, through information fusion, predict: the lane markings' predicted position (the lane markings on both sides of the vehicle will remain unchanged for the next few seconds); and the predicted position of the stationary vehicle ahead (based on the current vehicle speed and the relative position of the stationary vehicle ahead, it is estimated that the vehicle will reach a position parallel to the stationary vehicle ahead in 2 seconds). Since the predicted lane markings indicate that the vehicle will continue to travel within its designated lane without any signs of deviation, it can be determined that the vehicle's lane-keeping is good. However, if the predicted position of the stationary vehicle ahead intersects with the vehicle's path, it means that if the vehicle continues at its current speed, a collision with the stationary vehicle ahead will occur in 2 seconds. This indicates a significant hazard and triggers the corresponding warning mechanism, reminding the driver to take braking measures or swerve.

[0060] Furthermore, based on the vehicle's current driving parameters and the predicted lane lines and obstacle positions, the vehicle's future driving trajectory is plotted. If the trajectory shows that the vehicle will gradually approach one lane line and exceed the safety limit, the risk of the vehicle deviating from the lane can also be assessed, and a lane departure warning can be issued in a timely manner to guide the driver to adjust the steering wheel and stay near the center line of the lane.

[0061] Optionally, by combining the second relative positional relationship between the vehicle and the stationary target object (i.e., the predicted future positional relationship) with the vehicle's driving parameters (i.e., operating parameters), a comprehensive assessment of the vehicle's driving status can be achieved. This is not limited to the current positional information, but more importantly, it allows for the prediction and warning of potential future hazards, thereby providing drivers with immediate safety advice and preventing possible traffic accidents.

[0062] Through the above steps S102 to S110, multimodal information of the vehicle during its driving process is collected. This multimodal information may include image data and / or radar data of the vehicle during its driving process. The multimodal information can be identified to obtain an identification result. Based on the identification result, the first relative positional relationship between the stationary object and the vehicle at the current moment, as well as the positional information of multiple stationary objects, can be determined. Based on the positional information, at least one target stationary object that meets the correlation requirements with the vehicle can be identified from the multiple stationary objects. Based on the first relative positional relationship between the target stationary object and the vehicle at the current moment, the second relative positional relationship between the target stationary object and the vehicle at a future moment can be predicted. Based on the second relative positional relationship, the degree of deviation of the vehicle can be determined to determine the driving state of the vehicle. This achieves the goal of accurately determining the driving state of the vehicle using multimodal information, solves the technical problem of low accuracy in identifying the driving state of the vehicle, and realizes the technical effect of improving the accuracy of identifying the driving state of the vehicle.

[0063] The above-mentioned method of this application will be further described below.

[0064] As an optional implementation, step S104, the multimodal data includes at least one image data and at least one radar data, and the multimodal information is identified to obtain an identification result, including: identifying the image data to obtain a first identification result, and identifying the radar data to obtain a second identification result, wherein the first identification result and the second identification result are used to characterize the relative positional relationship between the stationary object and the vehicle in different coordinate systems.

[0065] In this embodiment, the aforementioned multimodal data may include, but is not limited to, multiple image data and radar data collected from multiple angles of the vehicle. For example, multiple image data may be collected based on cameras installed at different locations on the vehicle, and multiple radar data may be collected based on radars installed at different locations on the vehicle.

[0066] Optionally, image data is identified to obtain a first identification result, and radar data is identified to obtain a second identification result. For example, a multi-functional forward-looking camera performs image detection and data segmentation. Image detection outputs shape information and position information, and data segmentation outputs pixel information, i.e., pixel value. For millimeter-wave radar, the output is position information, motion information and other information (returned radar wave intensity) in the coordinate system. The processed data can be used as input for data association and information fusion.

[0067] Optionally, in lane departure detection technology based on information fusion, the multimodal data may include at least one image data (from a multi-functional forward-looking camera) and at least one radar data (from millimeter-wave radar). The above data is processed for identification, and the resulting first and second identification results describe the relative positional relationship between stationary objects (such as lane lines, road signs, stationary vehicles, etc.) and the vehicle in different coordinate systems.

[0068] Optionally, the first recognition result originates from image data processing, specifically the image captured by the multi-functional forward-facing camera. By applying technologies such as deep learning and image processing to the image, lane lines, traffic signs, stationary obstacles, etc., are identified in the image, thereby obtaining the position information of the stationary object in the image coordinate system.

[0069] For example, suppose a camera captures an image containing the road ahead, identifying lane lines ahead and a stop sign on the right. Based on the image processing algorithm, we can determine: the lane lines are located in the lower right corner of the image coordinate system, offset from the image center by (X1, Y1); the stop sign is located in the middle right side of the image coordinate system, offset from the image center by (X2, Y2).

[0070] Optionally, the second identification result mentioned above originates from the processing of radar data, specifically the ranging information returned by millimeter-wave radar. Radar can provide information on the target's distance and angle in the physical world coordinate system, as well as the target's motion state, such as whether it is stationary. Through radar data processing, the precise position information of a stationary object in the radar coordinate system, including distance and angle, can be obtained.

[0071] Optionally, although the first and second recognition results describe the same or similar stationary objects, the information provided by the two results differs in angle and accuracy because they are based on image coordinate systems and radar coordinate systems, respectively. Image data provides rich scene details and target shape and color information, but it is greatly affected by factors such as lighting conditions and occlusion; radar data, on the other hand, is not limited by lighting conditions and provides precise distance and angle information of the target, but it is not as good as image data in distinguishing target details. Combining these two recognition results can mutually verify and complement each other, increasing the reliability and accuracy of recognition. For example, image data may not accurately give the exact distance of an object, but radar data can fill this gap. At the same time, radar data may not be able to distinguish different traffic signs, while image data can provide this identification capability.

[0072] Optionally, by fusing the first and second recognition results, the system can form a complete understanding of the stationary object in two coordinate systems, thereby more accurately judging the vehicle's driving status and potential risks, and providing solid data support for lane departure detection.

[0073] As an optional implementation, the method involves using the recognition results to determine a target stationary object from multiple stationary objects, including: acquiring multiple historical stationary objects in a lane at a historical time; using the location information in the recognition results to match historical stationary objects with stationary objects to obtain an object group that matches the historical stationary objects, wherein the object group includes at least one stationary object; determining the association probability between historical stationary objects and stationary objects, wherein the association probability is used to characterize the similarity between historical stationary objects and stationary objects; based on the association probability, determining associated stationary objects associated with historical stationary objects from the object group, wherein the association probability between associated stationary objects and historical stationary objects is greater than the association probability between other stationary objects in the object group (excluding associated stationary objects) and historical stationary objects; and determining the target stationary object from multiple associated stationary objects associated with multiple historical stationary objects.

[0074] In this embodiment, multiple historical stationary objects in the lane are acquired at historical times. Using the location information from the identification results, historical stationary objects and stationary objects can be matched to obtain an object group matching the historical stationary objects. This object group may include stationary objects whose locations are similar to those at historical stationary times. The association probability between the historical stationary objects and the stationary objects in the object group is determined. Based on this association probability, the associated stationary object most similar to the historical stationary object can be determined from the object group. Through this method, multiple associated stationary objects can be obtained from multiple historical stationary objects. From these multiple associated stationary objects, a target stationary object can be determined, and based on this target stationary object, the vehicle's driving status can be further determined.

[0075] Optionally, after obtaining the identification results, observation pairing can be performed. The observed data (i.e., the identification results) can be compared with the true state (e.g., position, velocity, direction, etc.) of known targets (i.e., historical stationary objects) to determine the stationary objects related to the historical stationary objects from among multiple stationary objects, thus obtaining an object group. For example, weighted Euclidean distance can be used to calculate the distance between each historical stationary object and the current stationary object. Then, the closest observation value is taken as the true state of the target.

[0076] Optionally, after obtaining the first and second identification results, based on the location information of the first identification result, a first stationary object associated with the vehicle can be determined from multiple still images, and based on the location information of the second identification result, a second stationary object associated with the vehicle can be determined from multiple still images. Based on the first and second stationary objects, an object group is constructed. The Probability Data Association (PDA) algorithm is used for data association, with the following general steps: Using motion equations and observation equations, the associated data—that is, radar data and image data (image data and radar data in the same object group)—are modeled in a unified coordinate system. The motion equations characterize the mathematical relationship between the state vector and time, and the state vector can be associated with the vehicle's position and velocity vectors. The observation equations characterize the relationship between the state vector and the observed values. Then, for the obtained data, the likelihood ratio combined with Bayes' theorem is used to calculate the association probability between the stationary objects in the object group and historical stationary objects.

[0077] Optionally, multiple historical still objects are acquired. Based on the recognition results, at least one still object matching the historical still objects is determined from the multiple still objects to construct an object group. The association probability between still objects in the object group and historical still objects can be determined separately. Based on the association probability, associated still objects related to the historical still objects can be determined from the object group. The target still object is determined from the multiple associated still objects related to the multiple historical still objects.

[0078] Optionally, historical data on stationary objects detected in the lane over a past period will be collected. This data includes, but is not limited to, information such as the location, size, and type of the stationary objects, as well as their relative positional relationship with vehicles. Historical data helps establish a reference frame, enabling the system to better understand and predict the current environment. Based on the current recognition results from the camera and radar, the system will attempt to match these stationary objects with stored historical stationary objects. By comparing location information, appearance features (provided only by the camera), echo intensity (provided only by the radar), etc., a set of current stationary objects that may match a specific stationary object in the historical data will be formed. This process is accomplished by calculating the weighted Euclidean distance between the observed data (i.e., the current recognition result) and the historical state.

[0079] Furthermore, the association probability measures the degree of matching between observed data and historical targets, i.e., the consistency between the two in terms of position, shape, velocity, etc. In this step, the association probability between all possible matching stationary objects and historical stationary objects can be calculated using observed data and historical target states. The higher the probability, the higher the degree of matching between the current observed data and the historical target. The stationary object with the highest association probability can be found in the matched object group and defined as the "associative stationary object." This means that for each historical stationary object, the system selects a stationary object in the current observation that is most likely to match it. This step essentially establishes the most probable mapping relationship between each historical stationary object and the currently identified stationary object.

[0080] Optionally, after matching and calculating the probability of each historical stationary object with the current stationary object, all associated stationary objects can be further analyzed to determine which one or more stationary objects pose the greatest threat or have the most direct impact on vehicle safety under the current driving condition. The selection of target stationary objects may be based on various factors such as association probability, relative position, and vehicle driving direction to ensure the accuracy and timeliness of early warning and auxiliary decision-making.

[0081] Optionally, the aforementioned state equations describe how the distance, angle, and other states between the vehicle and obstacles evolve over time, establishing a mathematical relationship between the current state and the previous state. The motion equations, on the other hand, transform the laws of physical motion into a calculable form, namely, the mathematical relationship between the vehicle's state (position, velocity) and time. Even if the observed object itself has high precision, its state (position, velocity, etc.) relative to the vehicle will change over time; the motion equations quantify this change.

[0082] Optionally, the variables representing position and velocity in the equation of motion, and the variables representing velocity and distance in the state equation, can be transformed into a unified coordinate system with the vehicle as the origin.

[0083] As an optional implementation, determining the target static object from multiple associated static objects associated with multiple historical static objects includes: determining the weighted Euclidean distance between the associated static objects and the static object; and determining the associated static objects whose weighted Euclidean distance satisfies a distance threshold as the target static object.

[0084] In this embodiment, weighted Euclidean distance is also used to calculate the distance from each observation data to the real target. However, it is necessary to satisfy the requirement of minimizing the total distance or association cost to achieve optimal allocation. That is, the associated static object with the highest association probability is determined as the target static object.

[0085] As an optional implementation, step S108, based on the first relative positional relationship between the target stationary object and the vehicle at the current moment, predicts the second relative positional relationship between the target stationary object and the vehicle at a future moment, including: acquiring the vehicle's operating data at the current moment, wherein the operating data is used to characterize the vehicle's operating state; and using the operating data to convert the first relative positional relationship into the second relative positional relationship.

[0086] In this embodiment, the vehicle's current operating data can be obtained, and the first relative position relationship can be converted into a second relative position relationship using this operating data.

[0087] Optionally, the Bayesian formula is used to update the state estimate of the stationary target object using the true state of the target (i.e., the first relative positional relationship) to obtain the second relative positional relationship. The output is the updated target state estimate and the association probability between each observed object and the target (the more similar the object, the higher the association probability).

[0088] Optionally, the aforementioned operational data may include data such as the vehicle's direction of movement and speed. Based on this operational data, the position the vehicle will move to at a future time can be determined. Based on the position the vehicle moves to and the first relative position, the second relative positional relationship between the vehicle at its moved position and the target stationary object can be determined.

[0089] In this embodiment, the Bayesian formula is used to update the target's state estimate in order to obtain the operating state of the detected object at the next moment. The application of the Bayesian formula in this process can be understood as combining prior information with new observation information to update the estimate of the target's state.

[0090] Alternatively, Bayes' theorem is a method for updating probability estimates under known conditions. In state estimation, it is used to update the probability distribution of the target state. In multi-target tracking or data association, the target state can be its position, shape, etc., while the observation data comes from sensors such as cameras or radar.

[0091] In lane departure detection for autonomous vehicles, updating the target state estimate using Bayesian formulas typically involves the following steps: First, predict the target's state at the next moment using the current system model (e.g., equations of motion). Second, establish an observation model based on the target's true state and sensor characteristics, i.e., calculate the probability of sensor observations when the target is in its current state. For stationary obstacles or lane lines, this may only involve positional information. Third, use data association algorithms (e.g., PDA algorithms) to determine which sensor observations are likely related to the target's position. Then, calculate the likelihood probability between these data and the predicted target state. Fourth, combine the prior probability with the likelihood probability to calculate the posterior probability using Bayesian formulas. This step essentially updates the probability distribution of the target state, reflecting the impact of new observation data on the state estimate. Finally, update the target state estimate based on the posterior probability distribution. Within the Kalman filter framework, this involves updating the state vector and covariance matrix to reflect a more accurate estimate and uncertainty.

[0092] As an optional implementation, determining the vehicle's driving state based on the second relative positional relationship includes: constructing a system dynamic model corresponding to the vehicle using operational data, and acquiring noise data during the multimodal information acquisition process; constructing a state covariance matrix matching the second relative positional relationship based on the system dynamic model and the second relative positional relationship, and constructing a noise covariance matrix based on the noise data, wherein the state covariance matrix is ​​used to characterize the accuracy of the second relative positional relationship, and the noise covariance matrix is ​​used to characterize the accuracy of the multimodal information; transforming the state covariance matrix using the noise covariance matrix to obtain the Kalman gain corresponding to the second relative positional relationship; correcting the second relative positional relationship using the Kalman gain to obtain a third relative positional relationship; and determining the driving state based on the third relative positional relationship.

[0093] In this embodiment, a system dynamic model corresponding to the vehicle can be constructed based on the vehicle's operating data. Simultaneously, noise data from the collected multimodal information can be acquired. Based on the system dynamic model and the second relative positional relationship, a state covariance matrix matching the second relative positional relationship is constructed, and a noise covariance matrix is ​​constructed based on the noise data. The state covariance matrix is ​​transformed using the noise covariance matrix to obtain the Kalman gain corresponding to the second relative positional relationship; the second relative positional relationship is then corrected using the Kalman gain to obtain a third relative positional relationship; and the driving state is determined based on the third relative positional relationship.

[0094] Optionally, based on the system dynamic model (vehicle, operating state) and process noise (i.e., interference data, noise data), a covariance matrix matching the predicted state estimate (i.e., the second relative positional relationship) is obtained. This covariance matrix can be used to characterize the accuracy of the identification result, the predicted value of the output state, and the predicted state covariance matrix. The Kalman gain is calculated using the predicted state covariance matrix and the measurement noise covariance matrix.

[0095] Optionally, the aforementioned noise data can be process noise, which can be used to characterize external uncertainties, such as data generated due to sudden changes in vehicle speed, road bumps, or blurred obstacles. The noise covariance matrix can be used to quantify process noise and describe the intensity of process noise and the correlation between its components. The aforementioned state covariance is the time-updated estimation error covariance matrix; it characterizes the covariance of the state estimation error at a certain moment. The aforementioned noise covariance matrix can be experimentally determined or identified based on the system's dynamic characteristics, while the state covariance matrix is ​​calculated using formulas based on the observed data.

[0096] Optionally, the noise covariance matrix can be used to characterize the statistical properties of sensor measurement noise. It can be a matrix corresponding to the state vector, where the main diagonal elements are typically the variances of the individual measurement parameters, representing the uncertainty of each parameter; the off-diagonal elements are the covariances between the measurement parameters, representing their statistical correlation. In Kalman filtering, the noise covariance matrix can be used to calculate the Kalman gain and adjust the predicted state to better fit the actual measured values.

[0097] Alternatively, the Kalman gain can be calculated based on the state covariance matrix and the noise covariance matrix, which can be used to determine the weight of measurement information in state updates.

[0098] Optionally, the primary purpose of determining the Kalman gain is to achieve optimal updates to the state estimate, i.e., to find the best balance between the given measurement information and the predicted state. For example, the Kalman gain determines the weight of new measurement information in the state update; by adjusting it appropriately, the state estimate can be made closer to the true state, reducing estimation errors.

[0099] Optionally, in multi-sensor data fusion, the Kalman gain can take into account the reliability and measurement noise of different sensors, so that the information from high-precision sensors occupies a larger proportion in state updates, achieving more robust state estimation.

[0100] Optionally, the presence of test noise inevitably affects the accuracy of state estimation, and the noise covariance matrix is ​​a key parameter used to quantify this uncertainty. By calculating the Kalman gain, the fusion method of new information can be dynamically adjusted according to the uncertainties of the state and measurement, ultimately achieving the effect of optimizing state estimation.

[0101] Optionally, the confidence level of the second relative position can be determined using Kalman gain. If the confidence level is low, the second relative position can be corrected to obtain a third relative position relationship. This third relative position relationship can be the final predicted relative position relationship between the stationary target object and the vehicle.

[0102] As an optional implementation, determining the driving state based on a third relative positional relationship includes: determining the degree of offset between the vehicle and the target stationary object based on the third relative positional relationship; and determining the driving state of the vehicle based on the degree of offset.

[0103] In this embodiment, the degree of offset between the vehicle and the stationary target object can be determined based on the third relative positional relationship, and the driving state of the vehicle can be determined based on the degree of offset.

[0104] Optionally, based on the estimated state after data fusion, it can be determined whether the vehicle has deviated from the lane.

[0105] As an optional implementation, the driving state of the vehicle is determined based on the degree of offset, including: determining the driving state as normal driving state in response to the degree of offset meeting the degree of offset threshold; and determining the driving state as abnormal driving state in response to the degree of offset not meeting the degree of offset threshold.

[0106] In this embodiment, the aforementioned offset threshold can be a pre-set threshold, which can be the maximum offset value of the vehicle's movement. If the offset meets the offset threshold, that is, the offset is less than the offset threshold, the driving state can be determined to be a normal driving state. If the offset does not meet the offset threshold, that is, the offset is greater than the offset threshold, the driving state can be determined to be an abnormal driving state, and a prompt message needs to be issued.

[0107] This embodiment proposes a lane departure detection method based on multimodal information fusion. This method combines multiple information sources for lane departure detection. It integrates lane lines identified by a camera and lane obstacles identified by radar, employing key algorithms including information comparison and fusion. When the vehicle veers and approaches the identified lane lines, a warning is issued. The information fusion technique used in this method is at the target-result level; first, the camera and radar detect lane lines and obstacles. Then, the collected multimodal information is processed, including feature extraction and classification, before being fused with confidence scores. This method only retains the fused result, not the original data. However, it has low computational resource requirements, high real-time performance, and independent decision-making by each sensor, providing a certain degree of fault tolerance. The confidence score can be determined based on Kalman gain.

[0108] Figure 2 This is a flowchart of a lane departure detection method based on information fusion according to an embodiment of this application, as shown below. Figure 2 As shown, the method may include the following steps:

[0109] Step S202, data acquisition.

[0110] In this embodiment, millimeter-wave radar and a multi-functional forward-looking camera are selected, and multi-sensor data fusion is performed using an algorithm.

[0111] Optionally, a multi-functional forward-looking camera generates raw image data, and 2D target elements are obtained by performing 2D detection on the image data; millimeter-wave radar collects multi-modal information as a sparse two-dimensional discrete point cloud, and 2.5D target elements are obtained by processing the radar data.

[0112] Step S204, data processing.

[0113] In this embodiment, a multi-functional forward-looking camera performs image detection and data segmentation. Image detection outputs shape and position information; data segmentation outputs pixel information, i.e., pixel values. The millimeter-wave radar outputs position information, motion information, and other information (returned radar wave intensity) in a coordinate system. The processed data can be used as input for data association and information fusion.

[0114] Step S206: Data association, aligning the two data streams in time and space.

[0115] In this embodiment, observation pairing is performed by comparing observed data with the known true states of targets to determine which observations might be associated with which targets. Weighted Euclidean distance is used to calculate the distance from each observation to the true target, and then the closest observation is taken as the true target state. Weighted Euclidean distance is also used to calculate the distance from each observation to the true target, but the goal is to minimize the total distance or association cost to achieve optimal allocation. The PDA algorithm is used for data association, with the following general steps: A unified coordinate system is modeled for the association system, including motion equations and observation equations. Then, for the obtained data, the likelihood ratio combined with Bayes' theorem is used to calculate the probability of association with each target; based on the calculated association probabilities, the target state estimate is updated using Bayes' theorem, and the output is the updated target state estimate and the association probability between each observation and the target.

[0116] Step S208, data fusion.

[0117] In this embodiment, based on the system dynamic model and process noise, the covariance matrix of the predicted state is calculated, along with the predicted value of the output state and the predicted state covariance matrix. The Kalman gain (confidence level: accuracy) is then calculated using the predicted state covariance matrix and the measurement noise covariance matrix.

[0118] Optionally, the state is updated based on the actual measurement data. Covariance is updated by updating the covariance matrix of the state estimate to reflect the new uncertainties, outputting the updated state prediction, state covariance matrix, and Kalman gain. The updated state and covariance are used as the prediction for the next time step, and the above process is repeated. Kalman filtering yields the optimal estimated state (the prediction for the next time step) after observation correction.

[0119] Optionally, based on the estimated state after data fusion, it can be determined whether the vehicle has deviated from the lane.

[0120] In this embodiment, multimodal information of the vehicle during its driving process is collected. This multimodal information may include image data and / or radar data of the vehicle during its driving process. The multimodal information can be identified to obtain an identification result. Based on the identification result, a first relative positional relationship between a stationary object and the vehicle at the current moment, as well as the positional information of multiple stationary objects, can be determined. Based on the positional information, at least one target stationary object whose correlation with the vehicle meets the requirements can be identified from the multiple stationary objects. Based on the first relative positional relationship between the target stationary object and the vehicle at the current moment, a second relative positional relationship between the target stationary object and the vehicle at a future moment can be predicted. Based on the second relative positional relationship, the degree of deviation of the vehicle can be determined to determine the driving state of the vehicle. This achieves the goal of accurately determining the driving state of the vehicle using multimodal information, solves the technical problem of low accuracy in identifying the driving state of the vehicle, and realizes the technical effect of improving the accuracy of identifying the driving state of the vehicle.

[0121] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0122] According to an embodiment of this application, a vehicle driving state determination device is provided. It should be noted that the device can be used to execute the above-described vehicle driving state determination method.

[0123] Figure 3This is a schematic diagram of a vehicle driving state determination device according to an embodiment of this application. Figure 3 As shown, the vehicle driving status determination device may include: a data acquisition unit 302, an identification unit 304, a first determination unit 306, a prediction unit 308, and a second determination unit 310.

[0124] The acquisition unit 302 is used to acquire multimodal information of the vehicle during its driving process, wherein the multimodal information is used to characterize multiple stationary objects in the lane where the vehicle is located.

[0125] The recognition unit 304 is used to recognize multimodal information and obtain recognition results, wherein the recognition results are used to characterize the first relative positional relationship between the multiple stationary objects and the vehicle at the current moment.

[0126] The first determining unit 306 is used to determine a target stationary object from multiple stationary objects using the recognition result, wherein the degree of association between the target stationary object and the vehicle is greater than the degree of association between the vehicle and other stationary objects among the multiple stationary objects excluding the target stationary object.

[0127] The prediction unit 308 is used to predict the second relative position relationship between the target stationary object and the vehicle at a future time based on the first relative position relationship between the target stationary object and the vehicle at the current time.

[0128] The second determining unit 310 is used to determine the driving state of the vehicle based on the second relative position relationship.

[0129] In the vehicle driving state determination device of this embodiment, a data acquisition unit collects multimodal information of the vehicle during its driving process, wherein the multimodal information is used to characterize multiple stationary objects in the lane where the vehicle is located; a recognition unit identifies the multimodal information to obtain a recognition result, wherein the recognition result characterizes the first relative positional relationship between each of the multiple stationary objects and the vehicle at the current moment; a first determination unit uses the recognition result to determine a target stationary object from the multiple stationary objects, wherein the correlation between the target stationary object and the vehicle is greater than the correlation between the vehicle and the other stationary objects among the multiple stationary objects excluding the target stationary object; a prediction unit predicts a second relative positional relationship between the target stationary object and the vehicle at a future moment based on the first relative positional relationship between the target stationary object and the vehicle at the current moment; a second determination unit determines the driving state of the vehicle based on the second relative positional relationship, thereby achieving the goal of accurately determining the driving state of the vehicle using multimodal information, solving the technical problem of low accuracy in identifying the driving state of the vehicle, and realizing the technical effect of improving the accuracy of identifying the driving state of the vehicle.

[0130] Embodiments of this application also provide a vehicle, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods described in various embodiments of this application when it runs.

[0131] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0132] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0133] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0134] Embodiments of this application also provide a computer program that, when executed by a processor, implements the methods described in the various embodiments of this application.

[0135] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0136] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0137] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0138] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0139] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0140] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A running state determination method of a vehicle, characterized by, The method comprises: collecting multi-modal information of a vehicle during driving, wherein the multi-modal information is used to represent a plurality of static objects in a lane where the vehicle is located; identifying the multi-modal information to obtain an identification result, wherein the identification result is used to represent position information of the plurality of static objects and a first relative positional relationship between each of the static objects and the vehicle at a current time; determining a target static object from the plurality of static objects based on the identification result, wherein a degree of association between the target static object and the vehicle is greater than a degree of association between the vehicle and the static objects other than the target static object among the plurality of static objects; predicting a second relative positional relationship between the target static object and the vehicle at a future time based on the first relative positional relationship between the target static object and the vehicle at the current time; determining a driving state of the vehicle based on the second relative positional relationship.

2. The method of claim 1, wherein, The multi-modal data comprises at least one image data and at least one radar data, and the identifying the multi-modal information to obtain an identification result comprises: identifying the image data to obtain a first identification result and identifying the radar data to obtain a second identification result, wherein the first identification result and the second identification result are respectively used to represent relative positional relationships between the static objects and the vehicle in different coordinate systems.

3. The method of claim 2, wherein, The determining a target static object from the plurality of static objects based on the identification result comprises: obtaining a plurality of historical static objects in the lane at a historical time; matching the historical static objects and the static objects based on the position information in the identification result to obtain an object group matched with the historical static objects, wherein the object group comprises at least one of the static objects; determining an association probability between the historical static objects and the static objects, wherein the association probability is used to represent a similarity between the historical static objects and the static objects; determining an associated static object associated with the historical static object from the object group based on the association probability, wherein an association probability between the associated static object and the historical static object is greater than an association probability between other static objects in the object group and the historical static object other than the associated static object; determining the target static object from a plurality of associated static objects associated with a plurality of historical static objects.

4. The method of claim 3, wherein, The determining the target static object from a plurality of associated static objects associated with a plurality of historical static objects comprises: determining a weighted Euclidean distance between the associated static object and the static object; determining the associated static object in the plurality of associated static objects as the target static object when the weighted Euclidean distance satisfies a distance threshold.

5. The method of claim 1, wherein, The predicting a second relative positional relationship between the target static object and the vehicle at a future time based on the first relative positional relationship between the target static object and the vehicle at the current time comprises: acquire running data of the vehicle at the current time, wherein the running data is used to represent a running state of the vehicle; convert the first relative position relationship into the second relative position relationship by using the running data.

6. The method of claim 5, wherein, determining the driving state of the vehicle based on the second relative position relationship comprises: construct a system dynamic model corresponding to the vehicle by using the running data, and acquire noise data in a process of collecting the multi-modal information; construct a state covariance matrix matched with the second relative position relationship based on the system dynamic model and the second relative position relationship, and construct a noise covariance matrix based on the noise data, wherein the state covariance matrix is used to represent accuracy of the second relative position relationship, and the noise covariance matrix is used to represent accuracy of the multi-modal information; convert the state covariance matrix to obtain a Kalman gain corresponding to the second relative position relationship by using the noise covariance matrix; correct the second relative position relationship to obtain a third relative position relationship by using the Kalman gain; determine the driving state based on the third relative position relationship.

7. The method of claim 6, wherein, determining the driving state based on the third relative position relationship comprises: determine a deviation degree between the vehicle and the target stationary object based on the third relative position relationship; determine the driving state of the vehicle based on the deviation degree.

8. The method of claim 7, wherein, determining the driving state based on the deviation degree comprises: determine that the driving state is a normal driving state in response to the deviation degree satisfying a deviation degree threshold; determine that the driving state is an abnormal driving state in response to the deviation degree not satisfying the deviation degree threshold.

9. A vehicle characterized by comprising: comprise: a memory storing an executable program; a processor configured to execute the program, wherein the program is executed to perform the method of any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored executable program, wherein the executable program is executed to control a device where the storage medium is located to perform the method of any one of claims 1 to 8.

Images