Driving assistance method and device, vehicle and storage medium

Abstract

The embodiment of the invention discloses a driving assistance method and device, a vehicle and a storage medium, and relates to the technical field of vehicle control, and the method comprises the steps: obtaining a video stream obtained by collecting the surrounding environment of the vehicle through a camera, and obtaining the driving state information of the vehicle at the current moment; inputting each frame of image in the video stream into a pre-trained target detection model to obtain target information in each frame of image, and determining a dynamic target and a static target according to the target information; determining static environment information of each frame of image according to the category and the position of the static target in each frame of image, and determining a motion track of each dynamic target in a historical time period according to the position of the dynamic target in each frame of image; and inputting the motion trail of each dynamic target in the historical time period and the static environment information and the driving state information of each frame of image into a pre-trained driving decision prediction model to obtain a driving decision result of the vehicle at the next moment, and predicting the driving decision result at the next moment.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of vehicle control technology, and in particular, to a driving assistance method, device, vehicle and storage medium. Background Art

[0002] With the popularity of family cars, more and more novice drivers are starting to drive on the road independently. The existing driving assistance method uses vehicle-mounted radar or cameras to monitor the vehicle or obstacles ahead, and when a possible collision risk is detected, an alarm is issued in advance to remind the driver to take braking or avoidance measures in time.

[0003] However, the above scheme cannot be fully applied to the complex and changeable real road conditions, and cannot accurately predict the correct driving decision at the next moment. Therefore, for novice drivers who lack driving experience, they cannot take effective response measures when encountering emergencies, which increases the risk of traffic accidents and threatens personal safety. Summary of the invention

[0004] The embodiments of the present application provide a driving assistance method, device, vehicle and storage medium, which can predict the driving decision result at the next moment and realize the driving assistance function to solve the problem that the prior art cannot accurately predict the correct driving decision at the next moment.

[0005] In a first aspect, an embodiment of the present application provides a driving assistance method, which is applied to a vehicle, wherein a camera is installed on the vehicle, and the method comprises:

[0006] Obtain the video stream obtained by the camera capturing the surrounding environment of the vehicle, and obtain the driving status information of the vehicle at the current moment;

[0007] Input each frame of the video stream into a pre-trained target detection model to obtain target information in each frame, and determine dynamic targets and static targets based on the target information, which includes category and position;

[0008] Determine the static environment information of each frame of image according to the category and position of the static target in each frame of image, and determine the motion trajectory of each dynamic target in the historical time period according to the position of the dynamic target in each frame of image;

[0009] The motion trajectory of each dynamic target in the historical time period, the static environment information of each frame image and the driving status information are input into the pre-trained driving decision prediction model to obtain the driving decision result of the vehicle at the next moment to assist the driver in driving the vehicle.

[0010] In a second aspect, an embodiment of the present application provides a driving assistance device, which is applied to a vehicle, and a camera is installed on the vehicle, and the device includes:

[0011] The acquisition module is used to acquire the video stream obtained by the camera capturing the surrounding environment of the vehicle and to acquire the driving status information of the vehicle at the current moment;

[0012] The target detection module is used to input each frame of the video stream into a pre-trained target detection model to obtain target information in each frame of the video stream, and determine dynamic targets and static targets based on the target information, and the target information includes category and position;

[0013] A determination module, used to determine the static environment information of each frame of image according to the category and position of the static target in each frame of image, and determine the motion trajectory of each dynamic target in the historical time period according to the position of the dynamic target in each frame of image;

[0014] The decision prediction module is used to input the motion trajectory of each dynamic target in the historical time period, the static environment information of each frame image and the driving status information into a pre-trained driving decision prediction model to obtain the driving decision result of the vehicle at the next moment to assist the driver in driving the vehicle.

[0015] In a third aspect, an embodiment of the present application provides a vehicle, the vehicle comprising:

[0016] at least one processor; and a memory communicatively coupled to the at least one processor;

[0017] Among them, the memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the driving assistance method of any embodiment of the present application.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, a driving assistance method as in any embodiment of the present application is implemented.

[0019] In an embodiment of the present application, a video stream obtained by a camera capturing the surrounding environment of the vehicle can be obtained, and the driving status information of the vehicle at the current moment can be obtained; each frame image in the video stream is input into a pre-trained target detection model respectively to obtain the target information in each frame image, and dynamic targets and static targets are determined based on the target information, and the target information includes category and position; the static environment information of each frame image is determined based on the category and position of the static target in each frame image, and the motion trajectory of each dynamic target in the historical time period is determined based on the position of the dynamic target in each frame image; the motion trajectory of each dynamic target in the historical time period, the static environment information of each frame image, and the driving status information are input into a pre-trained driving decision prediction model to obtain the driving decision result of the vehicle at the next moment to assist the driver in driving the vehicle. In the above technical solution, the target information in each frame of the image can be accurately detected through a pre-trained target detection model, thereby improving the detection accuracy of the surrounding environment information, and then further through a pre-trained driving decision prediction model, the driving decision mode in different driving scenarios can be accurately captured, and thus it can be fully applicable to complex and changeable real road conditions; and, through the driving decision prediction model, the motion trajectory of dynamic targets, static environmental information, driving status information and other factors are comprehensively considered, and the driving decision result of the vehicle at the next moment can be accurately predicted, which solves the problem in the prior art that the correct driving decision at the next moment cannot be accurately predicted, realizes the driving assistance function, and thus reduces the risk of traffic accidents, and ensures the personal safety of occupants and other traffic participants on the road, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flowchart of a driving assistance method provided by an embodiment of the present application;

[0022] Figure 2 is another flowchart of the driving assistance method provided by an embodiment of the present application;

[0023] Figure 3 is a structural schematic diagram of a driving assistance device provided in an embodiment of the present application;

[0024] Figure 4 It is a structural schematic diagram of a vehicle provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0026] It should be noted that the terms "first", "second", "target", and "original" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having", and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0027] Figure 1 It is a flow chart of the driving assistance method provided by an embodiment of the present application. This embodiment can be applied to scenarios where it is necessary to predict the driving decisions of the vehicle in the future to assist the driver in driving the vehicle. A driving assistance method provided by this embodiment can be executed by a driving assistance device provided by an embodiment of the present application, and the device can be implemented by software and / or hardware. In a specific embodiment, the driving assistance device can be integrated in a vehicle, and the vehicle can include a vehicle controller, and a camera is installed on the vehicle. The execution subject of this method can be the vehicle controller of the vehicle, see Figure 1 The driving assistance method of this embodiment includes but is not limited to the following steps:

[0028] S110, obtaining a video stream obtained by a camera capturing the surrounding environment of the vehicle, and obtaining driving status information of the vehicle at the current moment.

[0029] Among them, the video stream is a dynamic data sequence composed of continuous image frames, which is used to characterize the surrounding environment of the vehicle within a historical time period, such as the real-time images of the road in front of the vehicle, the sides and rear of the vehicle, etc.; the historical time period is a period of time before the current moment, and the length of the historical time period can be determined based on actual conditions, for example, it can be a few seconds or minutes in the past.

[0030] The driving state information is various parameters of the vehicle during driving, which can reflect the motion state of the vehicle. For example, the driving state information can include information such as speed, acceleration, steering angle, and gear position.

[0031] Specifically, when it is necessary to predict the vehicle's driving decisions in the future, the video stream obtained by the camera on the vehicle collecting the vehicle's surrounding environment within a historical time period can be obtained. For example, when the vehicle is driving, the camera on the vehicle can capture the vehicle's surrounding environment in real time and send the captured image to the vehicle controller in the form of a video stream. At this time, the vehicle controller can intercept the video stream within the historical time period.

[0032] Afterwards, the vehicle's driving status information at the current moment can be obtained. For example, the vehicle's speed at the current moment can be measured by a speed sensor installed near the transmission output shaft or on the wheel; the vehicle's acceleration at the current moment can be measured by an inertial measurement unit; or the vehicle's steering angle at the current moment can be measured by a steering angle sensor installed on the steering column or steering gear.

[0033] S120, input each frame of the video stream into a pre-trained target detection model to obtain target information in each frame of the video stream, and determine dynamic targets and static targets based on the target information.

[0034] Among them, the target detection model is a pre-trained neural network model, which is used to detect the input image, identify various target objects in the image, such as vehicles, pedestrians, traffic signs, trees and other targets, and detect the target information corresponding to each target; the target detection model is pre-trained with a large amount of annotated image data, and can learn the characteristic patterns of different targets, such as the shape, color and texture of the vehicle, so that the target detection model can accurately detect the target from the new image. Exemplarily, the target detection model can be an SSD (Single Shot MultiBoxDetector) model.

[0035] The target information is the content output by the target detection model after detecting each frame of the image; the target information may include the category and position of the target; the category is the type of the target, such as vehicles, pedestrians, and traffic signs; the position is the bounding box coordinates corresponding to the target, which can accurately locate the position of the target in the image; the bounding box coordinates may be the pixel coordinates relative to the upper left corner of the image, and the bounding box coordinates at this time may include the upper left corner coordinates of the bounding box, the lower left corner coordinates of the bounding box, the upper right corner coordinates of the bounding box, the lower right corner coordinates of the bounding box, and the center coordinates of the bounding box.

[0036] Dynamic targets are target objects whose positions change rapidly over time in the vehicle's surroundings; for example, dynamic targets may include other moving vehicles, walking pedestrians, and moving animals. Static targets are target objects whose positions are relatively fixed in the vehicle's surroundings; for example, static targets may include roadside buildings, traffic signs, lane lines, crosswalks, street lights, and trees.

[0037] Specifically, after obtaining the video stream and driving status information, the video stream can be decomposed according to the frame rate to obtain each frame image, and then the format of each frame image is adjusted, such as scaling and color space conversion, so that the format of each frame image meets the input requirements of the target detection model.

[0038] Then, each frame image can be input into the pre-trained target detection model in chronological order. At this time, the target detection model performs target detection on each input frame image respectively to obtain the target information of each target in each frame image. That is, for the current frame image in each frame image, the current frame image can be input into the target detection model, and then the target detection model can perform feature extraction, target positioning and classification on the current frame image, determine the category and bounding box coordinates of each target in the current frame image, so as to obtain the target information corresponding to each target in the current frame image.

[0039] Afterwards, dynamic targets and static targets can be determined based on target information. For example, dynamic targets and static targets can be determined based on the position of the target in each frame of the image. Specifically, the position of the target in each frame of the image can be used to determine whether the target has changed position in the historical time period. If the position change of the target in the historical time period exceeds the preset distance threshold, it indicates that the target has changed position in the historical time period, and the target can be determined as a dynamic target; if the position change of the target in the historical time period does not exceed the preset distance threshold, it indicates that the target has not changed position in the historical time period, and the target can be determined as a static target. Among them, the position change is the sum of the position change values ​​of the target between adjacent frames in the historical time period; the preset distance threshold is a pre-set distance value, which is used to determine whether the target has changed position in the historical time period, and can filter out the target position changes caused by detection errors or other minor interference factors, so as to more accurately distinguish dynamic targets from static targets. By reasonably setting the preset distance threshold, the accuracy of the dynamic or static attribute judgment of the target can be improved, thereby providing a more reliable basis for subsequent driving decisions.

[0040] S130, determining static environment information of each frame of image according to the category and position of the static target in each frame of image, and determining the motion trajectory of each dynamic target in a historical time period according to the position of the dynamic target in each frame of image.

[0041] The static environment information is information about fixed objects around the vehicle that is constructed based on the category and position of static targets in each frame of the image. It depicts a relatively stable environment around the vehicle and provides background information for subsequent predicted driving decisions. For example, the static environment information may include the location of lane boundaries and the location and content of traffic signs.

[0042] The motion trajectory is the path of the dynamic target's position changing over time during a historical period of time. It reflects information such as the dynamic target's motion direction and speed change, and is of great significance for predicting the dynamic target's future behavior and determining driving decisions (such as whether to avoid or keep distance). The motion trajectory can include the dynamic target's position, speed, and acceleration at each trajectory point.

[0043] Specifically, after determining the static target and the dynamic target, any frame image can be selected as the current frame image, and the static target can be selected from the targets included in the current frame image, and then the static environment information of the current frame image can be determined based on the category and position of the static target in the current frame image using a machine learning method, so as to depict the relatively stable environment around the vehicle. For example, if the category of the static target is a traffic sign, the optical character recognition technology can be used to identify the specific content in the bounding box area of ​​the static target, such as the speed limit value and the indicated direction, etc., where the bounding box area is the area formed by the upper left corner coordinates, the lower left corner coordinates, the upper right corner coordinates and the lower right corner coordinates of the bounding box; if the category of the static target is a lane line, the edge detection algorithm can be used to detect the position of the lane line in the bounding box area of ​​the static target, so as to detect the position of the lane boundary. Afterwards, other frame images can be selected as the current frame image, and the above process can be repeated to obtain the static environment information of each frame image.

[0044] Afterwards, the motion trajectory of each dynamic target in the historical time period can be determined according to the position of the dynamic target in each frame image. For example, any dynamic target can be selected as the current dynamic target, and the position of the current dynamic target in each frame image can be obtained from the target information in each frame image. Then, these position points are connected in chronological order to obtain the motion trajectory of the current dynamic target in the historical time period, wherein the position of each trajectory point is the position of the current dynamic target in the corresponding frame image, and the time interval between adjacent frames is determined according to the frame rate of the video stream, and then the central difference method is used to determine the speed corresponding to each trajectory point based on the position change and time interval between adjacent trajectory points, and the second-order central difference method is used to determine the acceleration corresponding to each trajectory point based on the speed change and time interval between trajectory points, so as to obtain the motion trajectory of the current dynamic target in the historical time period; then, other dynamic targets are selected as the current dynamic targets, and the above process is repeated to obtain the motion trajectory of each dynamic target in the historical time period.

[0045] Exemplarily, the calculation process of the velocity of the third trajectory point is as follows: the position change between the second trajectory point and the third trajectory point (recorded as the first displacement) is calculated, and the position change between the third trajectory point and the fourth trajectory point (recorded as the second displacement) is calculated, and then the sum of the first displacement and the second displacement is calculated, and the ratio between the sum and 2 times the time interval is calculated to obtain the velocity of the third trajectory point. The calculation process of the acceleration of the third trajectory point is as follows: the difference between the velocity of the fourth trajectory point and the velocity of the second trajectory point is calculated, and the ratio between the difference and 2 times the time interval is calculated to obtain the acceleration of the third trajectory point.

[0046] S140, input the motion trajectory of each dynamic target in the historical time period, the static environment information of each frame image, and the driving state information into a pre-trained driving decision prediction model to obtain the driving decision result of the vehicle at the next moment to assist the driver in driving the vehicle.

[0047] Among them, the driving decision prediction model is a pre-trained neural network model, which is used to predict the driving decision results of the vehicle in the future. By learning a large amount of driving scene data, the driving decision prediction model can capture the relationship between the vehicle's driving state information, the motion trajectory of dynamic targets, and static environmental information and driving decision results, so as to learn the driving decision patterns in different driving scenarios.

[0048] The driving decision result is the output of the driving decision prediction model, which is used to characterize the driving action that the vehicle should take at the next moment, and can assist the driver in driving the vehicle and provide the driver with timely and reasonable driving suggestions. For example, the driving decision result may include vehicle speed control instructions (such as accelerating, decelerating or maintaining the current speed), steering operation instructions (such as turning left or right at a certain angle, or keeping straight), and following distance adjustment instructions (such as moving closer to or farther away from the target vehicle in front).

[0049] Specifically, after obtaining the static environment information of each frame image and the motion trajectory of each dynamic target in the historical time period, the motion trajectory of each dynamic target in the historical time period, the static environment information of each frame image, and the driving status information can be input into a pre-trained driving decision prediction model. At this time, the driving decision prediction model can extract features from the static environment information of each frame image to obtain a feature map corresponding to each frame image, and then predict the driving decision result of the vehicle at the next moment based on the feature map of each frame image, the motion trajectory of each dynamic target in the historical time period, and the driving status information. Specifically, the driving decision prediction model can predict the surrounding environment information at the next moment, such as the motion trend of the dynamic target at the next moment, based on the feature map corresponding to each frame image and the motion trajectory of each dynamic target in the historical time period, and use the learned driving decision modes under different driving scenarios to process the driving status information and the surrounding environment information at the next moment to predict the driving decision result of the vehicle at the next moment.

[0050] Optionally, after determining the driving decision result, the driving decision result can be displayed, for example, on the vehicle's central control screen, or by voice broadcasting, to assist the driver in driving the vehicle, especially to assist novice drivers in driving the vehicle independently. For example, when the vehicle is merging, the novice driver can be reminded to observe the rearview mirror and turn on the turn signal in advance; when the vehicle is driving on a narrow road, the novice driver can be reminded to pay attention to the distance from roadside obstacles; when encountering a roundabout, the rules and precautions for entering and exiting the roundabout can be explained in detail.

[0051] The technical solution of the embodiment of the present application can accurately detect the target information in each frame of the image through a pre-trained target detection model, thereby improving the detection accuracy of the surrounding environment information, and then further through a pre-trained driving decision prediction model, it can accurately capture the driving decision mode in different driving scenarios, and thus can be fully applicable to complex and changeable real road conditions; and, through the driving decision prediction model, by comprehensively considering multiple factors such as the motion trajectory of dynamic targets, static environmental information, and driving status information, the driving decision result of the vehicle at the next moment can be accurately predicted, which solves the problem in the prior art that the correct driving decision at the next moment cannot be accurately predicted, realizes the driving assistance function, thereby reducing the risk of traffic accidents, and ensuring the personal safety of occupants and other traffic participants on the road, thereby improving the user experience.

[0052] The following further describes a driving assistance method provided by an embodiment of the present application. Figure 2 is another flowchart of the driving assistance method provided by the embodiment of the present application. The embodiment of the present application is optimized based on the above embodiments. Figure 2 The method of this embodiment includes but is not limited to the following steps:

[0053] S201, obtaining a video stream obtained by a camera capturing the surrounding environment of the vehicle, and obtaining the driving state information of the vehicle at the current moment.

[0054] S202, input each frame of the video stream into a pre-trained target detection model to obtain target information in each frame of the video stream, and determine dynamic targets and static targets based on the target information.

[0055] Specifically, after obtaining the target information, dynamic targets and static targets can be determined according to the target information, that is, when the category is in the preset static category table, the target corresponding to the category is determined to be a static target; when the category is in the preset dynamic category table, the target corresponding to the category is determined to be a dynamic target. Among them, the preset static category table is a pre-set list for storing categories identified as static targets in driving scenarios; the preset dynamic category table is a pre-set list for storing categories identified as dynamic targets in driving scenarios, and the targets corresponding to these categories usually change position in the scene. By using the preset static category table and the preset dynamic category table, targets can be quickly classified without the need to track the target to determine whether the target has changed position, thereby reducing the implementation complexity and improving the classification efficiency. It can also reduce the misclassification caused by position detection errors or short-term visual interference, thereby improving the accuracy of classification and providing an accurate data basis for subsequent understanding of driving scenarios.

[0056] Optionally, the training process of the target detection model is as follows: obtain multiple sample images, and mark the category and position of each target in each sample image, and obtain the target information label corresponding to each sample image; input the sample images into the initial target detection model, use the target information labels corresponding to the sample images to guide the training output of the initial target detection model, train the initial target detection model, and obtain the target detection model.

[0057] Among them, the target information label includes category and location; the initial target detection model is an untrained model framework used to learn various features of different targets to realize the target detection function.

[0058] Specifically, a large amount of image data including various road traffic conditions and traffic signs can be collected to obtain multiple sample images, and these sample images can cover different weather conditions (such as sunny days, rainy days and cloudy days), different time scenes (such as daytime and nighttime), and different road types (such as urban roads, highways, and rural roads); then the multiple sample images are annotated, and the category and position of each target in each sample image are annotated to obtain the target information label corresponding to each sample image; then the sample image is input into the initial target detection model to obtain the training output, and the loss value between the training output and the target information label corresponding to the sample image is calculated, and then the parameters of the initial target detection model are updated using the back propagation algorithm to minimize the loss value, and the optimal initial target detection model, that is, the target detection model, is obtained. Through the training of a large number of annotated sample images, the target detection model can learn various features of different targets and can be exposed to different driving scenes, thereby improving the detection accuracy of the target detection model.

[0059] S203: Determine the static environment information of each frame of image according to the category and position of the static target in each frame of image.

[0060] Specifically, S204 or S206 may be executed afterwards.

[0061] S204 , marking each frame of image based on the position of the dynamic target in each frame of image to obtain a target frame image sequence.

[0062] Among them, the target frame image sequence is a series of frame images obtained after labeling each frame image based on the position of the dynamic target in each frame image; in the target frame image sequence, each frame image clearly marks the position of the dynamic target in the frame image.

[0063] Specifically, after obtaining the static environment information of each frame of image, a labeling tool can be used to label the position of the dynamic target in each frame of image based on the position of the dynamic target in the frame of image, so as to obtain a series of image frames. For example, a bounding box method is used to label the frame of image based on the bounding box coordinates of the dynamic target in the frame of image; and then these frame images are arranged in chronological order to form a target frame image sequence.

[0064] S205 , input the target frame image sequence into a pre-trained trajectory detection model to obtain the motion trajectory of each dynamic target in a historical time period.

[0065] Among them, the trajectory detection model is a pre-trained neural network model, which is used to extract the motion trajectory of the target from a series of image sequences with target position annotations, especially when the vehicle moves quickly or the target is partially blocked, the trajectory detection model can infer the position and state of the target by analyzing the previous frame image information, thereby reducing the occurrence of tracking loss; exemplarily, the trajectory detection model is a three-dimensional convolutional neural network (3D Convolutional Neural Networks, 3D-CNN).

[0066] Specifically, after obtaining the target frame image sequence, a pre-trained trajectory detection model can be obtained, and the target frame image sequence can be input into the pre-trained trajectory detection model. At this time, the trajectory detection model can extract spatiotemporal features of the target frame image sequence through multiple convolutional layers, pooling layers, and fully connected layers, and based on the extracted spatiotemporal features, capture the position and state of each dynamic target in each frame image, so as to continuously track each dynamic target, and then form the motion trajectory corresponding to the dynamic target in each frame image according to the time sequence, so as to obtain the motion trajectory of each dynamic target in the historical time period. After that, S208 can be executed.

[0067] S206: Determine the initial position of each dynamic target according to the position of the dynamic target in each frame of image.

[0068] The initial position is the position of the dynamic target in the corresponding frame image when the dynamic target first appears in the video stream within the historical time period, and is the starting point for subsequent tracking of the motion trajectory of the dynamic target.

[0069] Specifically, after obtaining the static environment information of each frame image, the position of the dynamic target in the corresponding frame image when the dynamic target first appears in the video stream can be determined as the initial position of the dynamic target. For example, since the historical time period is a relatively short time range, the frame image when the dynamic target first appears in the video stream can be considered to be the first frame image, and thus the position of the dynamic target in the first frame image can be determined as the initial position of the dynamic target; however, it is not ruled out that the frame image when the dynamic target first appears in the video stream is other frame images. In this case, the frame image when the dynamic target first appears in the video stream can be determined, and the position of the dynamic target in the frame image can be determined as the initial position of the dynamic target.

[0070] S207: Continuously track the dynamic targets based on the initial position of each dynamic target to obtain the motion trajectory of each dynamic target in the historical time period.

[0071] Specifically, after obtaining the initial position, the target tracking technology can be used to continuously track the dynamic target based on the initial position of each dynamic target, and the position and state of each dynamic target in each frame image can be captured, so as to obtain the motion trajectory of each dynamic target in the historical time period, which can improve the efficiency of target tracking; illustratively, the target tracking technology can be a Kalman filter or an optical flow algorithm, and the specific tracking process can refer to the standard tracking process of the Kalman filter or the standard tracking process of the optical flow algorithm, which will not be repeated here. Afterwards, S208 can be executed.

[0072] It should be noted that S204 and S205, and S206 and S207 are parallel schemes, and any one of the schemes can be selected for execution according to actual conditions, and this embodiment does not make any specific limitation to this.

[0073] S208, input the motion trajectory of each dynamic target in the historical time period, the static environment information of each frame image, and the driving state information into a pre-trained driving decision prediction model to obtain the driving decision result of the vehicle at the next moment to assist the driver in driving the vehicle.

[0074] Optionally, the driving decision prediction model may include a feature extraction module and a driving decision module; the feature extraction module is used for feature extraction, and exemplarily, the feature extraction module may be R-CNN (Region-based Convolutional Neural Networks); the driving decision module is used to learn the dependency and change law of the input driving environment in the time series, predict the surrounding environment information at the next moment, and predict the driving decision result at the next moment in combination with the driving state information of the vehicle, and exemplarily, the driving decision module may be a long short-term memory network (Long Short-Term Memory, LSTM); the training process of the driving decision prediction model is as follows, including Sa1-Sa4:

[0075] Sa1. Obtain a sample data set.

[0076] Among them, the sample data set includes sample driving state information, sample static environment information, sample dynamic target motion trajectory and driving decision result label; the sample static environment information is a sequence, including sample static environment information corresponding to multiple frames of images; the driving decision result label is the correct driving decision result corresponding to the sample driving state information, the sample static environment information, and the sample dynamic target motion trajectory, that is, the correct driving decision of the vehicle at the next moment corresponding to the sample driving state information.

[0077] Sa2. Input the sample static environment information into the feature extraction module to obtain the target feature map sequence.

[0078] Among them, the target feature map sequence is the output result obtained after the sample static environment information is input into the feature extraction module.

[0079] Specifically, after obtaining the sample data set, the sample static environment information can be input into the feature extraction module. At this time, the feature extraction module extracts features from the sample static environment information of each frame image in the sample static environment information respectively, obtains the target feature map corresponding to each frame image, and then arranges the target feature map in chronological order to form a target feature map sequence.

[0080] Sa3. Input the sample driving state information, sample dynamic target motion trajectory and target feature map sequence into the driving decision module to obtain the driving decision prediction result.

[0081] Among them, the driving decision prediction result is the prediction result of the driving decision module.

[0082] Specifically, after the target feature map sequence, the sample driving state information, the sample dynamic target motion trajectory and the target feature map sequence can be input into the driving decision module. At this time, the driving decision module can extract the sample dynamic target motion trajectory and the target feature map sequence, the dependency and change law in the time series, and predict the surrounding environment information at the next moment corresponding to the sample driving state information. Then, based on the surrounding environment information and the sample driving state information, the driving decision of the vehicle at the next moment corresponding to the sample driving state information is predicted, that is, the driving decision prediction result.

[0083] Sa4. Calculate the loss value between the driving decision prediction result and the driving decision result label, and use the back propagation algorithm to optimize the feature extraction module and the driving decision module to minimize the loss value and obtain the driving decision prediction model.

[0084] Specifically, after obtaining the driving decision prediction result, a supervised learning method can be used to compare the driving decision prediction result with the driving decision result label, and calculate the loss value corresponding to the loss function. At this time, the loss function can comprehensively consider errors in multiple aspects, such as driving behavior classification errors (such as incorrect judgment of whether braking is needed) and speed control errors (such as the deviation between the actual speed and the ideal speed), etc.; then a back propagation algorithm is used to update and adjust the parameters of the feature extraction module and the driving decision module according to the gradient information of the loss function, and the weights and biases of the feature extraction module and the driving decision module are continuously optimized to minimize the loss value, so that the feature extraction module and the driving decision module can gradually learn the mapping relationship from the input data to the driving decision result label, such as the speed adjustment mode and turning timing of the vehicle in different traffic environments and target motion conditions, so as to obtain a driving decision prediction model.

[0085] In the embodiment of the present application, by integrating sample driving state information, sample static environment information and sample dynamic target motion trajectory, the driving decision prediction model can comprehensively consider various factors in the driving scene; and the sample data set contains a variety of different driving scene information, covering a variety of road conditions, weather conditions and traffic conditions, so that the feature extraction module can effectively extract key features from complex static environment information, and the driving decision module can better learn the driving decision patterns in different driving scenarios, thereby improving the prediction accuracy of the driving decision prediction model.

[0086] S209: Obtain the real-time driving status information of the vehicle after the current moment.

[0087] Among them, the real-time driving status information is the data about the vehicle's driving status collected in real time by various vehicle-mounted sensors (such as speed sensors, acceleration sensors, gyroscopes and vehicle-mounted positioning systems, etc.) after the current moment, which can accurately reflect the vehicle's driving status.

[0088] Specifically, after obtaining the driving decision result, the vehicle's driving status information can be obtained in real time starting from the current moment, and the real-time driving status information of the vehicle after the current moment and before reaching the next moment can be obtained, and there are multiple real-time driving status information at this time.

[0089] S210: Determine the target state of the vehicle at the next moment according to the real-time driving state information.

[0090] The target state is the state that the vehicle will reach at the next moment, obtained through a certain prediction or calculation method based on the real-time driving state information of the vehicle.

[0091] Specifically, after obtaining the real-time driving status information, key parameters such as the vehicle's speed, position, and steering angle at the next moment can be predicted based on the real-time driving status information, so as to obtain the vehicle's target state at the next moment. For example, the speed that the vehicle can reach at the next moment can be predicted based on the speed and acceleration in the real-time driving status information to obtain the predicted speed, and the position that the vehicle can reach at the next moment can be predicted to obtain the predicted position; the steering angle that the vehicle can reach at the next moment can be predicted based on the steering angle and steering speed in the real-time driving status information to obtain the predicted steering angle.

[0092] S211. Determine whether the target state is the same as the expected state corresponding to the driving decision result.

[0093] The expected state is the ideal state that the vehicle should reach at the next moment based on the driving decision result. For example, when the driving decision result is to turn right at the intersection ahead, the expected state can be that the vehicle decelerates appropriately when approaching the intersection, the steering angle is gradually adjusted to the appropriate right turn angle, and the position of the vehicle can accurately enter the right turn lane.

[0094] Specifically, the target state can be compared with the expected state, and the content of the comparison may include key parameters such as speed, position and steering angle. For example, it can be compared whether the predicted speed in the target state is consistent with the speed range required by the expected state, whether the predicted position in the target state is consistent with the lane and location where the vehicle should be in the expected state, and whether the predicted steering angle in the target state meets the requirements of the turning operation in the expected state, etc., so as to determine whether the target state is the same as the expected state corresponding to the driving decision result; when the target state is the same as the expected state corresponding to the driving decision result, it indicates that the driver is driving the vehicle according to the driving decision result, and S201 can be executed at this time to continue to determine the driving decision results at other future times; when the target state is different from the expected state corresponding to the driving decision result, it indicates that the driver is not driving the vehicle according to the driving decision result, and S212 can be executed at this time.

[0095] S212: When the target state is different from the expected state corresponding to the driving decision result, display the driving decision result to remind the driver to drive the vehicle according to the driving decision result.

[0096] Specifically, when the target state is different from the expected state corresponding to the driving decision result, the driving decision result can be displayed on the vehicle's central control screen, or the driving decision result can be announced by voice to remind the driver to drive the vehicle according to the driving decision result, especially to remind novice drivers.

[0097] For example, if there is a crosswalk ahead, and the driving decision result is to slow down and stop before the crosswalk, and the target state determined according to the real-time driving state information is to accelerate, the driving decision result can be played by voice to remind the driver to slow down and stop before the crosswalk.

[0098] Optionally, after obtaining the driving decision result, the actual driving state information of the vehicle at each moment in the historical time period can be obtained; the driving behavior score corresponding to each moment is determined based on the actual driving state information at each moment and the expected state corresponding to the driving decision result at each moment; and the total driving behavior score of the driver in the historical time period is determined based on the driving behavior score corresponding to each moment.

[0099] The actual driving status information is the real driving status data of the vehicle in the historical time period.

[0100] The driving behavior score is a quantitative evaluation of the degree to which the driver's driving behavior at a specific moment is consistent with the expected state. The total driving behavior score is an overall evaluation of the driver's driving behavior in the entire historical time period, which is obtained by comprehensively considering the driving behavior scores corresponding to each moment in the historical time period.

[0101] Specifically, the actual state of the vehicle at each moment can be determined based on the actual driving state information at each moment, and then it is determined whether the actual state at each moment is the same as the expected state corresponding to the driving decision result at each moment. If the actual state is the same as the expected state corresponding to the driving decision result, the driving behavior score corresponding to the moment is determined to be the first preset score; if the actual state is different from the expected state corresponding to the driving decision result, the driving behavior score corresponding to the moment is determined to be the second preset score, and the first preset score is greater than the second preset score. Afterwards, the sum of the driving behavior scores corresponding to each moment in the historical time period is determined as the total driving behavior score, and the total driving behavior score is displayed to inform the driver of the driving performance in the historical time period. Among them, the first preset score and the second preset score are preset values, which can be adjusted according to actual conditions. For example, the first preset score is 1 and the second preset score is 0.

[0102] In the embodiment of the present application, an objective and accurate score can be given strictly based on the degree of compliance of the driver's operation with traffic regulations, so that the driver can clearly quantify his or her driving performance, especially novice drivers, thereby promoting the driver to continuously improve his or her driving skills.

[0103] Optionally, in order to help novice drivers learn and improve their driving skills more effectively, after the driver has driven for a period of time, the driver's driving behavior during this period can be compared with the corresponding driving decision results, and the situation where the driving behavior does not match the corresponding driving decision results can be recorded. Then, an analysis report can be generated based on the recorded content and displayed. The analysis report can accurately point out the traffic conditions that the driver needs to pay special attention to and the correct operating methods that should be taken, thereby helping the driver to draw valuable experience from actual driving experience; at the same time, targeted suggestions can be generated based on the analysis report, thereby helping the driver to clarify the direction of improvement and continuously improve his or her driving skills.

[0104] According to the technical solution of the embodiment of the present application, after determining the dynamic target and the static target according to the target information, each frame of the image can be annotated based on the position of the dynamic target in each frame of the image to obtain a target frame image sequence, and provide accurate position information input for the subsequent trajectory detection model, and the target frame image sequence is input into the pre-trained trajectory detection model to obtain the motion trajectory of each dynamic target in the historical time period. Through the pre-trained trajectory detection model, the position change of the dynamic target between different frames can be accurately tracked, especially when the vehicle moves quickly or the target is partially blocked, thereby improving the accuracy and efficiency of determining the motion trajectory, and providing an accurate data basis for the subsequent driving decision prediction model; or, the initial position of each dynamic target can be determined according to the position of the dynamic target in each frame of the image, providing an accurate starting reference point for target tracking, and the dynamic target is continuously tracked based on the initial position of each dynamic target to obtain the motion trajectory of each dynamic target in the historical time period, which can completely record the motion path of the dynamic target in the historical time period and improve the efficiency of target tracking, thereby improving the accuracy and efficiency of determining the motion trajectory, and providing an accurate data basis for the subsequent driving decision prediction model.

[0105] After obtaining the driving decision result of the vehicle at the next moment, the real-time driving status information of the vehicle after the current moment can be obtained, and the target state of the vehicle at the next moment can be determined based on the real-time driving status information. Then, when the target state is different from the expected state corresponding to the driving decision result, the driving decision result is displayed to remind the driver to drive the vehicle according to the driving decision result. By comparing the target state with the expected state, the difference between the actual driving trend of the vehicle and the driving decision result can be discovered in time, and the driver can be reminded in time to drive according to the driving decision result, which can help the driver better cope with complex traffic scenes, thereby reducing the risk of traffic accidents and ensuring the personal safety of occupants and other traffic participants on the road.

[0106] Figure 3 is a schematic diagram of the structure of the driving assistance device provided in the embodiment of the present application, with reference to Figure 3 , the driving assistance device may include:

[0107] The acquisition module 310 is used to acquire the video stream obtained by the camera capturing the surrounding environment of the vehicle and to acquire the driving state information of the vehicle at the current moment;

[0108] The target detection module 320 is used to input each frame of the video stream into a pre-trained target detection model to obtain target information in each frame of the video stream, and determine dynamic targets and static targets based on the target information, wherein the target information includes category and position;

[0109] A determination module 330, configured to determine the static environment information of each frame of image according to the category and position of the static target in each frame of image, and determine the motion trajectory of each dynamic target in a historical time period according to the position of the dynamic target in each frame of image;

[0110] The decision prediction module 340 is used to input the motion trajectory of each dynamic target in the historical time period, the static environment information of each frame image and the driving state information into a pre-trained driving decision prediction model to obtain the driving decision result of the vehicle at the next moment to assist the driver in driving the vehicle.

[0111] In one embodiment, the determination module 330 determines the motion trajectory of each dynamic target in the historical time period based on the position of the dynamic target in each frame image, including: labeling each frame image based on the position of the dynamic target in each frame image to obtain a target frame image sequence; inputting the target frame image sequence into a pre-trained trajectory detection model to obtain the motion trajectory of each dynamic target in the historical time period.

[0112] In one embodiment, the determination module 330 determines the motion trajectory of each dynamic target in the historical time period according to the position of the dynamic target in each frame image, including: determining the initial position of each dynamic target according to the position of the dynamic target in each frame image; continuously tracking the dynamic target based on the initial position of each dynamic target to obtain the motion trajectory of each dynamic target in the historical time period.

[0113] In one embodiment, the driving decision prediction model includes a feature extraction module and a driving decision module. The training process of the driving decision prediction model in the decision prediction module 340 is as follows: obtain a sample data set, the sample data set includes sample driving state information, sample static environment information, sample dynamic target motion trajectory and driving decision result label; input the sample static environment information into the feature extraction module to obtain a target feature map sequence; input the sample driving state information, the sample dynamic target motion trajectory and the target feature map sequence into the driving decision module to obtain a driving decision prediction result; calculate the loss value between the driving decision prediction result and the driving decision result label, and use the back propagation algorithm to optimize the feature extraction module and the driving decision module to minimize the loss value to obtain a driving decision prediction model.

[0114] In one embodiment, the target detection module 320 determines dynamic targets and static targets based on target information, including: when the category is in a preset static category table, determining that the target corresponding to the category is a static target; when the category is in a preset dynamic category table, determining that the target corresponding to the category is a dynamic target.

[0115] In one embodiment, the driving assistance device also includes a display module, which is specifically used to: obtain real-time driving status information of the vehicle after the current moment; determine the target state of the vehicle at the next moment based on the real-time driving status information; when the target state is different from the expected state corresponding to the driving decision result, display the driving decision result to remind the driver to drive the vehicle according to the driving decision result.

[0116] In one embodiment, the driving assistance device also includes a driving scoring module, which is specifically used to: obtain actual driving state information of the vehicle at each moment in a historical time period; determine the driving behavior score corresponding to each moment based on the actual driving state information at each moment and the expected state corresponding to the driving decision result at each moment; and determine the driver's total driving behavior score in the historical time period based on the driving behavior score corresponding to each moment.

[0117] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0118] The driving assistance device provided in this embodiment can be applied to the driving assistance method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0119] Figure 4 It is a structural schematic diagram of a vehicle provided in an embodiment of the present application. Figure 4 A block diagram of an exemplary vehicle 11 suitable for implementing embodiments of the present application is shown. Figure 4 The vehicle 11 shown is merely an example and should not bring any limitation to the function and scope of use of the present embodiment.

[0120] like Figure 4 As shown, the vehicle 11 is embodied in the form of a general purpose computing electronic device. The components of the vehicle 11 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting various system components (including the system memory 28 and the processing unit 16).

[0121] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0122] The vehicle 11 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the vehicle 11, including volatile and non-volatile media, removable and non-removable media.

[0123] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The vehicle 11 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0124] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0125] The vehicle 11 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the vehicle 11, and / or any device that enables the vehicle 11 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed through an input / output (I / O) interface 22. Furthermore, the vehicle 11 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 20.

[0126] like Figure 4 As shown, the network adapter 20 communicates with other modules of the vehicle 11 via the bus 18. It should be understood that although Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with the vehicle 11, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0127] The processing unit 16 executes various functional applications and page displays by running programs stored in the system memory 28, such as implementing a driving assistance method provided by any embodiment of the present application.

[0128] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, a driving assistance method provided by any embodiment of the present application is implemented, for example.

[0129] The computer storage medium of this embodiment can adopt any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to: electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing programs, which can be used by instruction execution systems, devices or devices or used in combination with them.

[0130] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0131] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0132] Computer program code for performing the operation of the present application can be written in one or more programming languages ​​or combinations thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0133] It should be understood by those skilled in the art that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0134] In addition, the acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0135] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the inventive concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A driving assistance method, characterized in that: Applied to a vehicle, wherein a camera is installed on the vehicle, the method comprises: Obtaining a video stream obtained by the camera capturing the surrounding environment of the vehicle, and obtaining driving state information of the vehicle at the current moment; Inputting each frame of the video stream into a pre-trained target detection model to obtain target information in each frame of the video stream, and determining dynamic targets and static targets according to the target information, wherein the target information includes category and position; Determine the static environment information of each frame of image according to the category and position of the static target in each frame of image, and determine the motion trajectory of each dynamic target in a historical time period according to the position of the dynamic target in each frame of image; The motion trajectory of each dynamic target in the historical time period, the static environment information of each frame image and the driving state information are input into a pre-trained driving decision prediction model to obtain the driving decision result of the vehicle at the next moment to assist the driver in driving the vehicle.

2. The driving assistance method according to claim 1, characterized in that: Determining the motion trajectory of each dynamic target in the historical time period according to the position of the dynamic target in each frame of the image includes: Annotating each frame of the image based on the position of the dynamic target in each frame of the image to obtain a target frame image sequence; The target frame image sequence is input into a pre-trained trajectory detection model to obtain the motion trajectory of each dynamic target in a historical time period.

3. The driving assistance method according to claim 1, characterized in that: Determining the motion trajectory of each dynamic target in the historical time period according to the position of the dynamic target in each frame of the image includes: Determining the initial position of each dynamic target according to the position of the dynamic target in each frame of image; The dynamic target is continuously tracked based on the initial position of each dynamic target to obtain the motion trajectory of each dynamic target in the historical time period.

4. The driving assistance method according to claim 1, characterized in that: The driving decision prediction model includes a feature extraction module and a driving decision module. The training process of the driving decision prediction model is as follows: Acquire a sample data set, wherein the sample data set includes sample driving state information, sample static environment information, sample dynamic target motion trajectory and driving decision result label; Inputting the sample static environment information into the feature extraction module to obtain a target feature map sequence; Inputting the sample driving state information, the sample dynamic target motion trajectory and the target feature map sequence into the driving decision module to obtain a driving decision prediction result; The loss value between the driving decision prediction result and the driving decision result label is calculated, and the feature extraction module and the driving decision module are optimized by using a back propagation algorithm to minimize the loss value, thereby obtaining the driving decision prediction model.

5. The driving assistance method according to claim 1, characterized in that: Determining the dynamic target and the static target according to the target information includes: When the category is in the preset static category table, determining that the target corresponding to the category is a static target; When the category is in the preset dynamic category table, it is determined that the target corresponding to the category is a dynamic target.

6. The driving assistance method according to claim 1, characterized in that: The method further comprises: Acquire real-time driving status information of the vehicle after the current moment; Determining a target state of the vehicle at the next moment according to the real-time driving state information; When the target state is different from the expected state corresponding to the driving decision result, the driving decision result is displayed to remind the driver to drive the vehicle according to the driving decision result.

7. The driving assistance method according to claim 1, characterized in that: The method further comprises: Acquire the actual driving status information of the vehicle at each moment in the historical time period; Determine the driving behavior score corresponding to each moment according to the actual driving state information at each moment and the expected state corresponding to the driving decision result at each moment; The total driving behavior score of the driver in the historical time period is determined according to the driving behavior score corresponding to each moment.

8. A driving assistance device, characterized in that: Applied to a vehicle, the vehicle is equipped with a camera, and the device comprises: An acquisition module, used to acquire a video stream obtained by the camera capturing the surrounding environment of the vehicle, and to acquire driving state information of the vehicle at a current moment; A target detection module, used to input each frame of the video stream into a pre-trained target detection model, obtain target information in each frame of the video stream, and determine dynamic targets and static targets according to the target information, wherein the target information includes category and position; A determination module, used to determine the static environment information of each frame of image according to the category and position of the static target in each frame of image, and determine the motion trajectory of each dynamic target in a historical time period according to the position of the dynamic target in each frame of image; The decision prediction module is used to input the motion trajectory of each dynamic target in the historical time period, the static environment information of each frame image and the driving state information into a pre-trained driving decision prediction model to obtain the driving decision result of the vehicle at the next moment to assist the driver in driving the vehicle.

9. A vehicle, characterized in that: The vehicle comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the driving assistance method described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the driving assistance method as claimed in any one of claims 1 to 7 is implemented.

Images