Driving intention prediction method and device based on neural network, equipment and storage medium

Abstract

The invention discloses a driving intention prediction method and device based on a neural network, equipment and a storage medium. The method comprises the following steps: acquiring a camera video of a driver, and obtaining image data of a driver state time sequence from the video; performing first data preprocessing on the image data, and extracting feature data from the image data after the first data preprocessing; performing secondary data preprocessing on the feature data; performing feature fusion on the feature data after the secondary data preprocessing according to a feature fusion network to obtain a fused combined feature data set; taking the combined feature data set as modeling data, and constructing a driving intention prediction model based on a convolutional long-short-term memory network; according to the invention, secondary data preprocessing is carried out on the acquired image data, feature fusion is carried out on the feature data according to the feature fusion network, the fused combined feature data set is used as modeling data, the accuracy of the constructed driving intention prediction model is high, and the actual emergency obstacle avoidance requirement can be met at the same time.

Description

Technical Field

[0001] The present invention relates to the field of vehicle obstacle avoidance, and in particular to a driving intention prediction method, device, equipment and storage medium based on a neural network. Background Art

[0002] When the vehicle encounters an emergency and cannot completely avoid a collision accident through the AEB system (automatic braking), the emergency avoidance system can control the vehicle to perform lateral avoidance (using the space of the adjacent lane) through joint planning of time and space (input: driving environment information, including information of other traffic participants, road-related information, etc.; vehicle information, such as vehicle speed, acceleration, steering wheel angle, etc.; output: emergency avoidance path planning and speed planning); the AES system (emergency obstacle avoidance system) can play an important role in reducing the risk of vehicle collisions, especially for high-level intelligent driving systems above L3.

[0003] However, during the avoidance process activated by AES, if the driver's driving intention does not match the emergency avoidance operation of the ADAS system (Advanced Driver Assistance System), it may cause the driver to panic and make misoperations, greatly interfering with the normal operation of the AES system. Especially in the case of continuous avoidance, the avoidance operation may fail. Therefore, it is very critical to identify the driver's driving intention.

[0004] The inventor has found through research that the driving intention recognition method in the prior art still has at least the following defects:

[0005] The driver's intention is judged based on the accelerator, brake pedal signals and steering wheel angle signals. However, this kind of driving intention recognition that relies on the driver's reaction and the signals after the action is implemented will inevitably bring too much time lag and cannot meet the actual needs in emergency obstacle avoidance scenarios. There is an urgent need for a driving intention recognition method that can meet the actual emergency obstacle avoidance needs.

[0006] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgment or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the invention

[0007] The purpose of the present invention is to provide a driving intention recognition method that can meet actual emergency obstacle avoidance needs.

[0008] The present invention provides a driving intention prediction method based on a neural network, comprising the steps of:

[0009] S11, acquiring a camera video of the driver, and obtaining image data of the driver's state time series from the video;

[0010] S12, performing a first data preprocessing on the image data, and extracting feature data from the image data after the first data preprocessing; the feature data includes: attention feature data and behavior feature data;

[0011] S13, performing secondary data preprocessing on the feature data; the secondary data preprocessing includes:

[0012] Normalizing and / or filtering the attention feature data and the behavior feature data;

[0013] S14, performing feature fusion on the feature data after the secondary data preprocessing according to the feature fusion network to obtain a fused combined feature data set;

[0014] S15. Using the combined feature data set as modeling data, a driving intention prediction model based on a convolutional long short-term memory network is constructed.

[0015] Preferably, in the embodiment of the present invention,

[0016] The attention feature data includes: facial key point coordinate data and head orientation coordinate data;

[0017] The behavior characteristic data includes: coordinate data of key points of the upper body and coordinate data of key points of the arms.

[0018] Preferably, in the embodiment of the present invention,

[0019] The facial key point coordinate data, the upper body key point coordinate data and the arm key point coordinate data are feature extracted according to a human body posture estimation model;

[0020] The head orientation coordinate data is feature extracted according to a head posture estimation model.

[0021] Preferably, in an embodiment of the present invention, normalizing and / or filtering the attention feature data and the behavior feature data includes:

[0022] Performing pixel value normalization processing and filtering processing on the facial key point coordinate data, the upper body key point coordinate data, and the arm key point coordinate data;

[0023] The head orientation coordinate data is subjected to filtering processing.

[0024] Preferably, in an embodiment of the present invention, the pixel value normalization process includes the formula:

[0025]

[0026] In the formula, x is the original pixel value, x normalizedis the normalized pixel value.

[0027] Preferably, in an embodiment of the present invention, the filtering process includes:

[0028] Determine the window size based on the smoothing effect and sensitivity to changes;

[0029] For each set of key point coordinates, starting from the current frame, select a time window containing N frames, where N is the preset window length, and for each frame in the time window, record the position coordinates of the corresponding key point;

[0030] Calculate the average value of the coordinate data of each corresponding key point in the time window as the position coordinate of the key point in the current window.

[0031] In another aspect of the present invention, a driving intention prediction device based on a neural network is also provided, comprising:

[0032] An image data acquisition unit, used to acquire a camera video of the driver and obtain image data of the driver's state time series from the video;

[0033] A feature data extraction unit, used to perform a first data preprocessing on the image data, and extract feature data from the image data after the first data preprocessing; the feature data includes: attention feature data and behavior feature data;

[0034] The feature data preprocessing unit is used to perform secondary data preprocessing on the feature data; the secondary data preprocessing includes:

[0035] Normalizing and / or filtering the attention feature data and the behavior feature data;

[0036] A modeling data generating unit, used for performing feature fusion on the feature data after the secondary data preprocessing according to a feature fusion network to obtain a fused combined feature data set;

[0037] The driving intention prediction model building unit is used to build a driving intention prediction model based on a convolutional long short-term memory network using the combined feature data set as modeling data.

[0038] Preferably, in an embodiment of the present invention, the attention feature data includes: facial key point coordinate data and head orientation coordinate data; the behavior feature data includes: upper body key point coordinate data and arm key point coordinate data.

[0039] On the other side of the embodiment of the present invention, a driving intention prediction device based on a neural network is also provided; the driving intention prediction device based on a neural network includes a computer program stored on a medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the methods described in the above aspects and achieves the same technical effects.

[0040] On the other hand, a storage medium is provided on which a computer program is stored. When the computer program is executed by a processor, each step of the driving intention prediction method based on a neural network as described in any one of the above items is implemented.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] In the present invention, the camera video of the driver is firstly acquired, and the image data of the driver's state time series is obtained from the video; the image data is subjected to the first data preprocessing, and the feature data is extracted from the image data after the first data preprocessing; the feature data is subjected to the second data preprocessing; then, the feature data after the second data preprocessing is subjected to feature fusion according to the feature fusion network, and a fused combined feature data set is obtained; finally, the combined feature data set is used as the modeling data, and a driving intention prediction model based on the convolutional long short-term memory network is constructed; in the present invention, the acquired image data is subjected to the second data preprocessing, and the feature data after the second data preprocessing is subjected to feature fusion according to the feature fusion network, and the fused combined feature data set is used as the modeling data, so that the driving intention prediction model constructed in this way is not only highly accurate, but also more able to meet the actual emergency obstacle avoidance needs.

[0043] Furthermore, the present invention performs coordinated control through the driving intention prediction model and the intelligent driving AES function algorithm, thereby avoiding the situation where the decision intention of the traditional intelligent driving system is contrary to the driver's intention.

[0044] In addition, the present invention obtains image data of the driver's state time series through a sensor, thereby realizing the prediction of the driver's driving intention at a relatively low cost.

[0045] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, and to make the above and other purposes, technical features and advantages of the present invention easier to understand, one or more preferred embodiments are listed below and described in detail with the accompanying drawings as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 is a step diagram of the driving intention prediction method based on neural network described in the present invention;

[0048] Figure 2 is a flow chart of the driving intention prediction method based on neural network described in the present invention;

[0049] Figure 3 It is a flow chart of the coordinated integration of the driving intention prediction method based on neural network and the intelligent driving AES function processing described in the present invention;

[0050] Figure 4 It is a flowchart of the intelligent driving AES function processing described in the present invention;

[0051] Figure 5 is a schematic structural diagram of the driving intention prediction device based on a neural network in the present invention;

[0052] Figure 6 It is a structural schematic diagram of the driving intention prediction device based on neural network described in the present invention. DETAILED DESCRIPTION

[0053] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific implementation modes.

[0054] Unless explicitly stated otherwise, throughout the specification and claims, the term “comprise” or variations such as “include” or “comprising”, etc., will be understood to include the stated elements or components but not to exclude other elements or components.

[0055] In this document, the terms "first", "second", etc. are used to distinguish two different elements or parts, and are not used to limit a specific position or relative relationship. In other words, in some embodiments, the terms "first", "second", etc. can also be interchangeable.

[0056] Embodiment 1

[0057] In order to meet the actual emergency obstacle avoidance needs of driving intention recognition, such as Figure 1 and Figure 2 As shown, in an embodiment of the present invention, a driving intention prediction method based on a neural network is provided, comprising the steps of:

[0058] S11, acquiring a camera video of the driver, and obtaining image data of the driver's state time series from the video;

[0059] The inventive concept of the present invention includes: during the emergency driving of the vehicle, the movement state of external vehicles and pedestrians can be detected through environmental perception sensors such as cameras and radars, and combined with the escape path planning and decision-making in the AES (automatic emergency steering) function, several feasible escape paths can be decided on the premise of ensuring the driving safety of the vehicle; at the same time, the driver detection camera in the car aimed at the driver obtains the driver's facial expression, head movement, upper body, arm movement and other continuous frame image information in real time, and performs data preprocessing on the image information. According to the feature fusion network, the feature data after the secondary data preprocessing is feature fused, and the fused combined feature data set is used as the modeling data. The driving intention prediction model constructed in this way has high accuracy. Finally, the prediction result of the driving intention prediction model is coordinated and integrated with the decision-making scheme of several feasible escape paths obtained by the AES decision, and finally an escape path that meets the driver's operating intention and the driving safety of the vehicle under emergency conditions is obtained. The vehicle is tracked and controlled according to this path, avoiding the situation where accidents occur due to the contradiction between the driver's subjective driving intention and the AES decision scheme under emergency conditions.

[0060] In an embodiment of the present invention, a sensor (such as a camera) is used to obtain a driver video in real time, and then an image is extracted from the video at a specific frame rate to obtain image data of the driver's state time series.

[0061] S12, performing a first data preprocessing on the image data, and extracting feature data from the image data after the first data preprocessing; the feature data includes: attention feature data and behavior feature data;

[0062] In an embodiment of the present invention, the attention feature data includes: facial key point coordinate data and head orientation coordinate data; the behavior feature data includes: upper body key point coordinate data and arm key point coordinate data.

[0063] Among them, the facial key point coordinate data, the upper torso key point coordinate data and the arm key point coordinate data are feature extracted according to the human body posture estimation model; the head orientation coordinate data are feature extracted according to the head posture estimation model.

[0064] In the embodiment of the present invention, the image data is subjected to a first data preprocessing, namely, pixel value normalization processing, including scaling the pixel value of each pixel point of the image from the range of 0-255 to between 0-1, and transforming it by the following formula:

[0065]

[0066] In the formula, x is the original pixel value, x normalized is the normalized pixel value.

[0067] S13, performing secondary data preprocessing on the feature data; the secondary data preprocessing includes:

[0068] Normalizing and / or filtering the attention feature data and the behavior feature data;

[0069] In the embodiment of the present invention, pixel value normalization processing and filtering processing are performed on the facial key point coordinate data, the upper body key point coordinate data and the arm key point coordinate data;

[0070] The head orientation coordinate data is subjected to filtering processing.

[0071] In the embodiment of the present invention, the pixel value normalization process includes the formula:

[0072]

[0073] In the formula, x is the original pixel value, x normalized is the normalized pixel value.

[0074] In the embodiment of the present invention, filtering is a simple moving average. Simple moving average (SMA) is a technique widely used for smoothing time series data, which can be used to reduce noise and highlight the main trend of the data. In the case of processing key point coordinate changes, SMA can help smooth the position fluctuations of human joints or facial feature points detected in the video frame, which is particularly important in gesture recognition, motion capture or driving behavior analysis. The following are the basic steps for using SMA to smooth key point coordinate changes, including:

[0075] Determine the window size based on the smoothing effect and sensitivity to changes;

[0076] The window size (or period) is the number of frames to consider when calculating the average. Larger windows provide smoother results, but may increase latency and react slower to sudden changes. Smaller windows react faster to changes, but are less smooth. When choosing a window size, there is a trade-off between smoothing and sensitivity to change.

[0077] For each set of key point coordinates, starting from the current frame, select a time window containing N frames, where N is the preset window length. For each frame in the time window, record the position coordinates of the corresponding key point. Then, calculate the average value of the coordinates of these key points in the time window as the representative position of the key point in the current window.

[0078] If the window length is 5, then for the tth frame, SMA will be the average of the keypoint coordinates in the 5 frames from t-4 to t.

[0079] Assume that the original coordinates of the key point in the tth frame are (x t ,y t ), then the SMA calculation formula is:

[0080]

[0081] The average coordinate value replaces the original coordinate value or takes the average coordinate value as the new coordinate value.

[0082] The calculated SMA coordinate values ​​replace the original coordinates or are used as new coordinate values ​​to achieve a smoothing effect.

[0083] Repeat the above steps to process the coordinates of each key point in the video, including facial key point coordinate data, upper body key point coordinate data, arm key point coordinate data and head orientation coordinate data.

[0084] S14, performing feature fusion on the feature data after the secondary data preprocessing according to the feature fusion network to obtain a fused combined feature data set;

[0085] In an embodiment of the present invention, the facial key point coordinate data and the head orientation coordinate data after secondary data preprocessing; the upper torso key point coordinate data and the arm key point coordinate data are subjected to a feature fusion network FPN (i.e., feature fusion is performed in time series) to obtain a fused combined feature data set.

[0086] S15. Using the combined feature data set as modeling data, a driving intention prediction model based on a convolutional long short-term memory network is constructed.

[0087] In an embodiment of the present invention, the fused combined feature data set is used as modeling data to construct a driving intention prediction model based on a convolutional long short-term memory network (Conv-LSTM), and the driving intention misidentification result can be obtained through the driving intention prediction model.

[0088] Furthermore, in the embodiment of the present invention, during the construction of the driving intention prediction model, in the training of the convolutional long short-term memory network, the data set is divided into a training set, a validation set, and a test set in a ratio of 80%:10%:10% (i.e., 8:1:1). The main purpose of this division is for model selection, parameter adjustment, and final performance evaluation. The rationality of this division method is that:

[0089] (1) Resource Utilization

[0090] Most of the data (80%) is used for training to ensure that the model has enough samples to learn;

[0091] (2) Generalization ability evaluation

[0092] 10% of the validation set is used to adjust the model during training to avoid overfitting;

[0093] 10% of the test set is used for the final evaluation to ensure the fairness of the evaluation.

[0094] The following is an explanation of the coordinated integration of the prediction results of the driving intention prediction model and several feasible escape paths obtained by AES decision-making in combination with specific application scenarios.

[0095] like Figure 3 and Figure 4 As shown in the figure, in an emergency obstacle avoidance scenario, such as when a pedestrian suddenly crosses the road ahead, AES obtains the target obstacle status and the vehicle status through sensor perception and fusion.

[0096] The information output after sensor perception and fusion recognition includes:

[0097] Valid target four-wheel vehicle sequence:

[0098] CarList=[C1,C2,C nc ]

[0099] C n ={ID nc ,v x ,v y ,v z ,,a x ,a y ,a z}

[0100] Among them, CarList is the four-wheel vehicle information list, nc is the vehicle serial number, C n is the vehicle dynamics information array, ID nc is the fused vehicle ID number, v x is the lateral velocity of the vehicle, v y is the longitudinal velocity of the vehicle, v z is the vertical velocity of the vehicle, a x is the lateral acceleration of the vehicle, a y is the longitudinal acceleration of the vehicle, a z is the vehicle vertical acceleration.

[0101] Valid target two-wheeled or three-wheeled vehicle sequence:

[0102] SCarList = [S1,S2,S ns ]

[0103] S ns ={ID ns ,v xs ,v ys ,v zs ,a xs ,a ys ,a zs}

[0104] Among them, SCarList is a list of two-wheeled or three-wheeled vehicle information, ns is the serial number of the vehicle, S ns is the vehicle dynamics information array, IDns is the fused vehicle ID number, v xs is the lateral velocity of the vehicle, v ys is the longitudinal velocity of the vehicle, v zs is the vertical velocity of the vehicle, a xs is the lateral acceleration of the vehicle, a ys is the longitudinal acceleration of the vehicle, a zs is the vehicle vertical acceleration.

[0105] Valid target pedestrian sequence:

[0106] Walkerist=[W1,W2 W nw ]

[0107] W nw ={ID nw ,v xw ,v yw ,v zw ,a xw ,a yw ,a zw}

[0108] Among them, Walkerist is the valid target pedestrian information list, nw is the number of valid target pedestrians, W is the pedestrian motion information array, ID nw is the pedestrian ID number after fusion, v xw is the pedestrian's lateral velocity, v yw is the pedestrian's longitudinal velocity, v zw is the vertical speed of the pedestrian, a xw is the lateral acceleration of the pedestrian, a yw is the pedestrian's longitudinal acceleration, a zw is the vertical acceleration of the pedestrian.

[0109] Valid target small object (such as falling objects, etc.) sequence:

[0110] SmallObjList=[O1,O2,O no ]

[0111] O no ={ID no ,vxo ,v yo ,v zo ,a xo ,a yo ,a zo}

[0112] Among them, SmallObjList is the small object information list, no is the number of small objects, O is the small object motion information array, ID no is the ID number of the small object after fusion, v xo is the lateral velocity of the small object, v yo is the longitudinal velocity of the small object, v zo is the vertical velocity of the small object, a xo is the lateral acceleration of the small object, a yo is the longitudinal acceleration of the small object, a zo is the vertical acceleration of the small object.

[0113] Lane recognition:

[0114] LaneList = [L1,L2,L nl ]

[0115] L nl ={ID nl ,LineType,LineColor}

[0116] LaneList is the lane line information list, nl is the nlth lane line, L nl is the lane line information array, ID nl To merge the lane line ID number, LineType is the lane line type (solid line, dashed line, double solid line and double dashed line, etc.), and LineColor is the lane line color (yellow, white, etc.).

[0117] Path planning:

[0118] The quintic polynomial path planning algorithm is used, the core of which is to calculate a safe and reliable escape route in time when encountering an emergency road condition ahead; including the following steps:

[0119] (1) Calculate the longitudinal limit escape time T:

[0120] T=(Pose_FrCar–Pose_MasterCar) / (V_FrCar–V_MasterCar)

[0121] Among them, Pose_FrCar is the longitudinal distance of the front vehicle, Pose_MasterCar is the longitudinal distance of the master vehicle, V_FrCar is the longitudinal speed of the front vehicle, and V_MasterCar is the longitudinal speed of the master vehicle.

[0122] (2) If the maximum escape time requirement is met (e.g., T>5s), multiple target escape destinations are selected based on the main target vehicle in front and on its left side of the road.

[0123] (3) Perform a fifth-order polynomial path planning calculation for each escape destination and combine it with the vehicle dynamics state to obtain multiple escape paths. The fifth-order polynomial calculation formula is:

[0124] q(t)=q0+a1(t-t0)+a2(t-t0) 2 +a3(t-t0) 3 +a4(t-t0) 4 +a5(t-t0) 5

[0125] q(t)=q0,q(t1)=q1

[0126]

[0127] In the formula, q is the coordinate point, which contains the horizontal and vertical coordinate information (q x ,q y ), Speed ​​information, including horizontal and vertical speed information It is acceleration information, including horizontal and vertical acceleration information

[0128] (4) The decision in the intelligent driving decision is to turn left, and several escape routes that best meet the vehicle dynamics boundaries are selected based on the lateral acceleration requirements.

[0129] At the same time, the driver's image data is obtained based on the driver detection sensor, and the prediction result of the driving intention prediction model is a left turn. By coordinating and integrating the escape route in the intelligent driving decision-making with the prediction result of the driving intention prediction model, an escape path that conforms to both the vehicle dynamics boundary and the driver's driving intention can be determined.

[0130] To summarize, in an embodiment of the present invention, a camera video of the driver is first acquired to obtain image data of the driver's state time series from the video; a first data preprocessing is performed on the image data, and feature data is extracted from the image data after the first data preprocessing; a second data preprocessing is performed on the feature data; then, feature fusion is performed on the feature data after the second data preprocessing according to a feature fusion network to obtain a fused combined feature data set; finally, the combined feature data set is used as modeling data to construct a driving intention prediction model based on a convolutional long short-term memory network; in the present invention, a second data preprocessing is performed on the acquired image data, and feature fusion is performed on the feature data after the second data preprocessing according to a feature fusion network, and the fused combined feature data set is used as modeling data, so that the driving intention prediction model constructed in this way is not only highly accurate, but also more able to meet the actual emergency obstacle avoidance needs.

[0131] Furthermore, the present invention performs coordinated control through the driving intention prediction model and the intelligent driving AES function algorithm, thereby avoiding the situation where the decision intention of the traditional intelligent driving system is contrary to the driver's intention.

[0132] In addition, the present invention obtains image data of the driver's state time series through a sensor, thereby realizing the prediction of the driver's driving intention at a relatively low cost.

[0133] Embodiment 2

[0134] Corresponding to the method embodiment, in another aspect of the embodiment of the present invention, a driving intention prediction device is also provided. Figure 5 A schematic diagram showing the structure of a driving intention prediction device provided by an embodiment of the present invention is shown. The driving intention prediction device is Figure 1 The device corresponding to the driving intention prediction method described in the corresponding embodiment, that is, the virtual device is used to implement Figure 1 In the corresponding embodiment of the driving intention prediction method, each virtual module constituting the driving intention prediction device can be executed by an electronic device, such as a network device, a terminal device or a server. Specifically, the driving intention prediction device in the embodiment of the present invention includes:

[0135] An image data acquisition unit 01 is used to acquire a camera video of the driver and obtain image data of the driver's state time series from the video;

[0136] In an embodiment of the present invention, a sensor (such as a camera) is used to obtain a driver video in real time, and then an image is extracted from the video at a specific frame rate to obtain image data of the driver's state time series.

[0137] The feature data extraction unit 02 is used to perform a first data preprocessing on the image data, and extract feature data from the image data after the first data preprocessing; the feature data includes: attention feature data and behavior feature data;

[0138] The feature data preprocessing unit 03 is used to perform secondary data preprocessing on the feature data; the secondary data preprocessing includes:

[0139] Normalizing and / or filtering the attention feature data and the behavior feature data;

[0140] In the embodiment of the present invention, the attention feature data includes: facial key point coordinate data and head orientation coordinate data; the behavior feature data includes: upper body key point coordinate data and arm key point coordinate data.

[0141] Among them, facial key point coordinate data, upper torso key point coordinate data and arm key point coordinate data are feature extracted according to the human body posture estimation model; head orientation coordinate data are feature extracted according to the head posture estimation model.

[0142] In the embodiment of the present invention, the image data is subjected to a first data preprocessing, namely, pixel value normalization processing, including scaling the pixel value of each pixel point of the image from the range of 0-255 to between 0-1, and transforming it by the following formula:

[0143]

[0144] In the formula, x is the original pixel value, x normalized is the normalized pixel value.

[0145] Preferably, in an embodiment of the present invention, the basic steps of using SMA to smooth the change of key point coordinates include:

[0146] Determine the window size based on the smoothing effect and sensitivity to changes;

[0147] The window size (or period) is the number of frames to consider when calculating the average. Larger windows provide smoother results, but may increase latency and react slower to sudden changes. Smaller windows react faster to changes, but are less smooth. When choosing a window size, there is a trade-off between smoothing and sensitivity to change.

[0148] For each set of key point coordinates, starting from the current frame, select a time window containing N frames, where N is the preset window length. For each frame in the time window, record the position coordinates of the corresponding key point. Then, calculate the average value of the coordinates of these key points in the time window as the representative position of the key point in the current window.

[0149] If the window length is 5, then for the tth frame, SMA will be the average of the keypoint coordinates in the 5 frames from t-4 to t.

[0150] Assume the original coordinates of the key point in the tth frame are (x t ,y t ), then the SMA calculation formula is:

[0151]

[0152] The average coordinate value replaces the original coordinate value or takes the average coordinate value as the new coordinate value.

[0153] The calculated SMA coordinate values ​​replace the original coordinates or are used as new coordinate values ​​to achieve a smoothing effect.

[0154] Repeat the above steps to process the coordinates of each key point in the video, including facial key point coordinate data, upper body key point coordinate data, arm key point coordinate data and head orientation coordinate data.

[0155] A modeling data generating unit 04 is used to perform feature fusion on the feature data after the secondary data preprocessing according to a feature fusion network to obtain a fused combined feature data set;

[0156] In an embodiment of the present invention, the facial key point coordinate data and the head orientation coordinate data after secondary data preprocessing; the upper torso key point coordinate data and the arm key point coordinate data are subjected to a feature fusion network FPN (i.e., feature fusion is performed in time series) to obtain a fused combined feature data set.

[0157] The driving intention prediction model building unit 05 is used to build a driving intention prediction model based on a convolutional long short-term memory network using the combined feature data set as modeling data.

[0158] In an embodiment of the present invention, the fused combined feature data set is used as modeling data to construct a driving intention prediction model based on a convolutional long short-term memory network (Conv-LSTM), and the driving intention misidentification result can be obtained through the driving intention prediction model.

[0159] To summarize, in an embodiment of the present invention, a camera video of the driver is first acquired to obtain image data of the driver's state time series from the video; a first data preprocessing is performed on the image data, and feature data is extracted from the image data after the first data preprocessing; a second data preprocessing is performed on the feature data; then, feature fusion is performed on the feature data after the second data preprocessing according to a feature fusion network to obtain a fused combined feature data set; finally, the combined feature data set is used as modeling data to construct a driving intention prediction model based on a convolutional long short-term memory network; in the present invention, a second data preprocessing is performed on the acquired image data, and feature fusion is performed on the feature data after the second data preprocessing according to a feature fusion network, and the fused combined feature data set is used as modeling data, so that the driving intention prediction model constructed in this way is not only highly accurate, but also more able to meet the actual emergency obstacle avoidance needs.

[0160] Furthermore, the present invention performs coordinated control through the driving intention prediction model and the intelligent driving AES function algorithm, thereby avoiding the situation where the decision intention of the traditional intelligent driving system is contrary to the driver's intention.

[0161] In addition, the present invention obtains image data of the driver's state time series through a sensor, thereby realizing the prediction of the driver's driving intention at a relatively low cost.

[0162] Embodiment 3

[0163] Corresponding to the method embodiment, in the embodiment of the present invention, a driving intention prediction device, such as a terminal, a server, etc., is also provided. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto.

[0164] An example of a hardware structure block diagram of a driving intention prediction device provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, it may include:

[0165] Processor 1, communication interface 2, memory 3 and communication bus 4;

[0166] The processor 1, the communication interface 2, and the memory 3 communicate with each other via the communication bus 4;

[0167] Optionally, the communication interface 2 may be an interface of a communication module, such as an interface of a GSM module;

[0168] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0169] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0170] The processor 1 is specifically used to execute the computer program stored in the memory 3 to perform the following steps:

[0171] S11, acquiring a camera video of the driver, and obtaining image data of the driver's state time series from the video;

[0172] S12, performing a first data preprocessing on the image data, and extracting feature data from the image data after the first data preprocessing; the feature data includes: attention feature data and behavior feature data;

[0173] S13, performing secondary data preprocessing on the feature data; the secondary data preprocessing includes:

[0174] Normalizing and / or filtering the attention feature data and the behavior feature data;

[0175] S14, performing feature fusion on the feature data after the secondary data preprocessing according to the feature fusion network to obtain a fused combined feature data set;

[0176] S15. Using the combined feature data set as modeling data, a driving intention prediction model based on a convolutional long short-term memory network is constructed.

[0177] The above product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the driving intention prediction method provided by the embodiment of the present invention.

[0178] Embodiment 4

[0179] In an embodiment of the present invention, a storage medium is further provided, which may store a program suitable for execution by a processor, wherein the program is used to:

[0180] S11, acquiring a camera video of the driver, and obtaining image data of the driver's state time series from the video;

[0181] S12, performing a first data preprocessing on the image data, and extracting feature data from the image data after the first data preprocessing; the feature data includes: attention feature data and behavior feature data;

[0182] S13, performing secondary data preprocessing on the feature data; the secondary data preprocessing includes:

[0183] Normalizing and / or filtering the attention feature data and the behavior feature data;

[0184] S14, performing feature fusion on the feature data after the secondary data preprocessing according to the feature fusion network to obtain a fused combined feature data set;

[0185] S15. Using the combined feature data set as modeling data, a driving intention prediction model based on a convolutional long short-term memory network is constructed.

[0186] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0187] The above product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the methods provided by other embodiments of the present invention.

[0188] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0189] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0190] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0191] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0192] It should be understood that in the embodiments of the present application, the various embodiments and features can be combined with each other to solve the aforementioned technical problems.

[0193] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0194] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A driving intention prediction method based on a neural network, characterized in that: Includes steps: S11, acquiring a camera video of the driver, and obtaining image data of the driver's state time series from the video; S12, performing a first data preprocessing on the image data, and extracting feature data from the image data after the first data preprocessing; The characteristic data includes: attention characteristic data and behavior characteristic data; S13, performing secondary data preprocessing on the feature data; the secondary data preprocessing includes: Normalizing and / or filtering the attention feature data and the behavior feature data; S14, performing feature fusion on the feature data after the secondary data preprocessing according to the feature fusion network to obtain a fused combined feature data set; S15. Using the combined feature data set as modeling data, a driving intention prediction model based on a convolutional long short-term memory network is constructed.

2. The driving intention prediction method based on neural network according to claim 1, characterized in that: The attention feature data includes: facial key point coordinate data and head orientation coordinate data; The behavior characteristic data includes: coordinate data of key points of the upper body and coordinate data of key points of the arms.

3. The driving intention prediction method based on neural network according to claim 2 is characterized in that: The facial key point coordinate data, the upper body key point coordinate data and the arm key point coordinate data are feature extracted according to a human body posture estimation model; The head orientation coordinate data is feature extracted according to a head posture estimation model.

4. The method for predicting driving intention based on a neural network according to claim 3, characterized in that: The attention feature data and the behavior feature data are normalized and / or filtered, including: Performing pixel value normalization processing and filtering processing on the facial key point coordinate data, the upper body key point coordinate data, and the arm key point coordinate data; The head orientation coordinate data is subjected to filtering processing.

5. The method for predicting driving intention based on a neural network according to claim 4, characterized in that: The pixel value normalization process includes the formula: In the formula, x is the original pixel value, x normalized is the normalized pixel value.

6. The method for predicting driving intention based on a neural network according to claim 4, characterized in that: The filtering process comprises: Determine the window size based on the smoothing effect and sensitivity to changes; For each set of key point coordinates, starting from the current frame, select a time window containing N frames, where N is the preset window length, and for each frame in the time window, record the position coordinates of the corresponding key point; Calculate the average value of the coordinate data of each corresponding key point in the time window as the position coordinate of the key point in the current window.

7. A driving intention prediction device based on a neural network, characterized in that: include: An image data acquisition unit, used to acquire a camera video of the driver, and obtain image data of the driver's state time series from the video; A feature data extraction unit, used for performing a first data preprocessing on the image data, and extracting feature data from the image data after the first data preprocessing; The characteristic data includes: attention characteristic data and behavior characteristic data; The feature data preprocessing unit is used to perform secondary data preprocessing on the feature data; the secondary data preprocessing includes: Normalizing and / or filtering the attention feature data and the behavior feature data; A modeling data generating unit, used for performing feature fusion on the feature data after the secondary data preprocessing according to a feature fusion network to obtain a fused combined feature data set; The driving intention prediction model building unit is used to build a driving intention prediction model based on a convolutional long short-term memory network using the combined feature data set as modeling data.

8. The driving intention prediction device based on neural network according to claim 7, characterized in that: The attention feature data includes: facial key point coordinate data and head orientation coordinate data; The behavior characteristic data includes: coordinate data of key points of the upper body and coordinate data of key points of the arms.

9. A driving intention prediction device based on a neural network, characterized in that: include: Memory for storing computer programs; A processor, configured to call and execute the computer program to implement the steps of the driving intention prediction method based on a neural network as described in any one of claims 1 to 6.

10. A storage medium, characterized in that: It comprises a software program, which is suitable for a processor to execute the steps of the driving intention prediction method based on a neural network as described in any one of claims 1 to 6.

Images