Assisted driving method, device, electronic device and storage medium

Abstract

The embodiment of the present invention discloses an assisted driving method, device, electronic device and storage medium, the method comprising: obtaining an external environment video of the environment to which a target vehicle belongs, and a target user video corresponding to the target vehicle; processing the obtained videos respectively based on a pre-constructed bidirectional neural network to obtain an external environment feature sequence to be processed and a user feature sequence to be processed; inputting the above feature sequences to be processed into a pre-constructed temporal convolutional network respectively to obtain an external environment feature sequence to be fused and a user feature sequence to be fused; determining a target feature sequence corresponding to the target vehicle based on the feature sequence to be fused, so as to determine the driving intention of the target user corresponding to the target vehicle based on the target feature sequence. The technical solution of the embodiment of the present invention realizes the comprehensive consideration of the feature information inside and outside the vehicle in the process of predicting driving intention, and at the same time enhances the stability of the driving intention prediction framework.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of automobile technology, and in particular to an assisted driving method, device, electronic device, and storage medium. Background Art

[0002] In the task of predicting driving intention, the method usually adopted is data-driven algorithm, which does not take into account the intrinsic relationship between various features, resulting in the inability to accurately predict the driver's driving intention. Based on this, the driver's cognition and decision-making process can be simulated in the algorithm framework, so that the algorithm can perform closer to the driver's decision-making performance in the prediction process.

[0003] When implementing the technical solution based on the above method, the inventor found the following problems:

[0004] In the process of simulating the driver's cognition and decision-making in the algorithm framework, although the driver's driving behavior can be well explained and analyzed, the driver's subsequent driving behavior cannot be predicted, resulting in the problem that the obtained driving behavior cannot assist the driver in driving. Summary of the invention

[0005] The present invention provides a driving assistance method, device, electronic device and storage medium, which realizes comprehensive consideration of feature information inside and outside the vehicle during the driving intention prediction process, and at the same time enhances the stability of the driving intention prediction framework.

[0006] In a first aspect, an embodiment of the present invention provides a driving assistance method, the method comprising:

[0007] Acquire an external environment video of the environment to which the target vehicle belongs, and a target user video corresponding to the target vehicle;

[0008] Based on a pre-built bidirectional neural network, the external environment video and the target user video are processed respectively to obtain a to-be-processed external environment feature sequence and a to-be-processed user feature sequence;

[0009] Inputting the to-be-processed external environment feature sequence and the to-be-processed target user feature sequence into a pre-constructed temporal convolutional network respectively, to obtain the to-be-fused external environment feature sequence and the to-be-fused user feature sequence;

[0010] Based on the external environment feature sequence to be fused and the user feature sequence to be fused, a target feature sequence corresponding to the target vehicle is determined, so as to determine the driving intention of the target user corresponding to the target vehicle based on the target feature sequence, so as to assist driving based on the driving intention.

[0011] In a second aspect, an embodiment of the present invention further provides a driving assistance device, the device comprising:

[0012] A video acquisition module, used to acquire an external environment video of the environment to which the target vehicle belongs, and a target user video corresponding to the target vehicle;

[0013] A module for determining a feature sequence to be processed, used to process the external environment video and the target user video respectively based on a pre-built bidirectional neural network to obtain a feature sequence of the external environment to be processed and a feature sequence of the user to be processed;

[0014] A feature sequence determination module to be fused, used to input the external environment feature sequence to be processed and the target user feature sequence to be processed into a pre-built temporal convolutional network, respectively, to obtain the external environment feature sequence to be fused and the user feature sequence to be fused;

[0015] The target feature sequence determination module is used to determine the target feature sequence corresponding to the target vehicle based on the external environment feature sequence to be fused and the user feature sequence to be fused, so as to determine the driving intention of the target user corresponding to the target vehicle based on the target feature sequence, so as to assist driving based on the driving intention.

[0016] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0017] one or more processors;

[0018] a storage device for storing one or more programs,

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the assisted driving method as described in any one of the embodiments of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the assisted driving method as described in any one of the embodiments of the present invention.

[0021] The technical solution of the embodiment of the present invention first obtains an external environment video of the environment to which the target vehicle belongs, and a target user video corresponding to the target vehicle, so as to obtain multimodal data inside and outside the car; based on a pre-constructed bidirectional neural network, the external environment video and the target user video are processed respectively to obtain an external environment feature sequence to be processed and a user feature sequence to be processed; further, the external environment feature sequence to be processed and the target user feature sequence to be processed are respectively input into a pre-constructed temporal convolutional network to obtain an external environment feature sequence to be fused and a user feature sequence to be fused; based on the external environment feature sequence to be fused and the user feature sequence to be fused, a target feature sequence corresponding to the target vehicle is determined, so as to determine the driving intention of the target user corresponding to the vehicle based on the target feature sequence to assist driving, and realize comprehensive consideration of feature information inside and outside the vehicle in the process of driving intention prediction. At the same time, the introduction of the temporal convolutional network not only avoids the use of future features in the process of multi-source information fusion, but also improves the training depth of the model and enhances the stability of the driving intention prediction framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required to describe the embodiments. Obviously, the drawings introduced are only drawings of a part of the embodiments to be described in the present invention, rather than all the drawings. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A schematic diagram of a flow chart of an assisted driving method provided in Embodiment 1 of the present invention;

[0024] Figure 2 A schematic diagram of a flow chart of an assisted driving method provided in Embodiment 2 of the present invention;

[0025] Figure 3 The driving intention prediction framework based on the temporal convolutional network provided in the second embodiment of the present invention;

[0026] Figure 4 This is a cognitive fusion strategy framework diagram provided in Embodiment 2 of the present invention;

[0027] Figure 5 A driving intention prediction network training framework based on a temporal convolutional network and cognitive fusion strategy provided in the second embodiment of the present invention;

[0028] Figure 6 This is a structural block diagram of a driving assistance device provided by Embodiment 3 of the present invention;

[0029] Figure 7This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0031] Embodiment 1

[0032] Figure 1 This is a flow chart of an assisted driving method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case of modeling a driving intention prediction task. The method can be executed by an assisted driving device, which can be implemented in the form of software and / or hardware. The hardware can be an electronic device, such as a mobile terminal, a PC or a server.

[0033] like Figure 1 As shown, the method specifically comprises the following steps:

[0034] S110, obtaining an external environment video of the environment to which the target vehicle belongs, and a target user video corresponding to the target vehicle.

[0035] The environment to which the vehicle belongs can be an area divided by the system based on a preset length or range with the current position of the vehicle as the center. For example, a circular area determined with the current position of the vehicle as the center and a radius of 10m is the environment to which the vehicle currently belongs. Correspondingly, the external environment video is the image data collected in real time by the vehicle in its environment. These image data can be used as a reflection of the external feature information of the vehicle. Those skilled in the art should understand that the actual road traffic environment is relatively complex. Therefore, the external environment video usually includes a variety of road traffic elements, such as motor vehicles, bicycles, pedestrians, and obstacles.

[0036] In this embodiment, while acquiring the external environment video, it is also necessary to acquire the target user video. The target user video refers to the real-time collected image data containing the driver, which can be used as a reflection of the internal characteristic information of the vehicle. Through the target user video, the system can identify and determine the driving state of the driver at each moment, for example, whether the driver's hands are in the steering wheel position, which direction the driver is facing at each moment, etc.

[0037] In actual application, external environment video and target user video can be obtained using a variety of image acquisition devices or sensors deployed on the vehicle. For example, external environment video can be obtained using a camera device installed outside the vehicle, and target user video can be obtained using a monitoring system installed inside the vehicle.

[0038] It can be understood that the acquired video contains multimodal information, wherein the multimodal information is information reflecting the target vehicle, road traffic conditions and driver status from multiple dimensions. At the same time, for the subsequent driving intention prediction task, the multimodal information can serve as the data basis for the task.

[0039] S120 , based on a pre-built bidirectional neural network, the external environment video and the target user video are processed respectively to obtain a to-be-processed external environment feature sequence and a to-be-processed user feature sequence.

[0040] In this embodiment, the multimodal information in the external environment video and the target user video is expressed in the form of multiple features. For example, the external environment features may be other vehicles, pedestrians, and obstacles in the environment to which the target vehicle belongs, and the user features may be the driver's head orientation, binocular information, and hand positions. Therefore, in order to obtain these features as a basis for subsequent driving intention prediction, the external environment video and the target user video need to be processed.

[0041] It should be noted that the model prediction of driving intention depends on multiple features inside and outside the car. Therefore, the modeling of driving intention prediction is essentially a learning modeling problem for time-series video. By extracting the features of each frame in the video, a time-series feature sequence composed of the features of each frame can be obtained. Specifically in this embodiment, it can be understood that one or more features can exist in each frame of the acquired external environment video and the target user video. Therefore, after processing the above video, the external environment feature sequence and the user feature sequence corresponding to each frame in the video can be obtained respectively.

[0042] In this embodiment, there are multiple ways to process the acquired video. Specifically, a pre-built bidirectional neural network can be used to process the video. Before explaining this method, an example of using a unidirectional neural network to process the video is first given.

[0043] Exemplarily, after each frame in the video is matched with each moment, a unidirectional recurrent neural network (RNN) can be used for processing. RNN is a recursive neural network that takes sequence data as input, recurses in the direction of sequence evolution, and all nodes are connected in a chain. In the process of training the driving intention prediction model, the input at the current moment and the hidden state at the previous moment (the state information related to the vehicle at the previous moment) can be calculated at each moment, thereby generating a new hidden state and the prediction result at the current moment. Furthermore, the new hidden state continues to be transmitted backward along the time series, which can ensure that the hidden state input at each moment contains the information of all previous moments. This cyclic autoregressive structure is a good expression for time series tasks.

[0044] However, in the above example, the information in the RNN flows unidirectionally. Processing the video in this way may result in insufficient capture of contextual information (i.e., some or all of the information in the environment to which the target vehicle belongs that can affect the prediction results). Therefore, in this embodiment, in order to more fully mine the contextual information and improve the training effect of the model, a pre-built bidirectional neural network can be used to process the external environment video and the target user video.

[0045] Specifically, the bidirectional neural network consists of a forward network and a backward network. In the driving intention prediction task, the forward network transmits information backward to update the state information of the forward network at the current moment, while the backward network uses the input at the current moment to update the state information of the backward network at the past moment, thereby better mining the contextual information.

[0046] In this embodiment, after the external environment video and the target user video are processed and two sets of feature sequences are obtained respectively, they need to be further processed to realize driving intention prediction. Therefore, the obtained feature sequences are the external environment feature sequence to be processed and the user feature sequence to be processed.

[0047] S130, inputting the to-be-processed external environment feature sequence and the to-be-processed target user feature sequence into a pre-built temporal convolutional network respectively, to obtain the to-be-fused external environment feature sequence and the to-be-fused user feature sequence.

[0048] In this embodiment, in order to further process the external environment feature sequence to be processed and the user feature sequence to be processed, it is also necessary to introduce a pre-built temporal convolutional network (TCN) into the driving intention prediction task. Among them, TCN is a network structure that can process time series data, can process long sequence inputs in parallel, and can also change the parameters of the convolution kernel to control the memory size of the model. Those skilled in the art should understand that TCN not only has a more stable gradient, but also makes the model training process easier.

[0049] Specifically, after obtaining the feature sequence based on the features to be processed in each frame of the video, it is processed by TCN as output. By utilizing the parallelism of the TCN network structure, flexible perception field, fixed gradient, smaller memory training, and variable input length, the feature sequence to be fused can be output. In actual application, there are two feature sequences to be processed based on the two situations of outside and inside the car. Correspondingly, the feature sequence to be fused obtained by TCN also includes the external environment feature sequence to be fused and the user feature sequence to be fused.

[0050] In this embodiment, by introducing TCN to further process the external environment feature sequence and the user feature sequence, the driving intention prediction as a time sequence task is optimized while avoiding the use of future moment features.

[0051] S140. Determine a target feature sequence corresponding to a target vehicle based on the external environment feature sequence to be fused and the user feature sequence to be fused, so as to determine a driving intention of a target user corresponding to the target vehicle based on the target feature sequence, so as to assist driving based on the driving intention.

[0052] In this embodiment, in order to realize driving intention prediction, it is also necessary to fuse the external environment feature sequence to be fused and the user feature sequence to be fused. It can be understood that in the driving intention prediction task, feature fusion is essentially a process of fusing multimodal information that affects the prediction result. There are many specific fusion methods, such as algorithms based on Bayesian decision theory, algorithms based on sparse representation theory, and theoretical algorithms based on deep learning, etc. This embodiment will not be repeated here.

[0053] Furthermore, after fusing multiple features in the external environment feature sequence to be fused and the user feature sequence to be fused, the target feature sequence can be obtained. In the driving intention prediction task, the driver's driving intention can at least be determined based on the target feature sequence, thereby achieving the technical effect of assisted driving. For example, when the system determines that the current vehicle needs to adjust its driving direction based on the target feature sequence, it can apply a corresponding torque on the steering wheel, or feedback the corresponding direction adjustment prompt information through the vehicle system, thereby assisting the driver to adjust the current driving direction of the vehicle.

[0054] The technical solution of this embodiment is to first obtain an external environment video of the environment to which the target vehicle belongs, and a target user video corresponding to the target vehicle, so as to obtain multimodal data inside and outside the car; based on a pre-constructed bidirectional neural network, the external environment video and the target user video are processed respectively to obtain an external environment feature sequence to be processed and a user feature sequence to be processed; further, the external environment feature sequence to be processed and the target user feature sequence to be processed are respectively input into a pre-constructed temporal convolutional network to obtain an external environment feature sequence to be fused and a user feature sequence to be fused; based on the external environment feature sequence to be fused and the user feature sequence to be fused, a target feature sequence corresponding to the target vehicle is determined, so as to determine the driving intention of the target user corresponding to the vehicle based on the target feature sequence to assist driving, and realize comprehensive consideration of feature information inside and outside the vehicle in the process of driving intention prediction. At the same time, the introduction of the temporal convolutional network not only avoids the use of future features in the process of multi-source information fusion, but also improves the training depth of the model and enhances the stability of the driving intention prediction framework.

[0055] Embodiment 2

[0056] Figure 2 A flowchart of an assisted driving method provided in the second embodiment of the present invention, based on the aforementioned embodiment, the features of multiple sources inside and outside the vehicle can be obtained based on visual sensors and image acquisition devices; LSTM can be used to fully utilize context information to perform time series prediction tasks, and at the same time, LSTM and CRF are combined to fully consider the constraints between each moment in the process of driving intention prediction; the introduction of a hole convolutional network reduces the memory consumption in the model training process and realizes the parallel optimization of the model. Furthermore, a time delay is added in the process of processing the features outside the vehicle, so that the model has higher rationality and interpretability at the cognitive level, thereby constructing a driving intention prediction network driven by cognitive priors and data bidirectionally. The specific implementation method can refer to the technical solution of this embodiment. Among them, the technical terms that are the same as or corresponding to the above embodiments are not repeated here.

[0057] like Figure 2 As shown, the method specifically comprises the following steps:

[0058] S210: Acquire an external environment video of the environment to which the target vehicle belongs, and a target user video corresponding to the target vehicle.

[0059] Optionally, an external environment video of the target vehicle's surroundings is collected based on a visual sensor installed on the target vehicle; a user video of the target user is collected based on an image acquisition device installed on the target vehicle, and vehicle attribute information of the target vehicle is collected based on a vehicle attribute information acquisition sensor; and the target user video is determined based on the user video and the vehicle attribute information.

[0060] In actual driving scenarios, when a driver controls a car, he or she first needs to receive visual and sound information about the surrounding environment through eyes and ears. The core of this is the visual information received by the driver's eyes. The driver uses this visual information to perceive the environmental information around the vehicle, such as whether there are vehicles or pedestrians in front, whether there are other vehicles on the side that are overtaking, etc. In response to this external environmental information, the driver will make corresponding driving decisions based on experience, such as whether to brake to slow down, whether to speed up to overtake, whether to stop changing lanes to allow the vehicle behind to overtake smoothly, etc.

[0061] Based on this, in the driving intention prediction task, in order to simulate the process of the driver receiving visual information and making driving decisions, visual sensors can be used to collect external environment videos. As the direct source of information for the machine vision system, the visual sensor is mainly composed of one or two graphic sensors, and can also be equipped with a light projector and other auxiliary equipment. It can be understood that the main function of the visual sensor is to obtain the original video images that need to be processed by the machine vision system. In the driving intention prediction task, the acquired external environment video can include a variety of road traffic elements, such as other vehicles and pedestrians.

[0062] Similar to the acquisition method of the external environment video, the target user video can be acquired based on the image acquisition device. At the same time, in order to obtain more comprehensive information inside the vehicle from multiple dimensions, it is also necessary to use sensors to collect vehicle attribute information, such as the vehicle's driving speed, acceleration, steering wheel steering angle, etc.

[0063] S220. Process each video frame of the external environment video based on the long short-term memory network to obtain the environment video features corresponding to each video frame; and process each video frame of the target user video based on the long short-term memory network to obtain the user video features corresponding to each video frame.

[0064] Among them, the bidirectional neural network includes the Long Short-Term Memory network. LSTM, as a time recurrent neural network, is specially designed to solve the long-term dependency problem of RNN. LSTM improves RNN by introducing the settings of forget gate, input gate and output gate, which alleviates the problem of gradient disappearance. It not only makes model training easier, but also can achieve better model performance than RNN in driving intention prediction tasks. It can be understood that the object processed by LSTM is the video frames extracted from the external environment video and the target user video.

[0065] Specifically, when using LSTM for model training, the forget gate is mainly responsible for the selective retention and forgetting of the information at the previous moment, that is, the useless information at the current moment is forgotten, while the useful information at the current moment continues to be retained; the input gate determines which new information is selected from the input at the current moment to be added to the state information at the current moment, and the new state information at the current moment can be obtained by combining the outputs of the forget gate and the input gate; the output gate is used to complete the update of the hidden layer state at the current moment and use it as the output at the next moment. After obtaining the LSTM forward propagation result, the LSTM model can be trained by defining the loss function and using the backpropagation algorithm and the gradient descent algorithm. By introducing the gate control mechanism, it is less likely for long-term dependent tasks to have gradient multiplication close to 0 during the gradient backpropagation process, which can better alleviate the gradient vanishing problem. Compared with the traditional RNN model training, the use of LSTM can achieve better model training results.

[0066] S230 , processing the environment video features and the user video features of each video frame based on the conditional random field to obtain a sequence of external environment features to be processed and a sequence of user features to be processed.

[0067] In this embodiment, although context information can be fully utilized to perform time series prediction tasks based on LSTM, the prediction results at each moment are only determined by the state information at the current moment and are independent of the prediction results at other moments. In other words, LSTM will output an independent prediction result at each moment. However, in the driving intention prediction task, the change of driving intention in adjacent close time is often continuous. For example, during driving, the driver will hardly have a situation where the intention to turn left at the previous moment suddenly changes to the intention to turn right at the current moment, and then changes back to the intention to turn left at the next moment. Therefore, using only LSTM in the modeling process cannot accurately describe the constraint relationship between each moment.

[0068] Based on this, this embodiment introduces a conditional random field (CRF) on the basis of LSTM. CRF is a discriminative probability model, which is often used to mark or analyze sequence data. It combines the characteristics of the maximum entropy model and the hidden Markov model and can be used as an undirected graph model. By combining LSTM and CRF, the video frames in the external environment video and the target user video can be processed under the premise of fully considering the constraint relationship at each moment, and then the corresponding feature sequence to be processed is obtained.

[0069] Specifically, compared to LSTM, which concatenates the best prediction results at each moment, the goal of CRF is to screen out the prediction results that maximize the probability of the entire sequence. This method fully considers the constraints between each moment. It can be understood that through these constraints, the results of the entire sequence prediction errors will be greatly reduced. The loss function in CRF includes the emission score and the transition score, where the emission score comes from the output result of the LSTM in front of the model, and the transition score represents the probability of transition between each category. In this embodiment, a transition matrix can be used to describe the probability of mutual conversion between all categories.

[0070] In this embodiment, since LSTM can make full use of contextual information for time series modeling, conditional random fields can effectively establish the constraint relationship between the prediction results at each moment, thereby obtaining a better global prediction result. Therefore, the LSTM-CRF network constructed by combining the two can give full play to the advantages of the two models. At the same time, this embodiment uses a bidirectional LSTM. In the driving intention prediction task, the forward LSTM transmits information from front to back, updates the state information of the forward network at the current moment, and the reverse LSTM uses the input at the current moment to update the state information of the reverse network at the past moment, which is equivalent to using the current moment's driving information inside and outside the car to update the driving state of the driver in the past, so that the context information is more fully mined, thereby obtaining a better prediction effect relative to the unidirectional LSTM-CRF.

[0071] S240: Input the to-be-processed external environment feature sequence and the to-be-processed target user feature sequence into a pre-built temporal convolutional network respectively, to obtain the to-be-fused external environment feature sequence and the to-be-fused user feature sequence.

[0072] Optionally, the external environment feature sequence to be processed and the target user feature sequence to be processed are processed respectively based on the dilation factor in the pre-constructed temporal convolutional network to obtain the external environment feature sequence to be fused and the user feature sequence to be fused.

[0073] In this embodiment, after determining the external environment feature sequence to be processed and the target user feature sequence to be processed, in order to reduce memory consumption during model training and achieve parallel optimization during model training, it is also necessary to further process the above-mentioned feature sequence to be processed using a temporal convolutional network.

[0074] Specifically, the dilated convolution improves the standard convolution by introducing interval sampling. Unlike the standard convolution, which performs convolution on the features of the current moment and the adjacent moments each time, the interval sampling introduces a dilation factor d. Instead of convolving the features of the adjacent moments, it convolves the features d time units away from the current moment.

[0075] In this embodiment, the calculation formula of the dilated convolution kernel is: Where k is the size of the convolution kernel, f is the parameter of the convolution kernel, and md·i is the i-th point in the past collected by the dilated convolution. The adjustable dilation factor can be used to obtain a flexible convolution receptive field, that is, a larger receptive field is obtained when the dilation factor is larger, and a smaller receptive field is obtained when the dilation factor is smaller. It should be noted that when the dilation factor is equal to 1, the dilated convolution degenerates into an ordinary standard convolution. In the stacking process of dilated convolution, the dilation factor of the shallow convolution part of the network is set to a smaller value, while the dilation factor of the deep convolution part of the network is set to a larger value. This ensures that the shallow network captures the feature information at each moment, and the deep network obtains a larger receptive field through layer-by-layer feature extraction, thereby completing the capture of long-term dependent information. In actual applications, the setting of the dilation factor d usually increases in the form of an exponential of 2 as the number of network layers increases.

[0076] From the above description, it can be determined that since the driving intention prediction task is a real-time time series prediction problem, future information cannot be used when the network makes predictions. However, the symmetry of standard convolution and dilated convolution may extract features after the current moment during operation, resulting in information leakage. Therefore, see Figure 3 From the driving intention prediction framework, it can be seen that this embodiment further introduces causal convolution to cancel the symmetry of traditional convolution operations and avoid the use of future features. That is to say, only the features of historical moments and the current moment are used for convolution operations. By fusing the internal and external features of the vehicle at historical moments and the current moment, the corresponding output can be obtained. It should be noted that this embodiment not only introduces void convolution and causal convolution to model time series features, but also introduces residual connections into the temporal convolutional network, further improving the training depth of the network.

[0077] Optionally, according to a preset interval period, multiple environment feature sequences to be used are extracted from the external environment feature sequence to be processed, and multiple user feature sequences to be used are extracted from the target user feature sequence to be processed; the multiple environment feature sequences to be used are input into the temporal convolutional network, and the environment high-order feature sequences corresponding to the multiple environment sequences to be used are output, and the environment high-order feature sequences are used as the external environment feature sequences to be fused; the multiple user feature sequences to be used are input into the temporal convolutional network, and the user high-order feature sequences corresponding to the user feature sequences to be used are output, and the user high-order feature sequences are used as the user feature sequences to be fused.

[0078] Specifically, in a driving scenario, by using a visual sensor to collect video information inside and outside the vehicle, a feature sequence with the time difference between frames can be obtained. Furthermore, to express the features inside and outside the vehicle collected by the sensor in mathematical form, the time difference between adjacent frames can be set as t f After collecting the interior and exterior features of the vehicle at a total of T moments, the interior features can be expressed as There are T features in total. The high-order features obtained by sending the feature sequence of these T moments to the temporal convolutional network can be expressed as There are T features in total, and the specific formula is: Among them, TCN i represents the temporal convolutional network module for processing the in-car features. Correspondingly, the out-of-car features can be expressed as And there are also T features in total, and the specific formula is Among them, TCN o Represents the temporal convolutional network module for processing vehicle external features. After fusing the obtained high-order features using independent temporal convolutional network modules, the final prediction result (i.e., driving intention prediction) can be obtained, represented by y t Indicates that there are T in total, and the specific formula is y t =F(M t ,N t ), where F represents the fusion of internal and external features of the vehicle and the final prediction.

[0079] S250, determining a target feature sequence corresponding to the target vehicle according to the external environment feature sequence to be fused, the user feature sequence to be fused, and the preset cognitive process time difference.

[0080] In the driving intention prediction algorithm, the features of the inside and outside of the car can be directly spliced ​​and fused, but this data-driven data fusion process does not take into account the driver's cognitive process of the external environment. Specifically, the driver's perception of the external environment has a certain reaction process. The driver's reaction and behavior at the current moment are actually the driving decisions made to the external environment at a certain moment in the past. Therefore, see Figure 4It can be seen that this embodiment integrates the current in-vehicle features with the in-vehicle features at a certain moment in the past, so that the algorithm can better imitate the driver's cognitive process based on the in-vehicle and in-vehicle features.

[0081] In this embodiment, the driver's cognitive process of the external environment is introduced into the model in the form of time difference, and the time difference can be set as t c , which represents the time it takes for the driver to see the external environment and make corresponding decisions. Based on this, the above high-order feature expression and fusion formula of the vehicle exterior can be optimized as follows:

[0082] By delaying the exterior features on the time axis, it is equivalent to adding a time delay module after the temporal convolutional network of the exterior features. Through the time delay module, in the process of integrating the interior and exterior features, the interior features are actually integrated with the exterior features at a certain moment in the past, which fully considers the driver's cognitive process of the external environment, making the model more reasonable and explainable at the cognitive level, and thus establishing the following Figure 5 The driving intention prediction network driven bidirectionally by cognitive priors and data is shown.

[0083] S260: Input the target feature sequence into a pre-trained driving intention prediction model to obtain a driving intention corresponding to the target feature sequence, so as to assist driving based on the driving intention.

[0084] In this embodiment, after obtaining the features outside the vehicle based on the external environment video and the features inside the vehicle based on the target user video, the target feature sequence can be determined using a bidirectional neural network and a temporal convolutional network. Further, the target feature sequence is input into a pre-trained driving intention prediction model to obtain the driving intention at each moment, thereby assisting the driver in driving the vehicle. Exemplarily, when the system determines that the vehicle needs to turn at the current moment based on the feature information inside and outside the vehicle (i.e., obtaining the turning driving intention), the driver's driving situation can be assisted by outputting torque to the steering wheel and displaying prompt information on the vehicle system.

[0085] The technical solution of this embodiment can obtain multi-source features inside and outside the vehicle based on visual sensors and image acquisition devices; LSTM can make full use of context information to perform time series prediction tasks. At the same time, LSTM and CRF are combined to fully consider the constraint relationship between each moment in the driving intention prediction process; the introduction of the atrous convolutional network reduces the memory consumption in the model training process and realizes the parallel optimization of the model. Furthermore, time delay is added in the process of processing the features outside the vehicle, so that the model has higher rationality and explainability at the cognitive level, thereby constructing a driving intention prediction network driven by cognitive priors and data in both directions.

[0086] Embodiment 3

[0087] Figure 6 This is a structural block diagram of an assisted driving device provided in the third embodiment of the present invention, which can execute the assisted driving method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. Figure 6 As shown, the device specifically includes: a video acquisition module 310 , a feature sequence to be processed determining module 320 , a feature sequence to be fused determining module 330 and a target feature sequence determining module 340 .

[0088] The video acquisition module 310 is used to acquire the external environment video of the environment to which the target vehicle belongs, and the target user video corresponding to the target vehicle.

[0089] The module 320 for determining the feature sequence to be processed is used to process the external environment video and the target user video respectively based on a pre-built bidirectional neural network to obtain the external environment feature sequence to be processed and the user feature sequence to be processed.

[0090] The feature sequence to be fused determining module 330 is used to input the external environment feature sequence to be processed and the target user feature sequence to be processed into a pre-built temporal convolutional network respectively to obtain the external environment feature sequence to be fused and the user feature sequence to be fused.

[0091] The target feature sequence determination module 340 is used to determine the target feature sequence corresponding to the target vehicle based on the external environment feature sequence to be fused and the user feature sequence to be fused, so as to determine the driving intention of the target user corresponding to the target vehicle based on the target feature sequence, so as to assist driving based on the driving intention.

[0092] On the basis of the above technical solutions, the video acquisition module 310 includes an external environment video acquisition unit and a target user video acquisition unit.

[0093] The external environment video acquisition unit is used to acquire the external environment video of the surrounding environment of the target vehicle based on the visual sensor arranged on the target vehicle; wherein the external environment video includes other vehicles and pedestrians.

[0094] The target user video acquisition unit is used to acquire a user video of the target user based on an image acquisition device arranged on the target vehicle, and to acquire vehicle attribute information of the target vehicle based on a vehicle attribute information acquisition sensor; and to determine the target user video based on the user video and the vehicle attribute information.

[0095] On the basis of the above technical solutions, the bidirectional neural network includes a long short-term memory network and a conditional random field, and the module for determining the feature sequence to be processed 320 includes a video frame processing unit and a unit for determining the feature sequence to be processed.

[0096] A video frame processing unit is used to process each video frame of the external environment video based on the long short-term memory network to obtain the environment video features corresponding to each video frame; and to process each video frame of the target user video based on the long short-term memory network to obtain the user video features corresponding to each video frame.

[0097] The unit for determining a feature sequence to be processed is used to process the environment video features and user video features of each video frame based on the conditional random field to obtain the external environment feature sequence to be processed and the user feature sequence to be processed.

[0098] Optionally, the module 330 for determining the feature sequence to be fused is further used to process the external environment feature sequence to be processed and the target user feature sequence to be processed respectively based on the dilation factor in the pre-constructed temporal convolutional network to obtain the external environment feature sequence to be fused and the user feature sequence to be fused.

[0099] On the basis of the above technical solutions, the module 330 for determining the feature sequence to be fused includes a feature sequence extraction unit, a unit for determining the external environment feature sequence to be fused, and a unit for determining the user feature sequence to be fused.

[0100] The feature sequence extraction unit is used to extract a plurality of to-be-used environment feature sequences from the to-be-processed external environment feature sequence and to extract a plurality of to-be-used user feature sequences from the to-be-processed target user feature sequence according to a preset interval period.

[0101] The unit for determining the external environment feature sequence to be fused is used to input the multiple environment feature sequences to be used into the temporal convolutional network, output the environment high-order feature sequences corresponding to the multiple environment sequences to be used, and use the environment high-order feature sequences as the external environment feature sequence to be fused.

[0102] The unit for determining the user feature sequence to be fused is used to input the multiple user feature sequences to be used into the temporal convolutional network, output a user high-order feature sequence corresponding to the user feature sequence to be used, and use the user high-order feature sequence as the user feature sequence to be fused.

[0103] Optionally, the target feature sequence determination module 340 is further used to determine a target feature sequence corresponding to the target vehicle according to the external environment feature sequence to be fused, the user feature sequence to be fused, and a preset cognitive process time difference.

[0104] Based on the above technical solutions, the auxiliary driving device also includes a driving intention determination module.

[0105] The driving intention determination module is used to input the target feature sequence into a pre-trained driving intention prediction model to obtain a driving intention corresponding to the target feature sequence, so as to assist driving based on the driving intention.

[0106] The technical solution provided in this embodiment first obtains an external environment video of the environment to which the target vehicle belongs, and a target user video corresponding to the target vehicle, so as to obtain multimodal data inside and outside the car; based on a pre-constructed bidirectional neural network, the external environment video and the target user video are processed respectively to obtain an external environment feature sequence to be processed and a user feature sequence to be processed; further, the external environment feature sequence to be processed and the target user feature sequence to be processed are respectively input into a pre-constructed temporal convolutional network to obtain an external environment feature sequence to be fused and a user feature sequence to be fused; based on the external environment feature sequence to be fused and the user feature sequence to be fused, a target feature sequence corresponding to the target vehicle is determined, so as to determine the driving intention of the target user corresponding to the vehicle based on the target feature sequence to assist driving, and realizes comprehensive consideration of feature information inside and outside the vehicle in the process of driving intention prediction. At the same time, the introduction of the temporal convolutional network not only avoids the use of future features in the process of multi-source information fusion, but also improves the training depth of the model and enhances the stability of the driving intention prediction framework.

[0107] The assisted driving device provided in the embodiment of the present invention can execute the assisted driving method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0108] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention.

[0109] Embodiment 4

[0110] Figure 7 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. Figure 7 A block diagram of an exemplary electronic device 40 suitable for implementing exemplary embodiments of the present invention is shown. Figure 7 The electronic device 40 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0111] like Figure 7As shown, the electronic device 40 is in the form of a general computing device. The components of the electronic device 40 may include but are not limited to: one or more processors or processing units 401, a system memory 402, and a bus 403 connecting different system components (including the system memory 402 and the processing unit 401).

[0112] Bus 403 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0113] The electronic device 40 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 40, including volatile and non-volatile media, removable and non-removable media.

[0114] System memory 402 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 404 and / or cache memory 405. Electronic device 40 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 406 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 not shown, usually called a "hard drive"). Although Figure 7 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 403 via one or more data medium interfaces. Memory 402 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of various embodiments of the present invention.

[0115] A program / utility 408 having a set (at least one) of program modules 407 may be stored, for example, in memory 402, such program modules 407 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 407 generally perform the functions and / or methods of the embodiments described herein.

[0116] The electronic device 40 may also communicate with one or more external devices 409 (e.g., keyboards, pointing devices, displays 410, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 40, and / or may communicate with any device that enables the electronic device 40 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 411. Furthermore, the electronic device 40 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 412. As shown, the network adapter 412 communicates with other modules of the electronic device 40 via the bus 403. It should be understood that although Figure 7 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 40, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0117] The processing unit 401 executes various functional applications and data processing by running the programs stored in the system memory 402, such as implementing the assisted driving method provided in the embodiment of the present invention.

[0118] Embodiment 5

[0119] Embodiment 5 of the present invention further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions are used to execute an assisted driving method when executed by a computer processor.

[0120] The method includes:

[0121] Acquire an external environment video of the environment to which the target vehicle belongs, and a target user video corresponding to the target vehicle;

[0122] Based on a pre-built bidirectional neural network, the external environment video and the target user video are processed respectively to obtain a to-be-processed external environment feature sequence and a to-be-processed user feature sequence;

[0123] Inputting the to-be-processed external environment feature sequence and the to-be-processed target user feature sequence into a pre-constructed temporal convolutional network respectively, to obtain the to-be-fused external environment feature sequence and the to-be-fused user feature sequence;

[0124] Based on the external environment feature sequence to be fused and the user feature sequence to be fused, a target feature sequence corresponding to the target vehicle is determined, so as to determine the driving intention of the target user corresponding to the target vehicle based on the target feature sequence, so as to assist driving based on the driving intention.

[0125] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0126] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, in which computer-readable project code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0127] Project code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0128] Computer project code for performing the operation of embodiments of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The project code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0129] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A driving assistance method, characterized in that: include: Acquire an external environment video of the environment to which the target vehicle belongs, and a target user video corresponding to the target vehicle; Based on a pre-built bidirectional neural network, the external environment video and the target user video are processed respectively to obtain a to-be-processed external environment feature sequence and a to-be-processed user feature sequence; Inputting the to-be-processed external environment feature sequence and the to-be-processed target user feature sequence into a pre-constructed temporal convolutional network respectively, to obtain the to-be-fused external environment feature sequence and the to-be-fused user feature sequence; Based on the external environment feature sequence to be fused and the user feature sequence to be fused, a target feature sequence corresponding to the target vehicle is determined, so as to determine the driving intention of the target user corresponding to the target vehicle based on the target feature sequence, so as to assist driving based on the driving intention.

2. The method according to claim 1, characterized in that The step of acquiring an external environment video of an environment to which a target vehicle belongs and a target user video corresponding to the target vehicle includes: Collecting an external environment video of the surrounding environment of the target vehicle based on a visual sensor disposed on the target vehicle; wherein the external environment video includes other vehicles and pedestrians; Collecting a user video of a target user based on an image acquisition device disposed on the target vehicle, and collecting vehicle attribute information of the target vehicle based on a vehicle attribute information acquisition sensor; The target user video is determined according to the user video and the vehicle attribute information.

3. The method according to claim 1, characterized in that The bidirectional neural network includes a long short-term memory network and a conditional random field. The pre-built bidirectional neural network processes the external environment video and the target user video respectively to obtain a to-be-processed external environment feature sequence and a to-be-processed user feature sequence, including: Processing each video frame of the external environment video based on the long short-term memory network to obtain an environment video feature corresponding to each video frame; and processing each video frame of the target user video based on the long short-term memory network to obtain a user video feature corresponding to each video frame; Based on the conditional random field, the environmental video features and the user video features of each video frame are processed respectively to obtain the external environmental feature sequence to be processed and the user feature sequence to be processed.

4. The method according to claim 1, characterized in that The step of inputting the to-be-processed external environment feature sequence and the to-be-processed target user feature sequence into a pre-built temporal convolutional network to obtain the to-be-fused external environment feature sequence and the to-be-fused user feature sequence comprises: Based on the expansion factor in the pre-constructed temporal convolutional network, the external environment feature sequence to be processed and the target user feature sequence to be processed are processed respectively to obtain the external environment feature sequence to be fused and the user feature sequence to be fused.

5. The method according to claim 1, characterized in that The step of inputting the to-be-processed external environment feature sequence and the to-be-processed target user feature sequence into a pre-built temporal convolutional network to obtain the to-be-fused external environment feature sequence and the to-be-fused user feature sequence comprises: According to a preset interval period, extracting a plurality of environment feature sequences to be used from the external environment feature sequences to be processed, and extracting a plurality of user feature sequences to be used from the target user feature sequences to be processed; Input the multiple environmental feature sequences to be used into a temporal convolutional network, output environmental high-order feature sequences corresponding to the multiple environmental sequences to be used, and use the environmental high-order feature sequences as external environmental feature sequences to be fused; The multiple user feature sequences to be used are input into a temporal convolutional network, a user high-order feature sequence corresponding to the user feature sequence to be used is output, and the user high-order feature sequence is used as a user feature sequence to be fused.

6. The method according to claim 1, characterized in that The step of determining a target feature sequence corresponding to the target vehicle based on the external environment feature sequence to be fused and the user feature sequence to be fused includes: A target feature sequence corresponding to the target vehicle is determined according to the external environment feature sequence to be fused, the user feature sequence to be fused, and a preset cognitive process time difference.

7. The method according to claim 1, characterized in that After obtaining the target feature sequence, the method further includes: The target feature sequence is input into a pre-trained driving intention prediction model to obtain a driving intention corresponding to the target feature sequence, so as to assist driving based on the driving intention.

8. A driving assistance device, characterized in that: include: A video acquisition module, used to acquire an external environment video of the environment to which the target vehicle belongs, and a target user video corresponding to the target vehicle; A module for determining a feature sequence to be processed, used to process the external environment video and the target user video respectively based on a pre-built bidirectional neural network to obtain a feature sequence of the external environment to be processed and a feature sequence of the user to be processed; A feature sequence determination module to be fused, used to input the external environment feature sequence to be processed and the target user feature sequence to be processed into a pre-built temporal convolutional network, respectively, to obtain the external environment feature sequence to be fused and the user feature sequence to be fused; The target feature sequence determination module is used to determine the target feature sequence corresponding to the target vehicle based on the external environment feature sequence to be fused and the user feature sequence to be fused, so as to determine the driving intention of the target user corresponding to the target vehicle based on the target feature sequence, so as to assist driving based on the driving intention.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the assisted driving method as described in any one of claims 1-7.

10. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to execute the assisted driving method as described in any one of claims 1 to 7 when executed by a computer processor.

Images