A lane-changing prediction method, device and computer equipment

Abstract

The application belongs to the technical field of intelligent driving, and particularly relates to a lane changing prediction method and device and computer equipment. The method is realized based on an in-vehicle camera, an out-vehicle camera and a speed sensor, and comprises the following steps: obtaining the state of a to-be-predicted vehicle, and extracting in-vehicle video and out-vehicle environment video; pre-processing the video to obtain an image frame sequence; inputting the image frame sequence into a pre-established and trained prediction model to obtain a predicted lane changing result; the prediction model comprises: a feature extraction module, which is used for processing the in-vehicle RGB image frame sequence to obtain an in-vehicle face image feature vector, and processing the out-vehicle environment image frame sequence to obtain an out-vehicle environment feature vector; a time sequence feature fusion module, which is used for fusing the in-vehicle face image feature vector and the out-vehicle environment feature vector at the same time, and completing time sequence modeling; and a prediction head, which is used for predicting according to the time sequence modeling and the state of the vehicle to obtain a prediction result of whether to change lanes and how to change lanes.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent driving, and particularly relates to a lane changing prediction method and device and a computer device. BACKGROUND

[0002] Driving behavior prediction technology is a key component in the field of intelligent transportation systems and autonomous driving. With the rapid development of big data, machine learning, and deep learning technologies, this field has made significant progress.

[0003] The construction of prediction models is based on a large amount of driving data, including speed, acceleration, steering angle, etc. By analyzing these data, the model can learn the patterns and rules of driving behavior. Early prediction models mainly rely on traditional machine learning algorithms, such as decision trees, support vector machines, and random forests. However, as the amount of data increases, the performance of these algorithms is gradually limited, prompting researchers to begin using deep learning technologies, such as convolutional neural networks, recurrent neural networks, and long short-term memory networks. These models can handle complex spatio-temporal data and capture long-term dependencies in driving behavior.

[0004] With the advancement of technology, many studies have adopted multi-modal data to model driving behavior, such as integrating visual data, sensor data, GPS data, and other data from different sources into the prediction model, which can provide more comprehensive driving environment information, thereby improving the accuracy of prediction. However, simple data fusion strategies, while effective, may not be able to describe the driver's perception of the external world, which is crucial for understanding the driver's intent. Some studies have attempted to use optical flow extraction and 3D convolutional neural networks to handle spatial and temporal correlations, but these methods are limited in their application in vehicle systems due to high computational cost and large memory requirements.

[0005] To improve the accuracy of driving behavior prediction, researchers have begun to attempt to model driving behavior from the driver's perspective. These methods are theoretically attractive, but the noise problem of physiological signals has affected their actual performance. To overcome this problem, some studies have combined driver characteristics with vehicle dynamics and scene information, proposing a state update unit. This unit selectively introduces environmental information at each time step based on a multi-head attention mechanism. However, this approach may increase the complexity of the model, leading to increased computational cost and the need for more training data to effectively learn the attention weights. Finally, by passing the environmental state, driver state, and their concatenation to a classifier, behavior probabilities can be generated and jointly predicted, thereby improving the accuracy and reliability of the prediction. At the same time, the traffic environment is dynamic and may have many unpredictable external disturbances, such as sudden events or atypical behavior of other drivers, which can affect the stability and robustness of the model. SUMMARY

[0006] The present application aims to overcome the defects of the prior art and proposes a lane changing prediction method, device and electronic equipment.

[0007] In order to achieve the above-mentioned purpose, the present application provides a lane changing prediction method, which is realized based on an in-vehicle camera, an out-of-vehicle camera and a speed sensor, comprising:

[0008] Obtaining the state of a vehicle to be predicted, extracting in-vehicle video and out-of-vehicle environment video;

[0009] Preprocessing the video to obtain an image frame sequence;

[0010] Inputting the image frame sequence into a prediction model established and trained in advance to obtain a predicted lane changing result;

[0011] The prediction model comprises a feature extraction module, a time sequence feature fusion module and a prediction head, wherein,

[0012] The feature extraction module is configured to process the in-vehicle RGB image frame sequence to obtain an in-vehicle face image feature vector, and is further configured to process the out-of-vehicle environment image frame sequence to obtain an out-of-vehicle environment feature vector;

[0013] The time sequence feature fusion module is configured to fuse the in-vehicle face image feature vector and the out-of-vehicle environment feature vector at the same time, and complete time sequence modeling;

[0014] The prediction head is configured to predict according to the time sequence modeling and the state of the vehicle to obtain the prediction result of whether to change lanes and the lane changing.

[0015] Preferably, the frame rate of the in-vehicle camera is 25fps, and the frame rate of the out-of-vehicle camera is 30fps.

[0016] Preferably, the state of the vehicle to be predicted comprises vehicle speed, x coordinate, y coordinate and time stamp.

[0017] Preferably, the preprocessing comprises:

[0018] Randomly sampling the in-vehicle video and the out-of-vehicle environment video to obtain the same number of key frames, and adjusting the frame size to obtain the image frame sequence.

[0019] Preferably, the feature extraction module comprises an in-vehicle feature extraction unit and an out-of-vehicle feature extraction unit, wherein,

[0020] The in-vehicle feature extraction unit adopts a Selfnet network, increases the number of channels through convolution operation and average pooling, and then uses global average pooling, and through view operation, each image becomes an in-vehicle face image feature vector with fixed length;

[0021] The off-vehicle feature extraction unit adopts a ResNet18 network to obtain a one-dimensional off-vehicle environment feature vector.

[0022] Preferably, the time sequence feature fusion module IoTM comprises a GRU layer and a multi-head attention mechanism; wherein,

[0023] The GRU layer comprises a first GRU layer for processing the in-vehicle face image feature vector F_in(t) at time t, a second GRU layer for processing the off-vehicle environment feature vector F_out(t) at time t, and a fusion feature vector H_in(t) at time t is obtained by further fusion.

[0024] The multi-head attention mechanism is used to query the off-vehicle environment feature vector F_out(t-3) at time t-3, and the fusion feature vector H_in(t) at time t and the fusion feature vector H_in(t-5) at time t-5 are connected by a residual connection to obtain the fusion feature vector H_in(t+1) at time t+1.

[0025] Preferably, the prediction head maps the in-vehicle face image feature vector to 64 dimensions, and the stacking of the fusion feature vector and the vehicle state in the last dimension is mapped to 64 dimensions through a linear layer, and then mapped to the probability of different categories through a linear layer, and the category corresponding to the maximum probability is selected as the lane change prediction result, and the lane change prediction result includes straight (i.e. non-lane change), left turn, right turn, left lane change and right lane change.

[0026] Preferably, the method comprises a training step of the prediction model, comprising:

[0027] A training set is established, which comprises pre-processed vehicle-mounted video and off-vehicle environment video and vehicle speed information;

[0028] The data of the training set is input into the prediction model, the loss function is calculated, and the training is performed through gradient backpropagation until the prediction model meeting the training requirements is obtained.

[0029] In a second aspect, the present application provides a lane change prediction device, which is realized based on an in-vehicle camera, an off-vehicle camera and a speed sensor mounted on a vehicle, and the device comprises:

[0030] An acquisition module is configured to acquire the state of a vehicle to be predicted, and extract in-vehicle video and off-vehicle environment video;

[0031] A preprocessing module is configured to pre-process the video to obtain an image frame sequence;

[0032] A prediction output module is configured to input the image frame sequence into a prediction model that is pre-established and trained to obtain a predicted lane change result.

[0033] The prediction model comprises a feature extraction module, a time sequence feature fusion module and a prediction head, wherein,

[0034] The feature extraction module is configured to process the vehicle-mounted RGB image frame sequence to obtain an in-vehicle face image feature vector, and is further configured to process the out-of-vehicle environment image frame sequence to obtain an out-of-vehicle environment feature vector.

[0035] The time sequence feature fusion module is configured to fuse the in-vehicle face image feature vector and the out-of-vehicle environment feature vector at the same time, and complete time sequence modeling.

[0036] The prediction head is configured to predict according to the time sequence modeling and the state of the vehicle to obtain the prediction result of whether to change lanes and the change lane.

[0037] In a third aspect, the present application provides a computer device comprising a memory, a processor and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0038] Compared with the prior art, the present application has the following advantages:

[0039] 1. The present application proposes a new time sequence feature fusion module (IOTM) to fuse the in-vehicle image face feature and historical information at time t and the out-of-vehicle environment semantic feature; the IOTM is used to model the in-vehicle and out-of-vehicle data in time sequence, and the in-vehicle feature with fused environment feature, the out-of-vehicle environment feature and their combination are passed through the prediction head to obtain the classification of driving behavior, effectively improving the speed and accuracy of driving behavior classification.

[0040] 2. The feature extraction module used in the present application focuses on the driver's perception and cognition of the external environment, which is crucial for accurately understanding and predicting the driver's driving intention. By combining advanced sensor technology and data processing algorithms, this module can efficiently acquire and analyze the real-time feedback of the driver's perception of the external environment, covering visual, auditory and other sensory inputs. At the same time, the module models the driver's internal cognitive process through deep learning technology, seamlessly integrating it with the dynamic changes of the external environment. This integration not only significantly improves the stability and robustness of the prediction model in complex driving scenarios, effectively dealing with various unexpected situations, but also optimizes resource utilization, maintaining low computational cost and memory requirements, suitable for real-time processing requirements of vehicle-mounted systems. The design of the module ensures high precision while reducing dependence on hardware resources, ensuring efficient operation in embedded devices and providing more reliable decision support for intelligent driving systems. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a lane change prediction model schematic diagram;

[0042] Figure 2 is a schematic diagram of a time sequence feature fusion module IOTM;

[0043] Figure 3 is a whole flow chart of lane change prediction. DETAILED DESCRIPTION

[0044] The application provides a lane change prediction method, comprising the following steps: acquiring vehicle-mounted video and off-vehicle environment video; performing random sampling processing on the video to obtain an image frame sequence; processing the vehicle-mounted RGB image sequence by using a pre-trained image processing model to obtain an image feature vector; processing the in-vehicle image sequence by using a pre-trained mobilefacenet model to obtain an in-vehicle (facial) image feature vector; processing the off-vehicle environment image sequence by using a pre-trained image processing model to obtain an image feature vector; processing the off-vehicle environment (optical flow / semantic segmentation) image sequence by using a pre-trained ResNet-18 model to obtain an off-vehicle environment feature vector; a new time sequence feature fusion module (Inside-features and Outside-features Temporal Merge) IOTM is proposed to fuse the in-vehicle image facial feature and historical information and the off-vehicle environment semantic feature at a t time; the IOTM is used for time sequence modeling of the in-vehicle and off-vehicle data, and the in-vehicle feature fused with the environment feature, the off-vehicle environment feature and their combination are input into a prediction head to obtain the classification of driving behavior.

[0045] The technical solutions of the application will be described in detail below with reference to the drawings and embodiments.

[0046] Embodiment 1

[0047] The application embodiment provides a lane change prediction method, which extracts features from in-vehicle video sequences and off-vehicle video sequences, fuses speed information, uses GRU and the IOTM module proposed by the application to fuse in-vehicle features and off-vehicle features, and gives a final prediction result.

[0048] A data set is prepared, vehicle-mounted video, off-vehicle environment video and speed information are acquired, the obtained video is randomly sampled to extract key frames, and an image frame sequence is obtained; the in-vehicle image sequence and the off-vehicle image sequence are both 15 frames. After performing a driver-preserving cropping operation on the vehicle-mounted image, the frame size of the image is adjusted to 112x112, and the frame size of the environment image is adjusted to 144x96. The frame rate of the in-vehicle camera is 25fps, and the frame rate of the off-vehicle camera is 30fps.

[0049] The vehicle state text contains four numbers, which represent the speed of the vehicle (represented by -1 for unknown), the x-coordinate, the y-coordinate, and the timestamp. The proposed model can be divided into three modules: feature extraction, temporal feature fusion, and prediction. As shown in FIG. 1. Figure 1

[0050] 1. The feature extraction part uses a pre-trained image processing model to process the vehicle-mounted RGB image sequence to obtain an image feature vector: first, the pre-trained convolutional network selfnet model is used to process the in-vehicle image sequence. The channel number is increased through convolution operation and average pooling, and the last step uses global average pooling to make the last three dimensions become [64, 1, 1], and then through the view operation, each image becomes a one-dimensional vector with a length of 64. For the out-of-vehicle data, a similar method is used to process the out-of-vehicle environment image using a pre-trained lightweight ResNet18, which meets the real-time requirement, and also obtains a one-dimensional image feature vector.

[0051] (1) The Selfnet network for processing in-vehicle data includes: a sequential module for organizing a neural network, which includes a convolutional layer, a ReLu activation function layer, an adaptive average pooling layer, and a maximum pooling layer, wherein the input of the in-vehicle data processing model is an in-vehicle image sequence, and the convolutional neural network is used to convert the vehicle-mounted image into a 64-channel feature map; the output of the vehicle-mounted image processing model is a 1440x64-dimensional feature vector.

[0052] The Selfnet network architecture is sequentially connected by 12 modules, each module contains 4 small modules, and an adaptive average pooling operation is connected after the 4 small modules. Each small module includes convolution operation, ReLU activation function and maximum pooling operation. The pooling kernel size of the maximum pooling operation of the small module is 2x2, the convolution kernel of the convolution operation of the first two small modules is 5x5, the step is 1, and the padding is 1. The convolution kernel of the convolution operation of the last two small modules is 7x7, and the other structures are the same.

[0053] (2) The ResNet18 network for processing out-of-vehicle data includes: a convolutional layer, a batch normalization layer, a ReLu activation function layer, an average pooling layer, a first convolutional network layer, a second convolutional network layer, a third convolutional network layer, a fourth convolutional network layer, a first fully connected layer, a second fully connected layer, and a residual structure, wherein the input of the resnet model is an out-of-vehicle environment image sequence, and the ResNet18 network is used to convert the out-of-vehicle environment image into a 32-channel feature map; the output of the ResNet18 model is a 1440x32-dimensional feature vector.

[0054] ​The ResNet18 network structure is composed of a convolution operation with a convolution kernel size of 7*7, a step of 2, and padding of 3, a batch normalization operation, a ReLU activation function, an average pooling operation with a pooling kernel size of 3*3, a step of 2, and padding of 1, four convolutional neural network layers, and two small modules containing adaptive average pooling structures and fully connected layers, wherein the output dimensions of each convolutional neural network are 64, 128, 256, and 512 dimensions respectively, and the convolutional network layers are composed of 1*1 convolution operations and batch normalization operations.

[0055] 2. The time sequence feature fusion part extracts the features of the in-vehicle image sequence and the out-of-vehicle image sequence, and uses the time sequence feature fusion module (IOTM) proposed by the present application to fuse the in-vehicle image face features and the out-of-vehicle environment semantic features at time t.

[0056] First, the GRU is used to model the time sequence of the in-vehicle and out-of-vehicle data, and the in-vehicle features and the out-of-vehicle environment features are fused. The internal structure of the GRU is as follows:

[0057] z t =σ(W z ·[h t-1 ,x t ])

[0058] Wherein, z t is the update gate, which determines the proportion of the hidden state h t-1 at the previous moment that is retained in the hidden state h t at the current moment. x t is the current in-vehicle or out-of-vehicle feature map input, W z is the weight matrix, and σ(·) is the activation function.

[0059] r t =σ(W r ·[h t-1 ,x t ])

[0060] Wherein, r t is the reset gate, which determines the proportion of the hidden state h t-1 at the previous moment that is reset in the calculation of the candidate hidden state at the current moment. W r is the weight matrix.

[0061]

[0062] is the candidate hidden state, which combines the current input x t and the hidden state h t-1 at the previous moment after being processed by the reset gate.

[0063]

[0064] h t is the hidden state at the current time, which is the hidden state h t-1 at the previous time and the weighted sum of the candidate hidden states .

[0065] The fusion module IOTM of the vehicle external environment features and the vehicle internal features comprises a first GRU layer for processing a vehicle internal image sequence, a second GRU layer for processing a vehicle external image sequence, and a multi-head attention layer for fusing the environment feature information into the vehicle internal image features, wherein the input of the fusion module is the feature maps of the vehicle internal and external at each time t, the module outputs the hidden state of the vehicle internal data at time t by updating the hidden state of the vehicle internal sequence at the previous time, the input features at this time and the hidden state of the vehicle external sequence at this time, and the output comprises t. Considering that the actual driving task is a process of mutual interaction of the vehicle external environment and the vehicle internal environment, the attention mechanism is used to query the historical features H_in(t) at time t using the vehicle external image features at time t-3, which also takes into account the reaction time of the human brain and the number of camera frames. The structure of the IOTM module is shown in Figure 2 , wherein F_in represents the vehicle internal features, F_out represents the vehicle external features, and H_in represents the time series fused features.

[0066] The cross-attention operation in the time series feature fusion unit (IOTM) has a query of 96x64 environment features, a key of 96x64 driver features and a value of 96x64 driver features, and the result is fused through a residual structure and a driver hidden state vector.

[0067] 3. Prediction head

[0068] The driver hidden state is mapped into 64 dimensions through a linear layer, the stacking of the outdoor environment hidden state and the vehicle state in the last dimension is mapped into 64 dimensions through a linear layer, the stacking of the driver hidden state, the outdoor environment hidden state and the vehicle state in the last dimension is mapped into 64 dimensions through a linear layer, and finally it is mapped into the confidence of the category of the final lane change prediction through a linear layer. Specifically, the output is the probability of five categories (straight, left turn, right turn, left lane change and right lane change), such as 0.1, 0.2, 0.6, 0.05, 0.05, and the highest probability is the predicted category.

[0069] As shown in Figure 3 , is a whole flow chart of lane change prediction, which is realized based on the vehicle-mounted vehicle internal camera, vehicle external camera and speed sensor, comprising:

[0070] Obtaining the state of the vehicle to be predicted, extracting the vehicle internal video and the vehicle external environment video;

[0071] preprocessing the video to obtain a sequence of image frames;

[0072] inputting the sequence of image frames into a prediction model that is pre-established and trained to obtain a prediction result of lane changing.

[0073] The prediction model comprises a feature extraction module, a time sequence feature fusion module, and a prediction head, wherein

[0074] The feature extraction module is configured to process the sequence of vehicle-mounted RGB image frames to obtain a facial image feature vector in the vehicle, and is further configured to process the sequence of out-of-vehicle environment image frames to obtain an out-of-vehicle environment feature vector.

[0075] The time sequence feature fusion module is configured to fuse the facial image feature vector in the vehicle and the out-of-vehicle environment feature vector at the same time and complete time sequence modeling.

[0076] The prediction head is configured to make a prediction according to the time sequence modeling and the state of the vehicle to obtain a prediction result of whether to change lanes and the lane changing.

[0077] Embodiment 2

[0078] Embodiment 2 of the present application provides a lane changing prediction device, which is realized based on a vehicle-mounted in-vehicle camera, an out-of-vehicle camera, and a speed sensor, and realizes the method of embodiment 1. The device comprises:

[0079] The acquisition module is configured to acquire the state of a vehicle to be predicted and extract in-vehicle video and out-of-vehicle environment video.

[0080] The preprocessing module is configured to preprocess the video to obtain a sequence of image frames.

[0081] The prediction output module is configured to input the sequence of image frames into a prediction model that is pre-established and trained to obtain a prediction result of lane changing.

[0082] The prediction model comprises a feature extraction module, a time sequence feature fusion module, and a prediction head, wherein

[0083] The feature extraction module is configured to process the sequence of vehicle-mounted RGB image frames to obtain a facial image feature vector in the vehicle, and is further configured to process the sequence of out-of-vehicle environment image frames to obtain an out-of-vehicle environment feature vector.

[0084] The time sequence feature fusion module is configured to fuse the facial image feature vector in the vehicle and the out-of-vehicle environment feature vector at the same time and complete time sequence modeling.

[0085] The prediction head is configured to make a prediction according to the time sequence modeling and the state of the vehicle to obtain a prediction result of whether to change lanes and the lane changing.

[0086] Embodiment 3

[0087] Embodiment 3 of the present application provides a computer device, comprising: at least one processor, a memory, at least one network interface and a user interface. The various components in the device are coupled together by a bus system. It can be understood that the bus system is used to realize the connection communication between the components. In addition to including a data bus, the bus system also includes a power supply bus, a control bus and a status signal bus.

[0088] The user interface can include a display, a keyboard or a pointing device (e.g., a mouse, a trackball, a touchpad or a touchscreen, etc.).

[0089] It can be understood that the memory in the embodiments disclosed in the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM) and direct Rambus RAM (DRRAM). The memory described herein is intended to include, but not be limited to, these and any other suitable types of memory.

[0090] In some embodiments, the memory stores elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system and an application program.

[0091] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks. The application programs include various application programs, such as a media player (Media Player), a browser (Browser), and the like, for implementing various application services. The program for implementing the method of the embodiments of the present disclosure can be included in the application programs.

[0092] In the above-described embodiments, the processor can be configured to, by invoking the program or the instruction stored in the memory, specifically, the program or the instruction stored in the application program:

[0093] perform the steps of the method of Embodiment 1.

[0094] The method of Embodiment 1 can be applied to the processor or implemented by the processor. The processor can be an integrated circuit chip having a signal processing capability. In the implementation process, the steps of the above-described method can be completed by hardware integrated logic circuits in the processor or by the instructions in the form of software. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in Embodiment 1 can be implemented or executed by the processor. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor or the like. The steps of the method disclosed in conjunction with Embodiment 1 can be directly embodied as a hardware coding processor for execution, or a combination of hardware and software modules in the coding processor. The software module can reside in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, or the like storage medium in the art. The storage medium is located in the storage memory, and the processor reads information in the storage memory and combines the hardware to complete the steps of the above-described method.

[0095] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For a hardware implementation, the processing units can be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSP Devices), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, other electronic units designed to perform the functions described herein, or a combination thereof.

[0096] For a software implementation, the techniques described herein can be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. The software codes can be stored in memory units and executed by processors. The memory unit can be implemented within the processor or external to the processor.

[0097] Finally, it should be noted that the above-mentioned embodiments are merely given as an illustration of the technical solution of the present application, but are not a limitation. Although the present application has been described in detail with reference to the embodiments, it should be understood that modifications or equivalent replacements to the technical solution of the present application can be made without departing from the spirit and scope of the present application, and all such modifications and equivalent replacements should be included in the scope of the claims of the present application.

Claims

1. A lane changing prediction method, realized based on an in-vehicle camera, an out-vehicle camera and a speed sensor, comprising: obtaining the state of a vehicle to be predicted, extracting in-vehicle video and out-vehicle environment video; preprocessing the video to obtain an image frame sequence; inputting the image frame sequence into a pre-established and trained prediction model to obtain a predicted lane changing result; the prediction model comprises a feature extraction module, a time sequence feature fusion module and a prediction head, wherein the feature extraction module is configured to process the in-vehicle RGB image frame sequence to obtain an in-vehicle face image feature vector, and is also configured to process the out-vehicle environment image frame sequence to obtain an out-vehicle environment feature vector; the time sequence feature fusion module is configured to fuse the in-vehicle face image feature vector and the out-vehicle environment feature vector at the same time, and complete time sequence modeling; the prediction head is configured to predict according to the time sequence modeling and the state of the vehicle to obtain the prediction result of whether to change lanes and the lane changing; the time sequence feature fusion module IoTM comprises a GRU layer and a multi-head attention mechanism; wherein the GRU layer comprises a first GRU layer configured to process the in-vehicle face image feature vector F_in(t) at time t, a second GRU layer configured to process the out-vehicle environment feature vector F_out(t) at time t, and a fusion feature vector H_in(t) at time t is obtained by fusion; the multi-head attention mechanism is configured to query the out-vehicle environment feature vector F_out(t-3) at time t-3, and fuse the fusion feature vector H_in(t) at time t and the fusion feature vector H_in(t-5) at time t-5 through residual connection to obtain the fusion feature vector H_in(t+1) at time t+1.

2. The lane change prediction method of claim 1, wherein The frame rate of the in-vehicle camera is 25 fps, and the frame rate of the out-vehicle camera is 30 fps.

3. The lane change prediction method of claim 1, wherein The state of the vehicle to be predicted includes vehicle speed, x coordinate, y coordinate and time stamp.

4. The lane change prediction method of claim 1, wherein, The preprocessing comprises: randomly sampling the in-vehicle video and the out-vehicle environment video to obtain the same number of key frames, and adjusting the frame size to obtain the image frame sequence.

5. The lane change prediction method of claim 1, wherein, The feature extraction module comprises an in-vehicle feature extraction unit and an out-vehicle feature extraction unit, wherein the in-vehicle feature extraction unit adopts a Selfnet network, increases the number of channels through convolution operation and average pooling, uses global average pooling, and uses view operation to make each image into a fixed-length in-vehicle face image feature vector; the out-vehicle feature extraction unit adopts a ResNet18 network to obtain a one-dimensional out-vehicle environment feature vector.

6. The lane change prediction method of claim 1, wherein, The prediction head maps the in-vehicle face image feature vector to 64 dimensions, stacks the fusion feature vector and the vehicle state on the last dimension through a linear layer to map to 64 dimensions, and then maps to the probability of different categories through a linear layer. The category corresponding to the maximum probability is selected as the lane changing prediction result, and the lane changing prediction result includes straight, left turn, right turn, left lane change and right lane change.

7. The lane change prediction method of claim 1, wherein, The method comprises a training step of the prediction model, comprising: establishing a training set, the training set comprising preprocessed in-vehicle video and out-vehicle environment video and vehicle speed information; The data of the training set is input into the prediction model, a loss function is calculated, and the training is performed through gradient back propagation until a prediction model meeting the training requirements is obtained.

8. A lane change prediction device realized based on an in-vehicle camera, an out-of-vehicle camera, and a speed sensor, characterized by, The device comprises: An acquisition module is configured to acquire a state of a vehicle to be predicted, extract in-vehicle video and out-of-vehicle environment video. A preprocessing module is configured to preprocess the video to obtain image frame sequences. A prediction output module is configured to input the image frame sequences into a prediction model that is pre-established and trained to obtain a lane-changing prediction result. The prediction model comprises a feature extraction module, a time-series feature fusion module and a prediction head, wherein The feature extraction module is configured to process in-vehicle RGB image frame sequences to obtain in-vehicle face image feature vectors, and process out-of-vehicle environment image frame sequences to obtain out-of-vehicle environment feature vectors. The time-series feature fusion module is configured to fuse the in-vehicle face image feature vectors and the out-of-vehicle environment feature vectors at the same time, and complete time-series modeling. The prediction head is configured to predict according to the time-series modeling and the state of the vehicle to obtain a prediction result of whether to change lanes and the lane-changing. The time-series feature fusion module IoTM comprises a GRU layer and a multi-head attention mechanism, wherein The GRU layer comprises a first GRU layer configured to process in-vehicle face image feature vectors F_in(t) at time t, a second GRU layer configured to process out-of-vehicle environment feature vectors F_out(t) at time t, and a fusion feature vector H_in(t) at time t is obtained by fusion. The multi-head attention mechanism is configured to query the out-of-vehicle environment feature vector F_out(t-3) at time t-3, and fuse the fusion feature vector H_in(t) at time t and the fusion feature vector H_in(t-5) at time t-5 through residual connection to obtain the fusion feature vector H_in(t+1) at time t+1.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1-8. The processor executes the computer program to implement the steps of the method of claim 1.

Images