Lane change prediction method and device and computer equipment

Abstract

The invention belongs to the technical field of intelligent driving, and particularly relates to a lane changing prediction method and device and computer equipment. The method is realized based on a vehicle-mounted in-vehicle camera, a vehicle-mounted out-vehicle camera and a vehicle-mounted speed sensor, and comprises the following steps: acquiring the state of a to-be-predicted vehicle, and extracting an in-vehicle video and an out-vehicle environment video; preprocessing the video to obtain an image frame sequence; inputting the image frame sequence into a pre-established and trained prediction model to obtain a predicted lane changing result; the prediction model comprises a feature extraction module used for processing a vehicle-mounted RGB image frame sequence to obtain an in-vehicle face image feature vector and processing an out-vehicle environment image frame sequence to obtain an out-vehicle environment feature vector; the time sequence feature fusion module is used for fusing the in-vehicle face image feature vector and the out-vehicle environment feature vector at the same moment and completing time sequence modeling; and the prediction head is used for performing prediction according to time sequence modeling and the state of the vehicle to obtain prediction results of whether to change the lane and how to change the lane.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent driving technology, and in particular relates to a lane change prediction method, device and computer equipment. Background Art

[0002] Driving behavior prediction technology is a key component in the field of intelligent transportation systems and autonomous driving. With the rapid development of technologies such as big data, machine learning, and deep learning, this field has made significant progress.

[0003] The construction of the prediction model is based on a large amount of driving data, including vehicle speed, acceleration, steering angle, etc. By analyzing this data, the model can learn the patterns and laws of driving behavior. Early prediction models mainly relied on traditional machine learning algorithms, such as decision trees, support vector machines, and random forests. However, as the amount of data increases, the performance of these algorithms is gradually limited, prompting researchers to begin using deep learning techniques, such as convolutional neural networks, recurrent neural networks, and long short-term memory networks. These models can process complex spatiotemporal data and capture the long-term dependencies of driving behavior.

[0004] With the advancement of technology, many studies have adopted multimodal data to model driving behavior. For example, integrating data from different sources such as visual data, sensor data, and GPS data into the prediction model can provide more comprehensive information about the driving environment, thereby improving the accuracy of the prediction. However, although simple data fusion strategies are effective, they may not be able to describe the driver's perception of the external world, which is crucial for understanding the driver's intentions. Some studies have tried to use optical flow extraction and 3D convolutional neural networks to handle spatial and temporal correlations, but these methods are limited in their application in vehicle systems due to their high computational cost and large memory requirements.

[0005] In order to improve the accuracy of driving behavior prediction, researchers began to try to model driving behavior from the driver's perspective. These methods are attractive in theory, but the noise problem of physiological signals affects their actual performance. To overcome this problem, some studies have combined driver characteristics with vehicle dynamics and scene information and proposed a state update unit. This unit selectively introduces environmental information at each time step based on a multi-head attention mechanism. However, this approach may increase the complexity of the model, resulting in increased computational costs, and requires more training data to effectively learn attention weights. Finally, by passing the environment state, the driver state, and their concatenation into the classifier, the behavior probability can be generated and jointly predicted, thereby improving the accuracy and reliability of the prediction. At the same time, the traffic environment is dynamically changing, and there may be many unpredictable external disturbances, such as emergencies or atypical behaviors of other drivers, which may affect the stability and robustness of the model. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the prior art and to propose a lane change prediction method, device and electronic equipment.

[0007] In order to achieve the above object, the present invention provides a lane change prediction method, which is implemented based on an in-vehicle camera, an out-vehicle camera and a speed sensor, and includes:

[0008] Obtain the state of the vehicle to be predicted, and extract the in-vehicle video and the video of the outside environment;

[0009] Preprocess the video to obtain an image frame sequence;

[0010] The image frame sequence is input into a pre-established and trained prediction model to obtain the predicted lane change result;

[0011] The prediction model includes: a feature extraction module, a temporal feature fusion module and a prediction head, wherein:

[0012] The feature extraction module is used to process the vehicle-mounted RGB image frame sequence to obtain the feature vector of the in-vehicle face image, and is also used to process the vehicle-exterior environment image frame sequence to obtain the vehicle-exterior environment feature vector;

[0013] The temporal feature fusion module is used to fuse the in-vehicle face image feature vector and the in-vehicle environment feature vector at the same time and complete the temporal modeling;

[0014] The prediction head is used to make predictions based on time series modeling and the state of the vehicle to obtain whether to change lanes and the prediction results of the lane change.

[0015] Preferably, the frame rate of the in-vehicle camera is 25fps, and the frame rate of the out-vehicle camera is 30fps.

[0016] Preferably, the state of the vehicle to be predicted includes vehicle speed, x-coordinate, y-coordinate and timestamp.

[0017] Preferably, the pretreatment comprises:

[0018] The in-vehicle video and the outdoor environment video are randomly sampled to obtain the same number of key frames, and the frame size is adjusted to obtain an image frame sequence.

[0019] Preferably, the feature extraction module includes an in-vehicle feature extraction unit and an out-vehicle feature extraction unit, wherein:

[0020] The in-car feature extraction unit adopts a Selfnet network, increases the number of channels through convolution operations and average pooling, and then uses global average pooling to convert each image into a fixed-length in-car face image feature vector through a view operation;

[0021] The vehicle exterior feature extraction unit adopts a ResNet18 network to obtain a one-dimensional vehicle exterior environment feature vector.

[0022] Preferably, the temporal feature fusion module IoTM includes: a GRU layer and a multi-head attention mechanism; wherein,

[0023] The GRU layer includes: a first GRU layer for processing the feature vector F_in(t) of the face image inside the vehicle at time t, a second GRU layer for processing the feature vector F_out(t) of the environment outside the vehicle at time t, and then fusing to obtain the fused feature vector H_in(t) at time t;

[0024] The multi-head attention mechanism is used to use the vehicle exterior environment feature vector F_out(t-3) at time t-3 for query, and to connect the fused feature vector H_in(t) at time t and the fused feature vector H_in(t-5) at time t-5 through residual connection; to obtain the fused feature vector H_in(t+1) at time t+1.

[0025] Preferably, the prediction head maps the feature vector of the facial image in the car to 64 dimensions, maps the stack of the fused feature vector and the vehicle state in the last dimension to 64 dimensions through a linear layer, and then maps it to probabilities of different categories through a linear layer, and selects the category corresponding to the maximum probability as the lane change prediction result, and the lane change prediction result includes going straight (i.e., not changing lanes), turning left, turning right, changing left lanes, and changing right lanes.

[0026] Preferably, the method comprises a step of training a prediction model, comprising:

[0027] Establishing a training set, wherein the training set includes preprocessed vehicle-mounted video, vehicle-exterior environment video, and vehicle speed information;

[0028] Input the data of the training set into the prediction model, calculate the loss function, and perform training through gradient backpropagation until a prediction model that meets the training requirements is obtained.

[0029] In a second aspect, the present invention provides a lane change prediction device, which is implemented based on an in-vehicle camera, an out-vehicle camera and a speed sensor, and the device includes:

[0030] The acquisition module is used to obtain the state of the vehicle to be predicted and extract the in-vehicle video and the video of the external environment;

[0031] A preprocessing module, used for preprocessing the video to obtain an image frame sequence;

[0032] A prediction output module is used to input the image frame sequence into a pre-established and trained prediction model to obtain a predicted lane change result;

[0033] The prediction model includes: a feature extraction module, a temporal feature fusion module and a prediction head, wherein:

[0034] The feature extraction module is used to process the vehicle-mounted RGB image frame sequence to obtain the feature vector of the in-vehicle face image, and is also used to process the vehicle-exterior environment image frame sequence to obtain the vehicle-exterior environment feature vector;

[0035] The temporal feature fusion module is used to fuse the in-vehicle face image feature vector and the in-vehicle environment feature vector at the same time and complete the temporal modeling;

[0036] The prediction head is used to make predictions based on time series modeling and the state of the vehicle to obtain whether to change lanes and the prediction results of the lane change.

[0037] In a third aspect, the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0038] Compared with the prior art, the advantages of the present invention are:

[0039] 1. The present invention proposes a new time series feature fusion module (IOTM) to fuse the facial features and historical information of the in-vehicle image at time t and the semantic features of the outside environment; IOTM is used to perform time series modeling on the in-vehicle and outside data, and the in-vehicle features of the fused environmental features, the outside environment features and their combination are used through a prediction head to obtain the classification of driving behavior, which effectively improves the speed and accuracy of driving behavior classification.

[0040] 2. The feature extraction module used in the present invention focuses on the driver's perception and cognition of the external environment, which is crucial for accurately understanding and predicting the driver's driving intentions. By combining advanced sensor technology and data processing algorithms, the module can efficiently obtain and analyze the driver's real-time feedback on the external environment, covering visual, auditory, and other sensory inputs. At the same time, the module models the driver's internal cognitive process through deep learning technology and seamlessly integrates it with the dynamic changes of the external environment. This fusion not only significantly improves the stability and robustness of the prediction model in complex driving scenarios, and can effectively respond to a variety of emergencies, but also optimizes resource utilization, maintains low computing costs and memory requirements, and is suitable for real-time processing requirements of vehicle-mounted systems. The design of the module reduces dependence on hardware resources while ensuring high precision, ensures efficient operation in embedded devices, and provides more reliable decision support for intelligent driving systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a schematic diagram of a lane change prediction model;

[0042] Figure 2 This is a schematic diagram of the timing feature fusion module IOTM;

[0043] Figure 3 This is the overall flow chart of lane change prediction. DETAILED DESCRIPTION

[0044] The present invention provides a lane change prediction method, comprising: acquiring an in-vehicle video and an outside-vehicle environment video; performing random sampling processing on the video to obtain an image frame sequence; using a pre-trained image processing model to process the in-vehicle RGB image sequence to obtain an image feature vector; (using a pre-trained mobilefacenet model to process the in-vehicle image sequence to obtain an in-vehicle (face) image feature vector; using a pre-trained image processing model to process the outside-vehicle environment image sequence to obtain an image feature vector; (using a pre-trained ResNet-18 model to process the outside-vehicle environment (optical flow / semantic segmentation) image sequence to obtain an outside-vehicle environment feature vector;) We propose a new temporal feature fusion module (Inside-features and Outside-features Temporal Merge) IOTM to fuse the face features and historical information of the in-vehicle image at time t and the semantic features of the outside-vehicle environment; using IOTM to perform temporal modeling on the in-vehicle and outside-vehicle data, and obtaining the classification of driving behavior through a prediction head using the in-vehicle features fused with environmental features, the outside-vehicle environment features and their combination.

[0045] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0046] Example 1

[0047] The embodiment of the present application provides a lane change prediction method, which extracts features from the in-vehicle video sequence and the out-vehicle video sequence, fuses the speed information, and uses the GRU and the IOTM module proposed by us to fuse the in-vehicle features and the out-vehicle features, and gives the final prediction result:

[0048] Prepare the data set, obtain the in-vehicle video, the video of the outside environment and the vehicle speed information; randomly sample the obtained video to extract the key frames, and obtain the image frame sequence, the in-vehicle image sequence and the outside image sequence are 15 frames each. After the in-vehicle image is cropped to retain the driver, the frame size of the image is adjusted to 112×112, and the frame size of the environment image is adjusted to 144×96. The frame rate of the in-vehicle camera is 25fps, and the frame rate of the outside camera is 30fps.

[0049] The vehicle status text contains four numbers, which represent the vehicle speed (unknown is represented by -1), x-coordinate, y-coordinate and timestamp. The model we proposed can be divided into three modules: feature extraction, time series feature fusion and prediction head. Figure 1 shown.

[0050] 1. Feature extraction: Use the pre-trained image processing model to process the vehicle RGB image sequence to obtain the image feature vector: First, use the pre-trained convolutional network selfnet model to process the in-vehicle image sequence for the in-vehicle data and the out-of-vehicle data. Increase the number of channels through convolution operations and average pooling. In the last step, use global average pooling to make the last three dimensions become [64,1,1]. Then use the view operation to make each image a one-dimensional vector of length 64. For the out-of-vehicle data, use a similar method to use the pre-trained lightweight ResNet18 to process the out-of-vehicle environment images to meet the real-time requirements, and also obtain a one-dimensional image feature vector.

[0051] (1) The Selfnet network for processing in-vehicle data includes: a sequential module for organizing a neural network, including a convolutional layer, a ReLu activation function layer, an adaptive average pooling layer, and a maximum pooling layer, wherein the input of the in-vehicle data processing model is an in-vehicle image sequence, and the convolutional neural network is used to convert the in-vehicle image into a 64-channel feature map; the output of the in-vehicle image processing model is a 1440×64-dimensional feature vector.

[0052] The Selfnet network architecture consists of 12 modules connected in sequence. Each module contains 4 small modules. An adaptive average pooling operation is connected after the 4 small modules. Each small module includes a convolution operation, a ReLU activation function, and a maximum pooling operation. The pooling kernel size of the maximum pooling operation of the small module is 2×2. The convolution kernel of the convolution operation of the first two small modules is 5×5, the step size is 1, and the padding is 1. The convolution kernel of the convolution operation of the last two small modules is 7×7. The other structures are the same.

[0053] (2) The ResNet18 network for processing the data outside the vehicle includes: a convolution layer, a batch normalization layer, a ReLu activation function layer, an average pooling layer, a first convolution network layer, a second convolution network layer, a third convolution network layer, a fourth convolution network layer, a first fully connected layer, a second fully connected layer and a residual structure, wherein the input of the resnet model is a sequence of images of the outside vehicle environment, and the ResNet18 network is used to convert the outside vehicle environment image into a 32-channel feature map; the output of the ResNet18 model is a 1440×32-dimensional feature vector.

[0054] The ResNet18 network structure consists of a convolution operation with a convolution kernel size of 7×7, a stride of 2, and a padding of 3, a batch normalization operation, a ReLU activation function, an average pooling operation with a pooling kernel size of 3×3, a stride of 2, and a padding of 1, four convolutional neural network layers, and two small modules containing an adaptive average pooling structure and a fully connected layer. The output dimensions of each convolutional neural network are 64, 128, 256, and 512 dimensions, respectively, and the convolutional network layers are composed of 1×1 convolution operations and batch normalization operations.

[0055] 2. In the temporal feature fusion part, we extract the features of the in-vehicle image sequence and the out-vehicle image sequence, and use the proposed temporal feature fusion module (IOTM) to fuse the facial features of the in-vehicle image at time t and the semantic features of the out-vehicle environment.

[0056] First, GRU is used to perform time series modeling on the data inside and outside the car, and the in-car features and the external environment features of the fused environmental features are combined. The internal structure of GRU is as follows:

[0057] z t =σ(W z ·[h t-1 ,x t ])

[0058] Among them, z t is the update gate, which determines the hidden state h at the previous moment t-1 At the current moment, the hidden state h t What proportion of the x is retained. t is the current in-vehicle or out-vehicle feature map input, W z is the weight matrix and σ(·) is the activation function.

[0059] r t =σ(W r ·[h t-1 ,x t ])

[0060] Among them, r t is the reset gate, which determines the hidden state h at the previous moment t-1 Calculate the candidate hidden state at the current moment How much of the ratio is reset. r is the weight matrix.

[0061]

[0062] is a candidate hidden state that combines the current input x t and the previous hidden state h after the reset gate t-1 .

[0063]

[0064] h t is the hidden state at the current moment, which is the hidden state h at the previous moment t-1 and candidate hidden states The weighted sum of .

[0065] The module IOTM for fusing the features of the external environment and the features of the interior of the vehicle includes: a first GRU layer for processing the sequence of images inside the vehicle, a second GRU layer for processing the sequence of images outside the vehicle, and a multi-head attention layer for fusing the environmental feature information into the features of the image inside the vehicle, wherein the input of the fusion module is the feature map of the interior and exterior of the vehicle at each moment t, and the module updates the hidden state of the data inside the vehicle through the hidden state of the previous moment of the sequence inside the vehicle, the input features at this moment, and the hidden state of the sequence outside the vehicle at this moment, and the output is a sequence containing t. Considering that the actual driving task is a process of interaction between the external environment and the internal environment of the vehicle, we use the attention mechanism to use the external image features at moment t-3 to query the historical features H_in(t) at moment t, which also takes into account the reaction time of the human brain and the number of camera frames. The structure of the IOTM module is as follows Figure 2 As shown, F_in represents the features inside the vehicle, F_out represents the features outside the vehicle, and H_in represents the features after time series fusion.

[0066] The query of the cross attention operation in the temporal feature fusion unit (IOTM) is the 96×64 environmental features, and the key and value are the 96×64 driver features. The results are fused with the driver hidden state vector through the residual structure.

[0067] 3. Prediction Head

[0068] The driver's hidden state is mapped to 64 dimensions through a linear layer, the stack of the outdoor environment hidden state and the vehicle state on the last dimension is mapped to 64 dimensions through a linear layer, the stack of the driver's hidden state, the outdoor environment hidden state and the vehicle state on the last dimension is mapped to 64 dimensions through a linear layer, and finally mapped to the confidence of the final lane change prediction category through a linear layer. Specifically, the output is the probability of five categories (straight, left turn, right turn, left lane change and right lane change), such as 0.1, 0.2, 0.6, 0.05, 0.05, and the highest probability is the predicted category.

[0069] like Figure 3 The following is the overall flow chart of lane change prediction, which is based on the vehicle's in-vehicle camera, out-vehicle camera and speed sensor, including:

[0070] Obtain the state of the vehicle to be predicted, and extract the in-vehicle video and the video of the outside environment;

[0071] Preprocess the video to obtain an image frame sequence;

[0072] The image frame sequence is input into the pre-established and trained prediction model to obtain the predicted lane change result.

[0073] The prediction model includes: feature extraction module, time series feature fusion module and prediction head, among which,

[0074] A feature extraction module is used to process the vehicle-mounted RGB image frame sequence to obtain a feature vector of the face image inside the vehicle, and is also used to process the vehicle-exterior environment image frame sequence to obtain a feature vector of the vehicle-exterior environment;

[0075] The time series feature fusion module is used to fuse the feature vector of the face image inside the car and the feature vector of the environment outside the car at the same time and complete the time series modeling;

[0076] The prediction head is used to make predictions based on time series modeling and the vehicle's status to obtain the prediction results of whether to change lanes and the lane change.

[0077] Example 2

[0078] Embodiment 2 of the present invention provides a lane change prediction device, which is implemented based on an in-vehicle camera, an out-vehicle camera, and a speed sensor mounted on a vehicle to implement the method of embodiment 1. The device includes:

[0079] The acquisition module is used to obtain the state of the vehicle to be predicted and extract the in-vehicle video and the video of the external environment;

[0080] A preprocessing module, used for preprocessing the video to obtain an image frame sequence;

[0081] A prediction output module is used to input the image frame sequence into a pre-established and trained prediction model to obtain a predicted lane change result;

[0082] The prediction model includes: feature extraction module, time series feature fusion module and prediction head, among which,

[0083] A feature extraction module is used to process the vehicle-mounted RGB image frame sequence to obtain a feature vector of the face image inside the vehicle, and is also used to process the vehicle-exterior environment image frame sequence to obtain a feature vector of the vehicle-exterior environment;

[0084] The time series feature fusion module is used to fuse the feature vector of the face image inside the car and the feature vector of the environment outside the car at the same time and complete the time series modeling;

[0085] The prediction head is used to make predictions based on time series modeling and the vehicle's status to obtain the prediction results of whether to change lanes and the lane change.

[0086] Example 3

[0087] Embodiment 3 of the present invention provides a computer device, comprising: at least one processor, a memory, at least one network interface and a user interface. The various components in the device are coupled together through a bus system. It can be understood that the bus system is used to realize the connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus.

[0088] The user interface may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0089] It is understood that the memory in the disclosed embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0090] In some embodiments, the memory stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system and applications.

[0091] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present disclosure can be included in the application.

[0092] In the above embodiment, the processor may also call a program or instruction stored in the memory, specifically, a program or instruction stored in an application program, and is used to:

[0093] Execute the steps of the method of Example 1.

[0094] The method of embodiment 1 can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in embodiment 1 can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with embodiment 1 can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0095] It is understood that the embodiments described in the present invention can be implemented by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.

[0096] For software implementation, the technology of the present invention can be implemented by executing the functional modules (such as procedures, functions, etc.) of the present invention. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention is described in detail with reference to the embodiments, it should be understood by those skilled in the art that any modification or equivalent replacement of the technical solutions of the present invention does not depart from the spirit and scope of the technical solutions of the present invention and should be included in the scope of the claims of the present invention.

Claims

1. A lane change prediction method, based on an in-vehicle camera, an out-vehicle camera and a speed sensor, comprising: Obtain the state of the vehicle to be predicted, and extract the in-vehicle video and the video of the outside environment; Preprocess the video to obtain an image frame sequence; The image frame sequence is input into a pre-established and trained prediction model to obtain the predicted lane change result; The prediction model includes: a feature extraction module, a temporal feature fusion module and a prediction head, wherein: The feature extraction module is used to process the vehicle-mounted RGB image frame sequence to obtain the feature vector of the in-vehicle face image, and is also used to process the vehicle-exterior environment image frame sequence to obtain the vehicle-exterior environment feature vector; The temporal feature fusion module is used to fuse the in-vehicle face image feature vector and the in-vehicle environment feature vector at the same time and complete the temporal modeling; The prediction head is used to make predictions based on time series modeling and the state of the vehicle to obtain whether to change lanes and the prediction results of the lane change.

2. The lane change prediction method according to claim 1, characterized in that: The frame rate of the in-vehicle camera is 25fps, and the frame rate of the out-vehicle camera is 30fps.

3. The lane change prediction method according to claim 1, characterized in that: The state of the vehicle to be predicted includes vehicle speed, x-coordinate, y-coordinate and timestamp.

4. The lane change prediction method according to claim 1, characterized in that: The pre-processing comprises: The in-vehicle video and the outdoor environment video are randomly sampled to obtain the same number of key frames, and the frame size is adjusted to obtain an image frame sequence.

5. The lane change prediction method according to claim 1, characterized in that: The feature extraction module includes an in-vehicle feature extraction unit and an out-vehicle feature extraction unit, wherein: The in-car feature extraction unit adopts a Selfnet network, increases the number of channels through convolution operations and average pooling, and then uses global average pooling to convert each image into a fixed-length in-car face image feature vector through a view operation; The vehicle exterior feature extraction unit adopts a ResNet18 network to obtain a one-dimensional vehicle exterior environment feature vector.

6. The lane change prediction method according to claim 1, characterized in that: The temporal feature fusion module IoTM includes: a GRU layer and a multi-head attention mechanism; wherein, The GRU layer includes: a first GRU layer for processing the feature vector F_in(t) of the face image inside the vehicle at time t, a second GRU layer for processing the feature vector F_out(t) of the environment outside the vehicle at time t, and then fusing to obtain the fused feature vector H_in(t) at time t; The multi-head attention mechanism is used to use the vehicle exterior environment feature vector F_out(t-3) at time t-3 for query, and to connect the fused feature vector H_in(t) at time t and the fused feature vector H_in(t-5) at time t-5 through residual connection; to obtain the fused feature vector H_in(t+1) at time t+1.

7. The lane change prediction method according to claim 1, characterized in that: The prediction head maps the feature vector of the facial image in the car to 64 dimensions, maps the stack of the fused feature vector and the vehicle status on the last dimension to 64 dimensions through a linear layer, and then maps it to the probabilities of different categories through a linear layer, and selects the category corresponding to the maximum probability as the lane change prediction result. The lane change prediction results include straight going, left turn, right turn, left lane change and right lane change.

8. The lane change prediction method according to claim 1, characterized in that: The method includes a step of training a prediction model, including: Establishing a training set, wherein the training set includes preprocessed vehicle-mounted video, vehicle-exterior environment video, and vehicle speed information; Input the data of the training set into the prediction model, calculate the loss function, and perform training through gradient backpropagation until a prediction model that meets the training requirements is obtained.

9. A lane change prediction device, based on an in-vehicle camera, an out-vehicle camera and a speed sensor, characterized in that: The device comprises: The acquisition module is used to obtain the state of the vehicle to be predicted and extract the in-vehicle video and the video of the external environment; A preprocessing module, used for preprocessing the video to obtain an image frame sequence; A prediction output module is used to input the image frame sequence into a pre-established and trained prediction model to obtain a predicted lane change result; The prediction model includes: a feature extraction module, a temporal feature fusion module and a prediction head, wherein: The feature extraction module is used to process the vehicle-mounted RGB image frame sequence to obtain the feature vector of the in-vehicle face image, and is also used to process the vehicle-exterior environment image frame sequence to obtain the vehicle-exterior environment feature vector; The temporal feature fusion module is used to fuse the in-vehicle face image feature vector and the in-vehicle environment feature vector at the same time and complete the temporal modeling; The prediction head is used to make predictions based on time series modeling and the state of the vehicle to obtain a prediction result of whether to change lanes and the lane change.

10. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claim 1.

Images