Collision prediction system and method for smart vehicle with body based on multi-modal fusion

Abstract

The invention relates to an intelligent vehicle collision prediction system and method based on multi-modal fusion, and belongs to the field of automatic driving. The system is composed of a multi-mode sensor, a data storage unit, a data preprocessing module, a feature extraction module, a feature fusion module and a prediction output module. The method comprises the following steps: S1, collecting visual image data and non-visual image data; s2, preprocessing the visual image data and the non-visual image data; s3, performing feature extraction by using a feature extraction module; s4, performing feature splicing to form a comprehensive feature vector; s5, introducing a query vector, and performing feature fusion on the query vector and the comprehensive feature vector by using a feature fusion module; and S6, inputting the fusion features into a prediction output module for prediction, and obtaining a collision prediction result. According to the method, collision can be predicted with high precision, the edge calculation capability of the intelligent vehicle is exerted, the real-time performance of calculation is improved, and the traffic safety and the system intelligence level can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to a collision prediction system and method for an embodied intelligent vehicle based on multimodal fusion, which belongs to the field of autonomous driving, and is particularly suitable for collision prediction for an embodied intelligent vehicle based on multimodal fusion. Background Art

[0002] Embodied intelligent vehicles refer to mobile vehicles that can autonomously perceive the environment, make decisions, and perform actions in the physical world, such as unmanned vehicles, food delivery robots, and sorting robots. With the rapid development of embodied intelligent technology, it has been widely used in various scenarios such as industry, home, and public services. The safety issues of embodied intelligent vehicles in the process of interactive movement with people are increasingly receiving attention.

[0003] Traditional embodied intelligent collision prediction methods mainly rely on a single type of data, such as vision or radar detection, but in complex application environments, these methods often have difficulty accurately predicting potential collision risks. In recent years, deep learning technology, especially multimodal data fusion technology, has made significant progress in image recognition, behavior prediction and other fields. The Video Vision Transformer (Video-ViT) method given in reference [1] proposes a powerful video feature extraction model that can effectively capture the spatiotemporal dynamic information in the video. At the same time, the Q-Former method given in reference [2] proposes a feature fusion model based on the self-attention mechanism, which can integrate multi-source heterogeneous data and improve the accuracy of prediction.

[0004] However, these existing technologies still have deficiencies in multimodal data fusion and high-level semantic understanding, especially in guiding the model to focus on key features to improve prediction performance. Therefore, there is an urgent need for a system and method that can effectively integrate video data and traffic information and use advanced deep learning models for collision prediction.

[0005] [1]Tong Z, Song Y, Wang J, et al. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training[J]. Advances inneural information processing systems, 2022,35:10078-10093.

[0006] [2]Li J, Li D, Savarese S, et al. Blip-2: Bootstrapping language-imagepre-training with frozen image encoders and large language models [C] / / International conference on machine learning. PMLR, 2023:19730-19742. Summary of the invention

[0007] The purpose of the present invention is to provide an embodied intelligent vehicle collision prediction system and method based on multimodal fusion. For scenarios where pedestrians are present during the driving of an embodied intelligent vehicle, high-precision prediction of the collision between the vehicle and pedestrians is achieved through feature extraction and fusion of multimodal data combined with the semantic understanding ability of a large language model.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] The embodied intelligent vehicle collision prediction system based on multimodal fusion is characterized in that it is composed of a multimodal sensor, a data storage unit, a data preprocessing module, a feature extraction module, a feature fusion module, and a prediction output module; the multimodal sensor includes a visual image sensor and a non-visual image sensor, which respectively collect visual image data and non-visual image data according to a period; the data preprocessing module is respectively connected to the multimodal sensor and the feature extraction module, and is used for preprocessing the visual image data and the non-visual image data; the feature extraction module includes a visual feature extraction module and a non-visual feature extraction module; the visual feature extraction module and the non-visual feature extraction module respectively use a deep learning network to realize the feature extraction of the corresponding preprocessed visual image data and the non-visual image data; the feature fusion module is respectively connected to the feature extraction module, and the encoding of the self-attention mechanism is used to realize the fusion of visual and non-visual features related to the collision prediction task; the prediction output module is connected to the feature fusion module, and the result prediction is realized based on a large language model; the data storage unit is respectively connected to the data acquisition module and the prediction output module for data storage.

[0010] Furthermore, the visual feature extraction module adopts a pre-trained Video-VIT-base model; the non-visual feature extraction module adopts a pre-trained Transformer model based on the Video-VIT-small model architecture. For specific model construction, please refer to reference [1].

[0011] Furthermore, the feature fusion module adopts a feature fusion model based on Q-Former. For the specific model construction, please refer to reference [2].

[0012] Furthermore, the embodied intelligent vehicle collision prediction system based on multimodal fusion is entirely arranged on the embodied intelligent vehicle, and a wireless communication module is installed on the embodied intelligent vehicle for communicating with the terminal.

[0013] Furthermore, the embodied intelligent vehicle collision prediction system based on multimodal fusion is arranged on the embodied intelligent vehicle and the cloud server, and communicates through wireless communication modules installed on the embodied intelligent vehicle and the cloud server respectively.

[0014] Preferably, for scenarios where the computing power of the embodied intelligent vehicle is poor, the multimodal sensor is installed on the embodied intelligent vehicle; and the data storage unit, data preprocessing module, feature extraction module, feature fusion module, and prediction output module are installed on a cloud server.

[0015] Preferably, since the embodied intelligent vehicle has a certain computing power, the data preprocessing module can be split into a visual image preprocessing module and a non-visual image preprocessing module according to the visual image data and the non-visual image data; the multimodal sensor, the non-visual image preprocessing module, and the non-visual feature extraction module are installed on the embodied intelligent vehicle; the visual image preprocessing module, the visual feature extraction module, the data storage unit, the visual feature fusion module, and the prediction output module are installed on the cloud server; wherein, the visual image preprocessing module is connected to the visual feature extraction module, the non-visual image preprocessing module is respectively connected to the multimodal sensor and the non-visual feature extraction module, and the data storage unit is connected to the feature fusion module, and the features of the non-visual image are directly transmitted to the feature fusion module for feature fusion.

[0016] The embodied intelligent vehicle collision prediction method based on multimodal fusion includes the following steps:

[0017] S1: using a multimodal sensor to periodically collect visual image data and non-visual image data, and store them in a data storage unit in a time sequence;

[0018] S2: The data preprocessing module preprocesses the visual image data and the non-visual image data in the data storage unit;

[0019] S3: The feature extraction module extracts features from the preprocessed visual image data and non-visual image data to obtain visual image features and non-visual image features;

[0020] S4: concatenate visual image features and non-visual image features in the feature dimension to form a comprehensive feature vector;

[0021] S5: introduce a set of query vectors to be learned, and use the feature fusion module to fuse the query vectors with the comprehensive feature vectors to obtain fused features;

[0022] S6: Input the fused features into the prediction output module for high-level semantic understanding and prediction, and obtain the collision prediction results.

[0023] Further, the preprocessing described in step S2 comprises the following steps:

[0024] S201: Identify whether there is a pedestrian in the visual image data at the current moment, if not, wait for the visual image data at the next moment, otherwise continue to execute the following steps;

[0025] S202: resizing the image data in the visual image data and performing normalization processing;

[0026] S203: According to the time period of the visual image data, a fully connected layer is used to perform embedding mapping, dimensionality of different non-visual image data is aligned, and then they are spliced ​​together.

[0027] The visual feature V and the non-visual feature T described in step S4 are concatenated in the feature dimension to form a comprehensive feature vector F=Concat(V,T), where Concat(·) is a concat connection operation.

[0028] Further, the step S5 is specifically as follows:

[0029] S501: Artificially define N query vectors to be learned {q i |i=1,2,…,N},construct matrix Q=[q 1 ,q 2 ,…,q N ]; where N is a positive integer set artificially;

[0030] S502: Using the self-attention mechanism, the matrix Q interacts with the comprehensive feature vector F to obtain the attention feature;

[0031] Specifically,

[0032] S5021: Perform linear transformation on the query vector and the comprehensive feature vector to obtain the query matrix, key matrix, and value matrix, respectively:

[0033] Q′=Q·W Q ; K = F·W K ; V = F·WV ;

[0034] Among them, W Q , W K , W V is a trainable weight matrix.

[0035] S5022: Based on the query matrix, key matrix, and value matrix, the attention feature F′=Attention(Q′,K,V) is calculated using the single-head attention mechanism: where, softmax(·) is the softmax function operation, d k is the number of columns of Q′ and K.

[0036] Preferably, in order to improve the expression ability of the model, the visual image data and the non-visual image data are not spliced ​​during the processing in step S203 and step S4, and are respectively assigned to a separate attention mechanism according to the data source, and a multi-head attention mechanism is adopted, assuming that it is divided into H heads {head 1 ,head 2 ,…,head H}, each head calculates the attention feature separately and then concatenates the results to obtain the attention feature F′=MultiHeadAttention(Q′,K,V); where MultiHeadAttention(Q′,K,V)=Concat(head 1 ,head 2 ,…,head H )W o ; The calculation method of each head is the same as that of the single-head attention mechanism Atttention(·), W o is the output trainable linear transformation matrix.

[0037] S503: manually set a prompt word Prompt according to the embodied intelligent vehicle, and input the prompt word into the pre-trained large language model to obtain a text embedding vector P=LLM_Embedding(Prompt); wherein LLM_Embedding(·) is the embedding processing of the large language model;

[0038] S504: Map the text embedding vector P to the same dimension as the attention feature F′ through linear transformation to obtain the text feature P′=PW P ; Among them, W P is a trainable weight matrix;

[0039] S505: Use the single-head attention mechanism to interact the mapped text feature P′ with the attention feature F′ to obtain the fusion feature F″=Attention(Q P′ ,KF′ ,V F′ );in, is a trainable weight matrix.

[0040] Furthermore, the step S6 is specifically as follows: the prediction output module uses the large language model therein to receive the fusion feature F″ of the feature fusion module as input to generate a prediction result.

[0041] Preferably, the step S6 needs to be fine-tuned, that is, the difference between the predicted result and the true answer GroundTruth is used to define the loss function Loss using the open answer, that is,

[0042] Loss = L generation (LLM(F″), GroundTruth)

[0043] Among them, LLM(·) is the prediction result of the large language model, L generation (·) Adopt cross entropy loss function.

[0044] The beneficial effects of the present invention are: providing an embodied intelligent vehicle collision prediction system and method based on multimodal fusion, fusing the features of multimodal data through the Q-Former model, and organically embedding it into a large language model, thereby achieving high-precision prediction of collisions between embodied intelligent vehicles and pedestrians. At the same time, with the help of the linkage of cloud servers and the parallel transmission process of image and non-image data, the edge computing capabilities of embodied intelligent vehicles can be well utilized to improve the real-time performance of calculations. The method of the present invention has broad application prospects in the fields of intelligent transportation and autonomous driving, and can significantly improve traffic safety and system intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to make the purpose and technical solution of the present invention more clear, the present invention provides the following drawings for explanation:

[0046] Figure 1 It is a flow chart of the collision prediction method of embodied intelligent vehicle based on multi-modal fusion of the present invention;

[0047] Figure 2 It is a framework diagram of the embodied intelligent vehicle collision prediction system based on multimodal fusion in the local mode of Example 1 of the present invention; wherein 1 is the embodied intelligent vehicle, 2 is the terminal, 1101 is the multimodal sensor, 1102 is the data storage unit, 1103 is the data preprocessing module, 1104 is the feature extraction module, 1105 is the feature fusion module, 1106 is the prediction output module, 11041 is the visual feature extraction module, 11042 is the non-visual feature extraction module, and the arrow direction is the data flow direction;

[0048] Figure 32 is a framework diagram of an embodied intelligent vehicle collision prediction system based on multimodal fusion in a cloud mode of Example 2 of the present invention; wherein 1 is an embodied intelligent vehicle, 2 is a cloud server, 1201 is a multimodal sensor, 2202 is a data storage unit, 2203 is a data preprocessing module, 2204 is a feature extraction module, 2205 is a feature fusion module, 2206 is a prediction output module, 22041 is a visual feature extraction module, 22042 is a non-visual feature extraction module, and the direction of the arrow is the data flow direction;

[0049] Figure 4 It is a framework diagram of the embodied intelligent vehicle collision prediction system based on multimodal fusion in the linkage mode of Example 4 of the present invention; wherein 1 is the embodied intelligent vehicle, 2 is the cloud server, 1301 is the multimodal sensor, 1302 is the non-visual data preprocessing module, 1303 is the non-visual feature extraction module, 2302 is the data storage unit, 2203 is the visual data preprocessing module, 2304 is the visual feature extraction module, 2305 is the feature fusion module, 2306 is the prediction output module, and the direction of the arrow is the data flow. DETAILED DESCRIPTION

[0050] In order to make the purpose and technical solution of the present invention more clearly understood, the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0051] Example 1: With respect to a smart food delivery robot in a certain restaurant, collisions between the smart food delivery robot and guests are often caused by the guests' walking or movement during the food delivery process. In order to avoid such accidents, and considering that the smart food delivery robot itself has an integrated processor with relatively good performance, the present invention provides a "system and method for embodied smart vehicle collision prediction based on multimodal fusion in local mode", where the embodied smart vehicle (1) corresponds to the smart food delivery robot.

[0052] Specific, combined Figure 2, an embodied intelligent vehicle collision prediction system based on multimodal fusion in local mode, characterized in that it is composed of a multimodal sensor (1101), a data storage unit (1102), a data preprocessing module (1103), a feature extraction module (1104), a feature fusion module (1105), and a prediction output module (1106); the multimodal sensor (1101) includes a visual image sensor and a non-visual image sensor, which respectively collect visual image data and non-visual image data according to a period; the data preprocessing module (1103) is respectively connected to the multimodal sensor (1101) and the feature extraction module (1104), and is used for preprocessing the visual image data and the non-visual image data; the feature extraction module (1104) includes a visual feature extraction module (11041) and a non-visual feature extraction module (11042); the visual feature extraction module (11041) and the non-visual feature extraction module (11042) respectively use deep learning networks to realize feature extraction of corresponding pre-processed visual image data and non-visual image data; the feature fusion module (1105) is respectively connected to the feature extraction module (1104) and realizes the fusion of visual and non-visual features related to the collision prediction task by using the encoding of the self-attention mechanism; the prediction output module (1106) is connected to the feature fusion module (1105) and realizes result prediction based on a large language model; the data storage unit (1102) is respectively connected to the data acquisition module and the prediction output module (1106) for data storage.

[0053] The visual feature extraction module (11041) uses a pre-trained Video-VIT-base model to extract spatiotemporal features; the non-visual feature extraction module (11042) uses a pre-trained Transformer model based on the Video-VIT-small model architecture. Extracting spatiotemporal features

[0054] The feature fusion module (1105) adopts a feature fusion model based on Q-Former.

[0055] The embodied intelligent vehicle collision prediction system based on multimodal fusion in the local mode is entirely arranged on the embodied intelligent vehicle (1), and a wireless communication module is installed on the embodied intelligent vehicle (1) for communicating with the terminal (2).

[0056] The embodied intelligent vehicle collision prediction method based on multimodal fusion in local mode includes the following steps:

[0057] S1: Using a multimodal sensor (1101) to collect visual image data and non-visual image data at a period of 1 second, and storing them in a data storage unit (1102) in a time sequence.

[0058] S2: The data preprocessing module (1103) preprocesses the visual image data and non-visual image data in the data storage unit (1102).

[0059] The pretreatment comprises the following steps:

[0060] S201: Identify whether there is a pedestrian in the visual image data at the current moment, if not, wait for the visual image data at the next moment, otherwise continue to execute the following steps;

[0061] S202: resizing the image data in the visual image data and performing normalization processing;

[0062] S203: According to the time period of the visual image data, a fully connected layer is used to perform embedding mapping, dimensionality of different non-visual image data is aligned, and then they are spliced ​​together.

[0063] S3: The feature extraction module (1104) extracts features from the preprocessed visual image data and non-visual image data respectively to obtain visual image features and non-visual image features.

[0064] S4: Concatenate the visual image features and the non-visual image features in the feature dimension to form a comprehensive feature vector F = Concat(V, T), where Concat(·) is a concat connection operation.

[0065] S5: Introduce a set of query vectors to be learned, and use the feature fusion module (1105) to fuse the query vectors with the comprehensive feature vectors to obtain fused features.

[0066] Specifically:

[0067] S501: Artificially define N query vectors to be learned {q i |i=1,2,…,N},construct matrix Q=[q 1 ,q 2 ,…,q N ]; where N is a positive integer set artificially;

[0068] S502: Using the self-attention mechanism, the matrix Q interacts with the comprehensive feature vector F to obtain the attention feature;

[0069] Specifically,

[0070] S5021: Perform linear transformation on the query vector and the comprehensive feature vector to obtain the query matrix, key matrix, and value matrix respectively:

[0071] Q′=Q·W Q ; K = F·W K; V = F·W V ;

[0072] Among them, W Q , W K , W V is a trainable weight matrix.

[0073] S5022: Based on the query matrix, key matrix, and value matrix, the attention feature F′=Attention(Q′,K,V) is calculated using the single-head attention mechanism: where, softmax(·) is the softmax function operation, d k is the number of columns of Q′ and K.

[0074] S503: manually set a prompt word Prompt according to the embodied intelligent vehicle - "Please predict whether the embodied intelligent vehicle will collide with the guest, and output the collision probability and collision time", and input the prompt word into the pre-trained large language model to obtain a text embedding vector P = LLM_Embedding(Prompt); wherein LLM_Embedding(·) is the embedding processing of the large language model;

[0075] S504: Map the text embedding vector P to the same dimension as the attention feature F′ through linear transformation to obtain the text feature P′=PW P ; Among them, W P is a trainable weight matrix;

[0076] S505: Use the single-head attention mechanism to interact the mapped text feature P′ with the attention feature F′ to obtain the fusion feature F″=Attention(Q P′ ,K F′ ,V F′ );in, is a trainable weight matrix.

[0077] S6: Input the fused features into the prediction output module (1106) for high-level semantic understanding and prediction, and obtain collision prediction results.

[0078] Specifically, the prediction output module (1106) uses the large language model therein to receive the fused feature F″ of the feature fusion module (1105) as input, and generates the prediction results of the collision probability and the collision time.

[0079] When training the model parameters of the embodied intelligent vehicle collision prediction system based on multimodal fusion in the local mode, the step S6 needs to be fine-tuned, that is, the difference between the prediction result and the true answer GroundTruth is used to define the loss function Loss using an open answer, that is,

[0080] Loss = L generation (LLM(F″), GroundTruth)

[0081] Among them, LLM(·) is the prediction result of the large language model, L generation (·) Adopt cross entropy loss function.

[0082] S7: The embodied intelligent vehicle (1) performs motion control under the reinforcement learning framework based on the collision prediction results.

[0083] Example 2: For a sorting robot in a certain intelligent warehouse, collisions between sorting robots may occur due to the movement of other sorting robots during the sorting process. In order to avoid such accidents and considering that the computing power of the sorting robot itself is not strong, the present invention provides a "collision prediction system and method for embodied intelligent vehicles based on multimodal fusion in cloud mode", where the embodied intelligent vehicle (1) corresponds to the sorting robot.

[0084] Specific, combined Figure 3 The embodied intelligent vehicle collision prediction system based on multimodal fusion in cloud mode is arranged on the embodied intelligent vehicle (1) and the cloud server (2), and communicates through wireless communication modules respectively installed on the embodied intelligent vehicle (1) and the cloud server (2).

[0085] The system is composed of a multimodal sensor (1201), a data storage unit (2202), a data preprocessing module (2203), a feature extraction module (2204), a feature fusion module (2205), and a prediction output module (2206); the multimodal sensor (1201) is installed on an embodied intelligent vehicle (1); the data storage unit (2202), the data preprocessing module (2203), the feature extraction module (2204), the feature fusion module (2205), and the prediction output module (2206) are installed on a cloud server (2).

[0086] The multimodal sensor (1201) comprises a visual image sensor and a non-visual image sensor, which respectively collect visual image data and non-visual image data in a periodic manner; the data preprocessing module (2203) is connected to the multimodal sensor (1201) via a wireless communication module, and is directly connected to the feature extraction module (2204), and is used for preprocessing the visual image data and the non-visual image data; the feature extraction module (2204) comprises a visual feature extraction module (22041) and a non-visual feature extraction module (22042); the visual feature extraction module (22041) and the non-visual feature extraction module (22042) are connected to each other; The feature extraction module (22042) respectively uses a deep learning network to realize feature extraction of the corresponding pre-processed visual image data and non-visual image data; the feature fusion module (2205) is respectively connected to the feature extraction module (2204), and uses the encoding of the self-attention mechanism to realize the fusion of visual and non-visual features related to the collision prediction task; the prediction output module (2206) is connected to the feature fusion module (2205) to realize result prediction based on a large language model; the data storage unit (2202) is respectively connected to the data acquisition module and the prediction output module (2206) for data storage.

[0087] The visual feature extraction module (22041) uses a pre-trained Video-VIT-base model to extract spatiotemporal features; the non-visual feature extraction module (22042) uses a pre-trained Transformer model based on the Video-VIT-small model architecture. Extracting spatiotemporal features

[0088] The feature fusion module (2205) adopts a feature fusion model based on Q-Former.

[0089] The embodied intelligent vehicle collision prediction method based on multimodal fusion in the cloud mode includes the following steps:

[0090] S1: The embodied intelligent vehicle (1) uses a multimodal sensor (1201) to collect visual image data and non-visual image data at a cycle of 1 second, and sends the data to the cloud server (2) in a time sequence and stores it in the data storage unit (2202).

[0091] S2: The data preprocessing module (2203) preprocesses the visual image data and non-visual image data in the data storage unit (2202).

[0092] The pretreatment comprises the following steps:

[0093] S201: Identify whether there is a pedestrian in the visual image data at the current moment, if not, wait for the visual image data at the next moment, otherwise continue to execute the following steps;

[0094] S202: resizing the image data in the visual image data and performing normalization processing;

[0095] S203: Align the time periods of the visual image data and align the dimensions of the different non-visual image data.

[0096] S3: The feature extraction module (2204) extracts features from the preprocessed visual image data and non-visual image data respectively to obtain visual image features and non-visual image features.

[0097] S4: Introduce a set of query vectors to be learned, and use the feature fusion module (2205) to fuse the query vectors with the comprehensive feature vectors to obtain fused features.

[0098] Specifically:

[0099] S401: Artificially define N query vectors to be learned {q i |i=1,2,…,N},construct matrix Q=[q 1 ,q 2 ,…,q N ]; where N is a positive integer set artificially;

[0100] S402: Using the self-attention mechanism, the matrix Q interacts with the comprehensive feature vector F to obtain the attention feature;

[0101] Specifically,

[0102] S4021: Perform linear transformation on the query vector and the comprehensive feature vector to obtain the query matrix, key matrix, and value matrix respectively:

[0103] Q′=Q·W Q ; K = F·W K ; V = F·W V ;

[0104] Among them, W Q , W K , W V is a trainable weight matrix.

[0105] S4022: Based on the query matrix, key matrix, and value matrix, the attention feature F′=MultiHeadAttention(Q v ,K,V): where MultiHeadAttention(Q′,K,V)=Concat(head 1 ,head 2 ,…,head H )Wo , W o is the output trainable linear transformation matrix, H is the dimension of the data source, {head 1 ,head 2 ,…,head H The calculation method for each head in} is siftmax(·) is the softmax function operation, d k is the number of columns of Q′ and K.

[0106] S403: manually set a prompt word Prompt according to the embodied intelligent vehicle - "Please predict whether the embodied intelligent vehicle will collide with the guest, and output the collision probability and collision time", and input the prompt word into the pre-trained large language model to obtain a text embedding vector P = LLM_Embedding(Prompt); wherein LLM_Embedding(·) is the embedding processing of the large language model;

[0107] S404: Map the text embedding vector P to the same dimension as the attention feature F′ through linear transformation to obtain the text feature P′=PW P ; Among them, W P is a trainable weight matrix;

[0108] S405: Use the single-head attention mechanism to interact the mapped text feature P′ with the attention feature F′ to obtain the fusion feature F″=Attention(Q P′ ,K F′ ,V F′ );in, is a trainable weight matrix.

[0109] S5: Input the fused features into the prediction output module (2206) for high-level semantic understanding and prediction, and obtain collision prediction results.

[0110] Specifically, the prediction output module (2206) uses the large language model therein to receive the fused feature F″ of the feature fusion module (2205) as input, and generates the prediction results of the collision probability and the collision time.

[0111] When training the model parameters of the embodied intelligent vehicle collision prediction system based on multimodal fusion in the cloud mode, the step S5 needs to be fine-tuned, that is, the difference between the prediction result and the true answer GroundTruth is used to define the loss function Loss using an open answer, that is,

[0112] Loss = L generation (LLM(F″), GroundTruth)

[0113] Among them, LLM(·) is the prediction result of the large language model, L generation (·) Adopt cross entropy loss function.

[0114] S7: The cloud server (2) performs action screening under the reinforcement learning framework according to the collision prediction results, and drives the embodied intelligent vehicle (1) to perform actions.

[0115] Embodiment 3: With respect to a certain intelligent driving car, since pedestrians often collide with the intelligent driving car during driving, in order to avoid such accidents, taking the intelligent driving car into consideration, the present invention provides a "collision prediction system and method for an embodied intelligent vehicle based on multimodal fusion in a linkage mode", and the object corresponding to the embodied intelligent vehicle (1) is the intelligent driving car.

[0116] Specific, combined Figure 3 The embodied intelligent vehicle collision prediction system based on multimodal fusion in linkage mode is arranged on the embodied intelligent vehicle (1) and the cloud server (2), and communicates through wireless communication modules respectively installed on the embodied intelligent vehicle (1) and the cloud server (2).

[0117] The system is composed of a multimodal sensor (1301), a non-visual image preprocessing module (1302), a non-visual feature extraction module (1303), a data storage unit (2302), a visual image data preprocessing module (2303), a visual feature extraction module (2304), a feature fusion module (2305), and a prediction output module (2306). The multimodal sensor (1301), the non-visual image preprocessing module (1302), and the non-visual feature extraction module (1303) are installed on the embodied intelligent vehicle (1); the data storage unit (2302), the visual image data preprocessing module (2303), the visual feature extraction module (2304), the feature fusion module (2305), and the prediction output module (2306) are installed on the cloud server (2).

[0118] The multimodal sensor (1301) includes a visual image sensor and a non-visual image sensor, which respectively collect visual image data and non-visual image data in a periodic manner; the non-visual image preprocessing module (1302) is connected to the multimodal sensor (1301) and the non-visual feature extraction module (1303); the data storage unit (2302) is connected to the feature fusion module (2305), and the features of the non-visual image are directly transmitted to the feature fusion module (2305) for feature fusion; the visual data preprocessing module (2303) is connected to the multimodal sensor (1301) through a wireless communication module, and is directly connected to the visual feature extraction module (2304), and is used to perform visual image processing. Preprocessing of image data; the visual feature extraction module (2304) and the non-visual feature extraction module (1303) respectively use deep learning networks to realize feature extraction of the corresponding preprocessed visual image data and non-visual image data; the feature fusion module (2305) is respectively connected to the feature extraction module (2304), and uses the encoding of the self-attention mechanism to realize the fusion of visual and non-visual features related to the collision prediction task; the prediction output module (2306) is connected to the feature fusion module (2305), and realizes result prediction based on the large language model; the data storage unit (2302) is respectively connected to the data acquisition module and the prediction output module (2306) for data storage.

[0119] The visual feature extraction module (2304) uses a pre-trained Video-VIT-base model to extract features about extraction time and space; the non-visual feature extraction module (1303) uses a pre-trained Transformer model based on the Video-VIT-small model architecture.

[0120] Extracting spatiotemporal features

[0121] The feature fusion module (2305) adopts a feature fusion model based on Q-Former.

[0122] The embodied intelligent vehicle collision prediction method based on multimodal fusion in the cloud mode includes the following steps:

[0123] S1: Using a multimodal sensor (1301) to periodically collect visual image data and non-visual image data, and sending the visual image data to a cloud server (2) in a time sequence to store it in a data storage unit (2302).

[0124] S2: The non-visual data preprocessing module (1302) directly preprocesses the non-visual image data collected by the multimodal sensor (1301).

[0125] S3: The non-visual feature extraction module (1303) extracts features from the non-visual image data, obtains non-visual image features, and sends them to the cloud server (2) to be stored in the data storage unit (2302).

[0126] S4: The visual data preprocessing module (2303) preprocesses the visual image data in the data storage unit (2302).

[0127] S5: The visual feature extraction module (2304) extracts features from the preprocessed visual image data to obtain visual image features.

[0128] S4: Concatenate the visual image features and non-visual image features in the feature dimension to form a comprehensive feature vector.

[0129] S5: Introduce a set of query vectors to be learned, and use the feature fusion module (2305) to fuse the query vectors with the comprehensive feature vectors to obtain fused features.

[0130] S6: Input the fused features into the prediction output module (2306) for high-level semantic understanding and prediction, and obtain collision prediction results.

[0131] S7: The cloud server (2) performs action screening under the reinforcement learning framework according to the collision prediction results, and drives the embodied intelligent vehicle (1) to perform actions.

[0132] The detailed steps are almost the same as those in Example 1 and will not be described again here.

[0133] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. The embodied intelligent vehicle collision prediction system based on multimodal fusion is characterized by: It is composed of a multimodal sensor, a data storage unit, a data preprocessing module, a feature extraction module, a feature fusion module, and a prediction output module; the multimodal sensor includes a visual image sensor and a non-visual image sensor, which respectively collect visual image data and non-visual image data according to a period; the data preprocessing module is respectively connected to the multimodal sensor and the feature extraction module, and is used for preprocessing the visual image data and the non-visual image data; the feature extraction module includes a visual feature extraction module and a non-visual feature extraction module; the visual feature extraction module and the non-visual feature extraction module respectively use a deep learning network to realize the feature extraction of the corresponding preprocessed visual image data and the non-visual image data; the feature fusion module is respectively connected to the feature extraction module, and the encoding of the self-attention mechanism is used to realize the fusion of visual and non-visual features related to the collision prediction task; the prediction output module is connected to the feature fusion module, and the result prediction is realized based on the large language model; the data storage unit is respectively connected to the data acquisition module and the prediction output module for data storage; The visual feature extraction module adopts the pre-trained Video-VIT-base model; the non-visual feature extraction module adopts the pre-trained Transformer model based on the Video-VIT-small model architecture; the feature fusion module adopts the feature fusion model based on Q-Former; The embodied intelligent vehicle collision prediction system based on multimodal fusion is entirely arranged on the embodied intelligent vehicle, and a wireless communication module is installed on the embodied intelligent vehicle for communicating with the terminal.

2. The embodied intelligent vehicle collision prediction system based on multimodal fusion according to claim 1, characterized in that: The embodied intelligent vehicle collision prediction system based on multimodal fusion is arranged on the embodied intelligent vehicle and the cloud server, and communicates through wireless communication modules installed on the embodied intelligent vehicle and the cloud server respectively.

3. The embodied intelligent vehicle collision prediction system based on multimodal fusion according to claim 2, characterized in that: The multimodal sensor is installed on the embodied intelligent vehicle; the data storage unit, data preprocessing module, feature extraction module, feature fusion module, and prediction output module are installed on the cloud server.

4. The embodied intelligent vehicle collision prediction system based on multimodal fusion according to claim 2, characterized in that: The data preprocessing module is divided into a visual image preprocessing module and a non-visual image preprocessing module according to visual image data and non-visual image data; the multimodal sensor, non-visual image preprocessing module, and non-visual feature extraction module are installed on the embodied intelligent vehicle; the visual image preprocessing module, visual feature extraction module, data storage unit, visual feature fusion module, and prediction output module are installed on a cloud server; wherein, the visual image preprocessing module is connected to the visual feature extraction module, the non-visual image preprocessing module is respectively connected to the multimodal sensor and the non-visual feature extraction module, and the data storage unit is connected to the feature fusion module, and the features of the non-visual image are directly transmitted to the feature fusion module for feature fusion.

5. The embodied intelligent vehicle collision prediction method based on multimodal fusion applied to claims 1 to 4 is characterized in that: The steps include: S1: using a multimodal sensor to periodically collect visual image data and non-visual image data, and store them in a data storage unit in a time sequence; S2: The data preprocessing module preprocesses the visual image data and the non-visual image data in the data storage unit; S3: The feature extraction module extracts features from the preprocessed visual image data and non-visual image data to obtain visual image features and non-visual image features; S4: concatenate visual image features and non-visual image features in the feature dimension to form a comprehensive feature vector; S5: introduce a set of query vectors to be learned, and use the feature fusion module to fuse the query vectors with the comprehensive feature vectors to obtain fused features; S6: Input the fused features into the prediction output module for high-level semantic understanding and prediction, and obtain the collision prediction result; The pre-processing described in step S2 comprises the following steps: S201: Identify whether there is a pedestrian in the visual image data at the current moment, if not, wait for the visual image data at the next moment, otherwise continue to execute the following steps; S202: resizing the image data in the visual image data and performing normalization processing; S203: according to the time period of the visual image data, embedding mapping is performed using a fully connected layer, dimensionally aligning different non-visual image data, and then splicing them together; The step S5 is specifically as follows: S501: Artificially define N query vectors to be learned {q i |i=1,2,…,N},construct matrix Q=[q1,q2,…,q N ]; where N is a positive integer set artificially; S502: Use the self-attention mechanism to interact the matrix Q with the comprehensive feature vector F to obtain the attention feature F′; S503: manually set a prompt word Prompt according to the embodied intelligent vehicle, and input the prompt word into the pre-trained large language model to obtain a text embedding vector P=LLM_Embedding(Prompt); wherein LLM_Embedding(·) is the embedding processing of the large language model; S504: Map the text embedding vector P to the same dimension as the attention feature F′ through linear transformation to obtain the text feature P′=PW P ; Among them, W P is a trainable weight matrix; S505: Use the single-head attention mechanism to interact the mapped text feature P′ with the attention feature F′ to obtain the fusion feature F″=Attention(Q P′ ,K F′ ,V F′ );in, is a trainable weight matrix; The step S6 is specifically as follows: the prediction output module uses the large language model therein to receive the fusion feature F″ of the feature fusion module as input to generate a prediction result.

6. The embodied intelligent vehicle collision prediction method based on multimodal fusion according to claim 5, characterized in that: The step S502 is specifically as follows: S5021: Perform linear transformation on the query vector and the comprehensive feature vector to obtain the query matrix, key matrix, and value matrix, respectively: Q′=Q·W Q ;K=F·W K ;V=F·W V ; Among them, W Q , W K , W V is a trainable weight matrix; S5022: Based on the query matrix, key matrix, and value matrix, the attention feature F′=Attention(Q′,K,V) is calculated using the single-head attention mechanism: where, softmax(·) is the softmax function operation, d k is the number of columns of Q′ and K.

7. The embodied intelligent vehicle collision prediction method based on multimodal fusion according to claim 6, characterized in that: The visual image data and the non-visual image data are not spliced ​​in step S203 and step S4. According to the data source, they are respectively assigned to a separate attention mechanism. A multi-head attention mechanism is adopted, assuming that they are divided into H heads {head1, head2, …, head H }, each head calculates the attention feature separately and then concatenates the results to obtain the attention feature F′=MultiHeadAttention(Q′,K,V); where MultiHeadAttention(Q′,K,head)=Concat(head1,head2,…,head H )W o ; The calculation method of each head is the same as that of the single-head attention mechanism Atttention(·), W o is the output trainable linear transformation matrix.

8. The embodied intelligent vehicle collision prediction method based on multimodal fusion according to claim 5, characterized in that: The step S6 needs to be fine-tuned, that is, the difference between the predicted result and the true answer GroundTruth is used to define the loss function Loss using the open answer, that is, Loss=L generation (LLM(F″),GroundTruth) Among them, LLM(·) is the prediction result of the large language model, L generation (·) Adopt cross entropy loss function.

Images