Automatic driving safety assistance method and system based on multi-modal fusion

Abstract

The application relates to a multi-modal fusion-based automatic driving safety assistance method and system, and belongs to the field of automatic driving. The system is composed of a multi-modal sensor, a preprocessing unit, a feature encoder, a time sequence merging module, a context attention module, a classification module, a data storage unit and an output unit. The method comprises the following steps: S1: collecting visual image data and non-visual image data; S2: preprocessing the visual image data and the non-visual image data; S3: performing feature extraction by using the feature encoder; S4: performing key event division by using the time sequence merging module; S5: performing feature fusion by using the context attention module; S6: inputting the fused features into the classification module to predict the pedestrian behavior type; and S7: storing the predicted pedestrian behavior type result and feeding back the result to a car terminal system. The method can effectively identify and aggregate key events related to pedestrian behavior, and improve the accuracy and efficiency of pedestrian behavior intention prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to a multi-modal fusion based autonomous driving safety assistance method and system, belonging to the field of autonomous driving, and in particular to a multi-modal fusion learning based autonomous driving safety assistance. BACKGROUND

[0002] With the rapid development of autonomous driving technology, the interaction between vehicles and other road users in complex road environments becomes more frequent and important. As the most vulnerable participants on the road, the safety of pedestrians has always been the focus of autonomous driving and assisted driving systems. In the real world, accurately predicting whether a pedestrian has the intention to cross the street is of great significance to road traffic safety. Autonomous vehicles need to have the ability to perceive and understand the intentions of pedestrians in advance, so that appropriate measures such as slowing down, stopping or changing lanes can be taken in time before the pedestrian makes a crossing behavior. However, the behavior of pedestrians is often influenced by a variety of factors, such as traffic signs, traffic signals, road conditions, weather conditions, the state of the pedestrian himself, the speed of the ego vehicle, and the behavior of other road users. The complexity and diversity of these factors pose great challenges to accurate prediction of pedestrian crossing intentions.

[0003] Traditional pedestrian intention prediction methods mainly rely on the analysis of pedestrian motion trajectories, postures and facial expressions. However, these methods often struggle to effectively capture key events related to pedestrian behavior when dealing with high-redundancy video frames, resulting in poor prediction performance. In addition, in complex traffic environments, it is difficult to fully understand the intentions of pedestrians by relying solely on visual information. Therefore, there is an urgent need for a method that can effectively integrate multi-modal information, highlight key dynamic changes in pedestrian behavior, and capture pedestrian intentions in the time dimension, to improve the accuracy and reliability of pedestrian crossing intention prediction. SUMMARY

[0004] The present application aims to overcome the shortcomings of the prior art and provide a multi-modal fusion based autonomous driving safety assistance method and system for assisting in judging pedestrian crossing intentions. This method aims to improve the accuracy and efficiency of pedestrian behavior intention prediction by introducing real-time acquisition of visual image data and non-visual data of time series data based on a multi-modal fusion based autonomous driving safety assistance system sensor, introducing it into the temporal merging module (TMM) and the contextual attention module (CAB), and then effectively identifying and aggregating key events related to pedestrian behavior.

[0005] To achieve the above purpose, the present application provides the following technical solutions:

[0006] The application discloses an automatic driving safety auxiliary system based on multi-modal fusion, which is characterized by being composed of a multi-modal sensor, a preprocessing unit, a feature encoder, a time sequence merging module, a context attention module, a classification module, a data storage unit and an output unit; the multi-modal sensor comprises a visual image sensor and a non-visual image sensor, which respectively collect visual image data and non-visual image data according to a cycle; the preprocessing unit is connected with the data storage unit and the feature encoder, and is used for preprocessing the visual image data and the non-visual image data; the feature encoder comprises a visual encoder and a non-visual encoder; the visual encoder and the non-visual encoder respectively adopt a deep learning network to realize feature extraction of the preprocessed visual image data and non-visual image data; the time sequence merging module is connected with the feature encoder, is realized based on a deep learning network, and is used for event division; the context attention module is connected with the time sequence merging module, is realized based on a deep learning network, and is used for realizing feature fusion of events; the classification module is connected with the context attention module, is realized based on a classifier, and is used for dividing a pedestrian behavior type; the data storage unit is connected with the multi-modal sensor, the classification module and the output unit, and is used for data storage; and the output unit is a communication module, which is used for feeding back the pedestrian behavior type to a car terminal system.

[0007] Further, the visual encoder adopts a pre-trained Video-VIT-base model to extract visual features, and the non-visual encoder adopts a Transformer model based on a Video-VIT-small model architecture to extract non-visual features.

[0008] Preferably, the time sequence merging module is realized based on a KNN (K-Nearest Neighbor) network.

[0009] Further, the multi-modal sensor, the preprocessing unit, the feature encoder, the time sequence merging module, the context attention module, the classification module, the data storage unit and the output unit are arranged on a vehicle terminal.

[0010] Preferably, the multi-modal sensor is arranged on the vehicle terminal, and the preprocessing unit, the feature encoder, the time sequence merging module, the context attention module, the classification module, the data storage unit and the output unit are arranged on a cloud server.

[0011] The application further discloses an automatic driving safety auxiliary method based on multi-modal fusion, which comprises the following steps:

[0012] S1: collecting visual image data and non-visual image data periodically by using a vehicle-mounted multi-modal sensor, and storing them in a data storage unit in time sequence;

[0013] S2: preprocessing the visual image data and non-visual image data in the data storage unit by a preprocessing unit;

[0014] S3: extracting features from the preprocessed visual image data and non-visual image data respectively by using a feature encoder, to obtain visual image features and non-visual image features;

[0015] S4: dividing the visual image features and non-visual image features into key events respectively by using a time sequence merging module, to obtain visual image events and non-visual image events;

[0016] S5: fusing the visual image events and non-visual image events by using a context attention module, to obtain fused features;

[0017] S6: inputting the fused features into a classification module to predict the type of pedestrian behavior;

[0018] S7: storing the result of predicting the type of pedestrian behavior in the data storage unit, and feeding back to the automobile terminal system by using an output unit.

[0019] Further, the preprocessing of step S2 comprises the following steps:

[0020] S201: identifying whether there is a pedestrian in the current time visual image data, if not, waiting for the next time visual image data, otherwise, continuing to execute the following steps;

[0021] S202: performing image segmentation on the pedestrians in the visual image data, cropping and resizing the local image of each target pedestrian, and performing normalization processing;

[0022] S203: embedding mapping by using a fully connected layer according to the time period of the visual image data, aligning the dimensions of different non-visual image data, and then splicing them together.

[0023] The feature encoder in step S3 is divided into a visual encoder and a non-visual encoder, and the T-frame visual image features obtained by the visual encoder are denoted as The T-frame non-visual image features obtained by the non-visual encoder are denoted as

[0024] Further, the step S4 is based on a density peak clustering algorithm, specifically:

[0025] S401: calculating the local density of the visual image features and non-visual image features of any frame by using KNN;

[0026] Specifically, visual image features Visual image features I t Corresponding local density p t is: Non-visual image features Non-visual image features J t Corresponding local density is: KNN(I t , F v ) represents K nearest neighbors of I v in F t excluding I t itself, and KNN(J nv , F t ) is similar; K is a positive integer.

[0027] S402: Calculate the shortest distance from visual image features and non-visual image features of any frame to any visual image features and non-visual image features with higher density;

[0028] Specifically, the shortest distance d t from visual image features I t of the t-th frame to any visual image features with higher density is:

[0029] Wherein, m∈[1, T] is an index;

[0030] Similarly, the shortest distance d from non-visual image features J t of the t-th frame to any non-visual image features with higher density is:

[0031] Wherein, m∈[1, T] is an index.

[0032] S403: Select the cluster centers from the visual image features and non-visual image features with higher local density and larger shortest distance respectively, and perform clustering processing on the remaining visual image features and non-visual image features, to obtain the clustering sets of visual image events M and non-visual image events Wherein, M and are the total number of corresponding events.

[0033] Further, step S5 is specifically:

[0034] S501: Use visual image events Mthe attention weight of each event is calculated according to the self-event similarity of the event j ; specifically,

[0035]

[0036] wherein, represents the similarity between events E i and E M , is a trainable weight matrix, and M is the total number of corresponding events;

[0037] S502: In the manner of S501, the attention weight of each event of the non-visual image event is calculated

[0038] S503: The visual image events in the time period are fused to obtain the visual image event fusion feature H v ; specifically,

[0039]

[0040] wherein, is a trainable weight;

[0041] S504: In the manner of S503, the non-visual image events in the time period are fused to obtain the non-visual image event fusion feature

[0042] S505: The visual image event fusion feature H v and the non-visual image event fusion feature are further fused to obtain the fusion feature P; specifically,

[0043]

[0044] wherein, is a trainable weight.

[0045] Further, the step S6 is specifically: inputting the fused feature P into a classification module to predict the pedestrian behavior type.

[0046] Preferably, the classification module is a three-layer MLP full connection layer; and the corresponding training loss function is a cross-entropy loss function.

[0047] The application has the beneficial effects that: a multi-modal fusion-based automatic driving safety assistance method and system are provided, through time sequence merging and context attention mechanism, key dynamic changes of pedestrian behavior can be effectively captured, interference of redundant information is reduced, and prediction accuracy is improved; at the same time, time sequence features are aggregated and simplified, real-time prediction can be realized with lower calculation cost; in addition, visual image features and non-visual image features are fused, stronger adaptability and robustness can be achieved for different scenes and environments. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to make the purpose and technical scheme of the application clearer, the application is described below with the help of the following drawings:

[0049] Figure 1 The flow chart of the multi-modal fusion-based automatic driving safety assistance method of the application;

[0050] Figure 2 The automatic driving safety assistance system architecture diagram based on multi-modal fusion of the embodiment 1 of the application; the arrow is the data transmission direction;

[0051] Figure 3 The structural schematic diagram of the electronic device in the embodiment 3 of the application. DETAILED DESCRIPTION

[0052] In order to make the purpose and technical scheme of the application clearer, the application is described below with the help of the following drawings:

[0053] Embodiment 1: In the process of driving on the road, the automatic driving system can assist safe driving, and an important function is to assist the driver or independently judge whether the roadside pedestrian has the intention to cross the road, and take appropriate measures accordingly. To realize this function, the car needs to be upgraded at the device level, and the embodiment provides a multi-modal fusion-based automatic driving safety assistance system.

[0054] In combination with Figure 2 The multi-modal fusion-based automatic driving safety assistance system comprises a multi-modal sensor (1), a preprocessing unit (2), a feature encoder (3), a time sequence merging module (4), a context attention module (5), a classification module (6), a data storage unit (7), and an output unit (8).

[0055] The multi-modal sensor (1) comprises a visual image sensor (11) and a non-visual image sensor (12) for periodically collecting visual image data and non-visual image data, respectively; the preprocessing unit (2) is connected with the data storage unit (7) and the feature encoder (3) for preprocessing the visual image data and the non-visual image data; the feature encoder (3) comprises a visual encoder (31) and a non-visual encoder (32); the visual encoder (31) and the non-visual encoder (32) respectively use a deep learning network to realize feature extraction of the corresponding preprocessed visual image data and non-visual image data; the time sequence merging module (4) is connected with the feature encoder (3) and realized based on a deep learning network for event division; the context attention module (5) is connected with the time sequence merging module (4) and realized based on a deep learning network for realizing feature fusion of the event; the classification module (6) is connected with the context attention module (5) and realized based on a classifier for dividing the pedestrian behavior type; the data storage unit (7) is connected with the multi-modal sensor (1), the classification module (6) and the output unit (8) for data storage; and the output unit (8) is a communication module for feeding back the pedestrian behavior type to the car terminal system.

[0056] Further, the visual encoder (31) uses a pre-trained Video-VIT-base model to extract visual features; and the non-visual encoder (32) uses a Transformer model based on a Video-VIT-small model architecture to extract non-visual features.

[0057] The time sequence merging module (4) is realized based on a KNN network.

[0058] Embodiment 2: Based on the multi-modal fusion-based automatic driving safety auxiliary system given in Embodiment 1, the present embodiment provides a multi-modal fusion-based automatic driving safety auxiliary method for judging whether a roadside pedestrian has a crossing intention.

[0059] The multi-modal fusion-based automatic driving safety auxiliary method combines Figure 1 and comprises the following steps:

[0060] S1: The vehicle-mounted multi-modal sensor (1) collects visual image data and non-visual image data at a period of 0.5 seconds and stores them in the data storage unit (7) in time sequence.

[0061] S2: The preprocessing unit (2) preprocesses the visual image data and the non-visual image data in the data storage unit (7).

[0062] The preprocessing comprises the following steps:

[0063] S201: Identify whether there is a pedestrian in the current moment visual image data, if not, wait for the next moment of visual image data, otherwise continue to execute the following steps;

[0064] S202: Image segmentation of pedestrians in visual image data, cropping and resizing of local image of each target pedestrian, and normalization processing;

[0065] S203: According to the time period of visual image data, embedding mapping is carried out by using full connection layer, different non-visual image data are dimensionally aligned, and then they are spliced together.

[0066] S3: Feature extraction is carried out on the preprocessed visual image data and non-visual image data by using feature encoder (3) respectively, and visual image features and non-visual image features are obtained.

[0067] The feature encoder is divided into visual encoder and non-visual encoder, and the T-frame visual image features obtained by the visual encoder are denoted as The T-frame non-visual image features obtained by the non-visual encoder are denoted as

[0068] S4: Key event division is carried out on visual image features and non-visual image features respectively by using time sequence merging module (4), and visual image events and non-visual image events are obtained.

[0069] Specifically:

[0070] S401: Local density of visual image features and non-visual image features of any frame is calculated by using KNN;

[0071] Specifically, visual image features Visual image features I of the t-th frame t Corresponding local density p t is: Non-visual image features Non-visual image features J of the t-th frame t Corresponding local density is: Wherein, KNN(I t , F v ) represents K nearest neighbors in F v excluding itself I t , KNN(J t , F nv ) is similar; K is a positive integer.

[0072] S402: Calculate the shortest distance from the visual image feature and the non-visual image feature of any frame to any visual image feature and non-visual image feature with higher density;

[0073] Specifically, the shortest distance δ t from the visual image feature I t of the t-th frame to any visual image feature with higher density is:

[0074] Wherein, m ∈ [1, T] is the index;

[0075] Similarly, the shortest distance δ t from the non-visual image feature J of the t-th frame to any non-visual image feature with higher density is:

[0076] Wherein, m ∈ [1, T] is the index.

[0077] S403: Select the cluster centers from the visual image features and non-visual image features with higher local density and larger shortest distance respectively, and perform clustering processing on the remaining visual image features and non-visual image features, to obtain the clustering sets of visual image events ε = {E1, E2,..., EM} and non-visual image events M . Wherein, M and are the total number of corresponding events, which is set to 3.

[0078] S5: Use the context attention module (5) to fuse the features of the visual image events and the non-visual image events, to obtain the fused features.

[0079] Specifically:

[0080] S501: Calculate the attention weight α M of each event using the event similarity of the visual image events ε = {E1, E2,..., EM}; j Specifically:

[0081]

[0082] Wherein, represents the similarity between event E i and event E M , is a trainable weight matrix, and M is the total number of corresponding events;

[0083] S502: Calculate the attention weight α of each event of the non-visual image events in the manner of S501.

[0084] S503: Fuse visual image events in the time period to obtain visual image event fusion feature H v ; Specifically,

[0085]

[0086] Wherein, is a trainable weight;

[0087] S504: In the manner of S503, fuse non-visual image events in the time period to obtain non-visual image event fusion feature

[0088] S505: Fuse visual image event fusion feature H v and non-visual image event fusion feature to obtain fusion feature P; Specifically,

[0089]

[0090] Wherein, is a trainable weight.

[0091] S6: Input the fusion feature into the classification module (6) to predict the binary classification problem of whether the roadside pedestrian has the intention to cross the road.

[0092] The classification module (6) is a three-layer MLP full connection layer; and the corresponding training loss function is a cross-entropy loss function.

[0093] S7: Store the predicted pedestrian behavior type result into the data storage unit (7), and feed back to the car terminal system by using the output unit (8).

[0094] S8: The car terminal system decides whether to assist the driver to take emergency braking or deceleration measures according to whether the roadside pedestrian has the intention to cross the road.

[0095] All models are trained by using the RMSProp optimizer, and the learning rate is set to 10 -5 , and the L2 regularization coefficient λ = 1 × 10 -3 .

[0096] In order to better show the effectiveness of the method of the application in the prediction of the intention of pedestrians crossing the road, the comparative experiment was carried out on the computing platform Hassan equipped with 8 NVIDIA GTX 1080Ti graphics cards, and the model was realized by using the Pytorch deep learning framework. In the experiment, a plurality of public pedestrian behavior prediction data sets were used, including the JAAD data set, the JAAD-beh data set and the PIE data set. In the experiment, the method of the application was compared with the existing mainstream pedestrian crossing prediction method, and the comparison indexes included AUC, ACC, F1, Precision and Recall, and the results are shown in Tables 1, 2 and 3. As can be seen from the tables, the safety auxiliary method of the application is better than the existing mainstream method in almost all indexes.

[0097] The references of the mainstream methods are as follows:

[0098] [1]. Bhattacharyya A, Fritz M, Schiele B. Long-term on-board prediction of people in traffic scenes under uncertainty [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 4194-4202.+MultiRNN (2018).

[0099] [2]. Kotseruba I, Rasouli A, Tsotsos J K. Do they want to cross? understanding pedestrian intention for behavior prediction [C] / / 2020 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2020: 1688-1693.+SingleRNN (2020).

[0100] [3]. Kotseruba I, Rasouli A, Tsotsos J K. Benchmark for evaluating pedestrian action prediction [C] / / Proceedings of the IEEE / CVF winter conference on applications of computer vision. 2021: 1258-1268. + PCPA (2021).

[0101] [4]. Lorenzo J, Alonso IP, Izquierdo R, et al. Capformer: Pedestrian crossing action prediction using transformer [J]. Sensors, 2021, 21(17): 5694. + CAPformer (2021).

[0102] [5]. Yang D, Zhang H, Yurtsever E, et al. Predicting pedestrian crossing intention with feature fusion and spatio-temporal attention [J]. IEEE Transactions on Intelligent Vehicles, 2022, 7(2): 221-230. + Predicting (2022)

[0103] [6]. Ham J S, Bae K, Moon J. Mcip: Multi-stream network for pedestrian crossing intention prediction [C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 663-679. + MCIP (2022).

[0104] [7]. Dong M. Pedestrian Cross Forecasting with Hybrid Feature Fusion [C] / / Asian Conference on Machine Learning. PMLR, 2024:327-342. +Hybrid-Group(2024).

[0105] [8]. Yao Y, Atkins E, Roberson M J, et al. Coupling intent and action for pedestrian crossing behavior prediction [J]. arXiv preprint arXiv:2105.04133, 2021. +Coupling(2021).

[0106] [9]. Zhou Y, Tan G, Zhong R, et al. Pit: Progressive interaction transformer for pedestrian crossing intention prediction [J]. IEEE Transactions on Intelligent Transportation Systems, 2023. +PIT(2023).

[0107]

[10] . Zhang Z, Tian R, Ding Z. Trep: Transformer-based evidential prediction for pedestrian intention with uncertainty [C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(3):3534-3542. +TREP(2023).

[0108]

[11] . Rasouli A, Kotseruba I, Tsotsos J K. Pedestrian action anticipation using contextual feature fusion in stacked rnns [J]. arXiv preprint arXiv:2005.06582, 2020. +SF-GRU(2020).

[0109]

[12] . Lv N, Huang Y, Zhang H, et al. Pedestrian Crossing Prediction with Pathwise Feature Fusion and Stacked Gate Recurrent Unit [J]. IEEE Sensors Letters, 2024. +PFRN (2024).

[0110] Table 1 Comparative experimental results on the JAAD-beh dataset

[0111] Method Encoder AUC ACC F1 Prec Rec MultiRNN (2018) VGG+LSTM 0.5 -- 0.74 -- -- SingleRNN (2020) VGG+GRU 0.52 0.59 0.71 0.94 0.8 PCPA (2021) C3D+GRU 0.5 0.58 0.71 -- -- CAPformer (2021) Transformer 0.55 -- 0.74 -- -- Predicting (2022) VGG+GRU 0.54 0.92 0.74 0.95 0.85 MCIP (2022) VGG+GRU 0.55 0.94 0.78 -- -- Hybrid-Group (2024) VGG+GRU 0.91 0.97 0.79 -- -- The method of the invention Transformer 0.91 0.74 0.85 0.75 0.98

[0112] Table 2 Comparative experimental results on the JAAD dataset

[0113]

[0114]

[0115] Table 3 Comparative experimental results on the PIE dataset

[0116] Method Encoder AUC ACC F1 Prec Rec MultiRNN (2018) VGG+LSTM 0.8 0.83 0.71 0.99 0.73 SingleRNN (2020) VGG+GRU 0.94 0.76 0.45 0.93 0.36 SF-GRU (2020) VGG+GRU 0.83 0.84 0.72 0.96 0.8 PCPA (2021) C3D+GRU 0.86 0.87 0.77 -- -- Coupling (2021) VGG+GRU 0.88 0.84 0.9 0.96 0.84 CAPformer (2021) Transformer 0.85 -- 0.78 -- -- MCIP (2022) VGG+GRU 0.87 0.89 0.81 -- -- PFRN (2024) VGG+GRU 0.85 0.9 0.77 0.81 0.74 The method of the invention Transformer 0.88 0.87 0.92 0.89 0.96

[0117] In addition, the ablation experiment of the present embodiment further proves the effectiveness of the time sequence merging module (TMM) and the context attention module (CAB). After removing the TMM module or the CAB module, the model performance is decreased, which shows the key role of the two modules in feature extraction and fusion. The detailed comparison data is shown in Table 4, wherein the hook corresponds to the selection of the module

[0118] Table 4 Ablation experiment results of the MM module and the CAB module

[0119] TMM CAB AUC ACC F1 Precision Recall √ 0.86 0.85 0.91 0.88 0.93 √ 0.84 0.85 0.91 0.86 0.97 √ √ 0.88 0.87 0.92 0.89 0.96

[0120] Embodiment 3: For the scenario of embodiment 1, Figure 3 A schematic diagram of an electronic device (90) that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device is intended to represent a broad category of digital computers, and therefore should not be limited to a specific computer philosophy.

[0121] Electronic devices may also refer to various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0122] In order to utilize the vast computing and communication capabilities of the cloud to increase real-time performance, the electronic device (90) is a cloud server equipped with a pre-processing unit (2), a feature encoder (3), a time series merging module (4), a context attention module (5), a classification module (6), a data storage unit (7), and an output unit (8). The vehicle side is equipped with a multimodal sensor (1) to achieve separation of the computing side.

[0123] In order to cope with the situation of poor communication, the autonomous driving safety assistance system based on multimodal fusion can be arranged as a whole on the vehicle-mounted end, that is, the multimodal sensor (1), preprocessing unit (2), feature encoder (3), time series merging module (4), context attention module (5), classification module (6), data storage unit (7), and output unit (8) are arranged on the vehicle-mounted end.

[0124] like Figure 3 As shown, the electronic device (90) includes at least one processor (91) and a memory connected to the at least one processor (91), such as a read-only memory (ROM) (92), a random access memory (RAM) (93), etc., wherein the memory stores a computer program that can be executed by the at least one processor, and the processor (91) can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) (92) or the computer program loaded from the storage unit (98) to the random access memory (RAM) (93). Various programs and data required for the operation of the electronic device (90) can also be stored in the RAM 43. The processor (91), ROM 42 and RAM 43 are connected to each other via a bus (94). An input / output (I / O) interface (95) is also connected to the bus (94).

[0125] Multiple components in the electronic device (90) are connected to the I / O interface (95), including: an input unit (96), such as a keyboard, a mouse, etc.; an output unit (97), such as various types of displays, speakers, etc.; a storage unit (98), such as a magnetic disk, an optical disk, etc.; and a communication unit (99), such as a network card, a modem, a wireless communication transceiver, etc. The communication unit (99) allows the electronic device (90) to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0126] The processor (91) can be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the processor (91) include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, and the like. The processor (91) performs various methods and processes described above, such as the automatic driving safety assistance method based on multi-modal fusion.

[0127] In some embodiments, the automatic driving safety assistance method based on multi-modal fusion can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit (98). In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device (90) via the ROM (92) and / or the communication unit (99). When the computer program is loaded onto the RAM (93) and executed by the processor (91), one or more steps of the automatic driving safety assistance method based on multi-modal fusion described above can be performed. Alternatively, in other embodiments, the processor (91) can be configured to perform the automatic driving safety assistance method based on multi-modal fusion by any other appropriate means, such as by means of firmware.

[0128] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0129] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0130] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0133] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0134] Finally, it is to be noted that the above preferred embodiments are merely used to illustrate the technical solutions of the present application but not to limit the present application. Although the present application has been described in detail by the above preferred embodiments, it should be understood by those skilled in the art that various modifications can be made in form and details without departing from the scope of the present application defined by the claims.

Claims

1. An autonomous driving safety assistance method based on multimodal fusion, characterized in that: The following steps are involved: S1: using the vehicle-mounted multimodal sensor (1) to periodically collect visual image data and non-visual image data, and storing the data in a data storage unit (7) in a time sequence; S2: The pre-processing unit (2) pre-processes the visual image data and the non-visual image data in the data storage unit (7); S3: using the feature encoder (3) to extract features from the pre-processed visual image data and non-visual image data respectively, to obtain visual image features and non-visual image features; S4: using the temporal merging module (4) to divide the visual image features and non-visual image features into key events, thereby obtaining the visual image events and non-visual image events; The time series merging module (4) is implemented based on a KNN network; S5: Using the context attention module (5) to fuse the features of the visual image events and the non-visual image events to obtain the fused features; S6: Input the fused features into the classification module (6) to predict the pedestrian behavior type; S7: The pedestrian behavior type prediction result is stored in the data storage unit (7), and fed back to the automobile terminal system via the output unit (8).

2. The autonomous driving safety assistance method based on multimodal fusion according to claim 1, characterized in that: The pre-processing in step S2 comprises the following steps: S201: Identify whether there is a pedestrian in the visual image data at the current moment. If not, wait for the visual image data at the next moment, otherwise continue to perform the following steps; S202: Segmenting pedestrians in the visual image data, cropping and resizing the local image of each target pedestrian, and performing normalization processing; S203: According to the time period of the visual image data, a fully connected layer is used to perform embedding mapping, align the dimensions of different non-visual image data, and then splice them together.

3. The autonomous driving safety assistance method based on multimodal fusion according to claim 1, characterized in that: The step S4 is based on the density peak clustering algorithm, specifically: S401: Calculate the local density of visual image features and non-visual image features of any frame using KNN; Specific, visual image features The visual image feature I of the tth frame t The corresponding local density ρ t for: Non-visual image features The non-visual image feature J of the tth frame t The corresponding local density for: Among them, KNN(I t ,F v ) indicates that in F v Does not include itself I t K nearest neighbors, KNN (J t ,F nv ) is similar to it; K is a positive integer; S402: Calculating the shortest distance between the visual image feature and the non-visual image feature of any frame and any visual image feature and non-visual image feature with a higher density; Specifically, from the visual image feature I of the t-th frame t The shortest distance δ to any visual image feature with higher density t for: Among them, m∈[1,T] is the index; Similarly, from the non-visual image feature J of the t-th frame t The shortest distance to any non-visual image feature with higher density for: Among them, m∈[1,T] is the index; S403: Select cluster centers from the visual image features and non-visual image features with higher local density and longer shortest distance, and perform clustering on the remaining visual image features and non-visual image features. The obtained cluster sets are respectively visual image events ε={E1, E2,…, E M } and non-visual image events Among them, M and is the total number of corresponding events.

4. The autonomous driving safety assistance method based on multimodal fusion according to claim 1, characterized in that: The step S5 is specifically as follows: S501: Using visual image events ε={E1, E2,…, E M }’s own event similarity, calculate the attention weight α of each event j Specifically: in, Indicates event E i and event E M The similarities between is a trainable weight matrix, M is the total number of corresponding events; S502: Calculate the attention weight of each non-visual image event according to the method of S501 S503: Fusion of visual image events within the time period to obtain visual image event fusion features H v Specifically: in, is a trainable weight; S504: According to the method of S503, the non-visual image events in the time period are fused to obtain the non-visual image event fusion features S505: Fusion of visual image events with features H v and non-visual image event fusion features Further fusion is performed to obtain the fusion feature P; specifically: in, are trainable weights.

5. The autonomous driving safety assistance method based on multimodal fusion according to claim 1, characterized in that: The classification module (6) is a three-layer MLP fully connected layer; its corresponding training loss function is a cross entropy loss function.

6. An autonomous driving safety assistance system based on multimodal fusion applied to the method according to any one of claims 1 to 5, characterized in that: The invention comprises a multimodal sensor (1), a preprocessing unit (2), a feature encoder (3), a time series merging module (4), a context attention module (5), a classification module (6), a data storage unit (7), and an output unit (8); the multimodal sensor (1) comprises a visual image sensor (11) and a non-visual image sensor (12), which respectively collect visual image data and non-visual image data according to a period; the preprocessing unit (2) is connected to the data storage unit (7) and the feature encoder (3) respectively, and is used for preprocessing the visual image data and the non-visual image data; the feature encoder (3) comprises a visual encoder (31) and a non-visual encoder (32); the visual encoder (31) and the non-visual encoder (32) respectively use a deep learning network to realize feature extraction of the corresponding preprocessed visual image data and non-visual image data; the time series merging module (4) is connected to the feature encoder (3), is implemented based on a deep learning network, and is used for event segmentation; the context attention module (5) is connected to the time series merging module (4), is implemented based on a deep learning network, and is used for realizing feature fusion of events; The classification module (6) is connected to the context attention module (5), and is implemented based on a classifier for classifying pedestrian behavior types; the data storage unit (7) is respectively connected to the multimodal sensor (1), the classification module (6) and the output unit (8) for data storage; The output unit (8) is a communication module used to feed back the pedestrian behavior type to the automobile terminal system.

7. The autonomous driving safety assistance system based on multimodal fusion according to claim 6, characterized in that: The visual encoder (31) adopts a pre-trained Video-VIT-base model to extract visual features; the non-visual encoder (32) adopts a Transformer model based on the Video-VIT-small model architecture to extract non-visual features.

8. The autonomous driving safety assistance system based on multimodal fusion according to claim 6, characterized in that: The multimodal sensor, preprocessing unit, feature encoder, time series merging module, context attention module, classification module, data storage unit, and output unit are all arranged on the vehicle side.

9. The autonomous driving safety assistance system based on multimodal fusion according to claim 6, characterized in that: The multimodal sensor is arranged on the vehicle-mounted end; the preprocessing unit, feature encoder, time series merging module, context attention module, classification module, data storage unit, and output unit are arranged on the cloud server.

Images