Automatic driving safety assisting method and system based on multi-modal fusion

Abstract

The invention relates to an automatic driving safety assisting method and system based on multi-modal fusion, and belongs to the field of automatic driving. The system is composed of a multi-mode sensor, a preprocessing unit, a feature encoder, a time sequence merging module, a context attention module, a classification module, a data storage unit and an output unit. The method comprises the following steps: S1, collecting visual image data and non-visual image data; s2, preprocessing the visual image data and the non-visual image data; s3, performing feature extraction by using a feature encoder; s4, key event division is carried out by using a time sequence merging module; S5, feature fusion is carried out by using a context attention module; s6, inputting the fusion features into a classification module, and predicting pedestrian behavior types; and S7, storing a pedestrian behavior type prediction result, and feeding back the pedestrian behavior type prediction result to the automobile terminal system. According to the method, key events related to pedestrian behaviors can be effectively identified and aggregated, and the accuracy and efficiency of pedestrian behavior intention prediction are improved.

Description

Technical Field

[0001] The present invention relates to an autonomous driving safety assistance method and system based on multimodal fusion, belongs to the field of autonomous driving, and in particular to an autonomous driving safety assistance based on multimodal fusion learning. Background Art

[0002] With the rapid development of autonomous driving technology, the interaction between vehicles and other traffic participants in complex road environments has become more frequent and important. As the most vulnerable participants on the road, the safety of pedestrians has always been the focus of autonomous driving and assisted driving systems. In the real world, accurately predicting whether pedestrians have the intention to cross the street is of great significance to ensuring road traffic safety. Autonomous driving vehicles need to have the ability to perceive and understand pedestrians' intentions in advance so that they can take appropriate measures in time before pedestrians make crossing behaviors, such as slowing down, stopping or changing lanes. However, pedestrians' behavior is often affected by many factors, such as traffic signs, traffic lights, road conditions, weather conditions, pedestrians' own status, vehicle speed, and the behavior of other traffic participants. The complexity and diversity of these factors pose a huge challenge to the accurate prediction of pedestrians' intention to cross the street.

[0003] Traditional pedestrian intention prediction methods mainly rely on the analysis of pedestrian motion trajectories, postures, and facial expressions. However, these methods often have difficulty effectively capturing key events related to pedestrian behavior when processing highly redundant video frames, resulting in poor prediction performance. In addition, in complex traffic environments, it is difficult to fully understand pedestrian intentions by relying solely on single visual information. Therefore, there is an urgent need for a method that can effectively integrate multimodal information, highlight key dynamic changes in pedestrian behavior, and capture pedestrian intentions in the temporal dimension to improve the accuracy and reliability of pedestrian crossing intention prediction. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies of the above-mentioned prior art and provide an autonomous driving safety assistance method and system based on multimodal fusion for assisting in judging the intention of pedestrians crossing the street. The method intends to obtain visual image data and non-visual data of time series data in real time through the sensors of the autonomous driving safety assistance system based on multimodal fusion, and introduce them into the Temporal Merging Module (TMM) and the Contextual Attention Block (CAB), thereby effectively identifying and aggregating key events related to pedestrian behavior, thereby improving the accuracy and efficiency of pedestrian behavior intention prediction.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] The automatic driving safety assistance system based on multimodal fusion is characterized in that it is composed of a multimodal sensor, a preprocessing unit, a feature encoder, a time series merging module, a context attention module, a classification module, a data storage unit, and an output unit; the multimodal sensor includes a visual image sensor and a non-visual image sensor, which respectively collect visual image data and non-visual image data according to a period; the preprocessing unit is respectively connected to the data storage unit and the feature encoder, and is used for preprocessing the visual image data and the non-visual image data; the feature encoder includes a visual encoder (Visual Encoder), a non-visual encoder (Non-visual Encoder); the visual encoder and non-visual encoder respectively use deep learning networks to realize the feature extraction of the corresponding pre-processed visual image data and non-visual image data; the time series merging module is connected to the feature encoder, based on the deep learning network, and used for event segmentation; the context attention module is connected to the time series merging module, based on the deep learning network, and used to realize the feature fusion of events; the classification module is connected to the context attention module, based on the classifier, and used to divide the pedestrian behavior type; the data storage unit is respectively connected to the multimodal sensor, the classification module and the output unit for data storage; the output unit is a communication module, which is used to feed back the pedestrian behavior type to the automobile terminal system.

[0007] Furthermore, the visual encoder adopts a pre-trained Video-VIT-base model to extract visual features; the non-visual encoder adopts a Transformer model based on the Video-VIT-small model architecture to extract non-visual features.

[0008] Preferably, the time series merging module is implemented based on a KNN (K-Nearest Neighbor) network.

[0009] Furthermore, the multimodal sensor, preprocessing unit, feature encoder, time series merging module, context attention module, classification module, data storage unit, and output unit are all arranged on the vehicle side.

[0010] Preferably, the multimodal sensor is arranged on the vehicle-mounted end; the preprocessing unit, feature encoder, timing merging module, context attention module, classification module, data storage unit, and output unit are arranged on the cloud server.

[0011] The autonomous driving safety assistance method based on multimodal fusion includes the following steps:

[0012] S1: using the vehicle-mounted multimodal sensor to periodically collect visual image data and non-visual image data, and store them in a data storage unit in a time sequence;

[0013] S2: The preprocessing unit preprocesses the visual image data and the non-visual image data in the data storage unit;

[0014] S3: using a feature encoder to extract features from the preprocessed visual image data and non-visual image data respectively, to obtain visual image features and non-visual image features;

[0015] S4: using the temporal merging module to divide the visual image features and the non-visual image features into key events, and obtain the visual image events and the non-visual image events;

[0016] S5: Use the context attention module to fuse the features of visual image events and non-visual image events to obtain fused features;

[0017] S6: Input the fused features into the classification module to predict the pedestrian behavior type;

[0018] S7: The predicted pedestrian behavior type result is stored in the data storage unit, and fed back to the automobile terminal system using the output unit.

[0019] Further, the preprocessing described in step S2 comprises the following steps:

[0020] S201: Identify whether there is a pedestrian in the visual image data at the current moment, if not, wait for the visual image data at the next moment, otherwise continue to execute the following steps;

[0021] S202: performing image segmentation on pedestrians in the visual image data, cropping and resizing the local image of each target pedestrian, and performing normalization processing;

[0022] S203: According to the time period of the visual image data, a fully connected layer is used to perform embedding mapping, dimensionality of different non-visual image data is aligned, and then they are spliced ​​together.

[0023] The feature encoder in step S3 is divided into a visual encoder and a non-visual encoder. The visual encoder obtains the visual image features of T frames as The non-visual image features of T frames obtained by the non-visual encoder are recorded as

[0024] Further, the step S4 is based on a density peak clustering algorithm, specifically:

[0025] S401: Calculate the local density of visual image features and non-visual image features of any frame using KNN;

[0026] Specific, visual image features The visual image feature I of the tth frame t The corresponding local density ρ t for: Non-visual image features The non-visual image feature J of the tth frame t The corresponding local density for: Among them, KNN(I t , F v ) indicates that in F v Does not include itself t The K nearest neighbors, KNN (J t , F nv ) is similar to it; K is a positive integer.

[0027] S402: Calculate the shortest distance between the visual image feature and the non-visual image feature of any frame and any visual image feature and non-visual image feature with a higher density;

[0028] Specifically, from the visual image feature I of the tth frame t The shortest distance δ to any visual image feature with higher density t for:

[0029] Among them, m∈[1,T] is the index;

[0030] Similarly, from the non-visual image feature J of the tth frame t The shortest distance to any non-visual image feature with higher density for:

[0031] Among them, m∈[1,T] is the index.

[0032] S403: Select cluster centers from the visual image features and non-visual image features with higher local density and larger shortest distance, and perform clustering on the remaining visual image features and non-visual image features. The obtained cluster sets are respectively visual image events ε = {E1, E2, ..., E M} and non-visual image events Among them, M and is the total number of corresponding events.

[0033] Further, step S5 is specifically as follows:

[0034] S501: Using visual image events ε={E1,E2,...,E M}’s own event similarity, and calculate the attention weight ɑ for each event j ; Specifically:

[0035]

[0036] in, Indicates event E i and event E M The similarities between is a trainable weight matrix, M is the total number of corresponding events;

[0037] S502: Calculate the attention weight of each non-visual image event in the same manner as S501

[0038] S503: Fusion of visual image events within a time period to obtain visual image event fusion features H v ; Specifically:

[0039]

[0040] in, is a trainable weight;

[0041] S504: According to the method of S503, the non-visual image events within the time period are fused to obtain the non-visual image event fusion features

[0042] S505: Fusion of visual image events with features H v and non-visual image event fusion features Further fusion is performed to obtain the fusion feature P; specifically:

[0043]

[0044] in, are trainable weights.

[0045] Furthermore, the step S6 specifically includes: inputting the fused feature P into the classification module to predict the pedestrian behavior type.

[0046] Preferably, the classification module is a three-layer MLP fully connected layer; the corresponding training loss function is a cross entropy loss function.

[0047] The beneficial effects of the present invention are: providing an autonomous driving safety assistance method and system based on multimodal fusion, which can effectively capture the key dynamic changes of pedestrian behavior through time series merging and contextual attention mechanism, reduce the interference of redundant information, and improve the accuracy of prediction; at the same time, aggregating and simplifying time series features can achieve real-time prediction at a lower computing cost; in addition, by fusing visual image features and non-visual image features, it can have stronger adaptability and robustness for different scenarios and environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to make the purpose and technical solution of the present invention more clear, the present invention provides the following drawings for explanation:

[0049] Figure 1 This is a flow chart of an autonomous driving safety assistance method based on multimodal fusion according to the present invention;

[0050] Figure 2 This is an architecture diagram of an autonomous driving safety assistance system based on multimodal fusion according to Embodiment 1 of the present invention; wherein the arrow indicates the direction of data transmission;

[0051] Figure 3 It is a schematic diagram of the structure of the electronic device in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose and technical solution of the present invention more clearly understood, the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0053] Example 1: When a car is driving on the road, the automatic driving system can assist safe driving. One of its important functions is to assist the driver or independently determine whether pedestrians on the roadside intend to cross the road, and take appropriate countermeasures. To achieve this function, the car needs to be upgraded at the device level. This example provides an automatic driving safety assistance system based on multimodal fusion.

[0054] Combination Figure 2 The autonomous driving safety assistance system based on multimodal fusion is composed of a multimodal sensor (1), a preprocessing unit (2), a feature encoder (3), a time series merging module (4), a context attention module (5), a classification module (6), a data storage unit (7), and an output unit (8).

[0055] The multimodal sensor (1) comprises a visual image sensor (11) and a non-visual image sensor (12), which respectively collect visual image data and non-visual image data in a periodic manner; the preprocessing unit (2) is respectively connected to a data storage unit (7) and a feature encoder (3) for preprocessing the visual image data and the non-visual image data; the feature encoder (3) comprises a visual encoder (31) and a non-visual encoder (32); the visual encoder (31) and the non-visual encoder (32) respectively use a deep learning network to realize feature extraction of the corresponding preprocessed visual image data and the non-visual image data; The temporal merging module (4) is connected to the feature encoder (3), implemented based on a deep learning network, and used for event segmentation; the context attention module (5) is connected to the temporal merging module (4), implemented based on a deep learning network, and used for event feature fusion; the classification module (6) is connected to the context attention module (5), implemented based on a classifier, and used for pedestrian behavior type segmentation; the data storage unit (7) is respectively connected to the multimodal sensor (1), the classification module (6) and the output unit (8), and used for data storage; the output unit (8) is a communication module, which is used to feed back the pedestrian behavior type to the automobile terminal system.

[0056] Furthermore, the visual encoder (31) adopts a pre-trained Video-VIT-base model to extract visual features; and the non-visual encoder (32) adopts a Transformer model based on the Video-VIT-small model architecture to extract non-visual features.

[0057] The time series merging module (4) is implemented based on the KNN network.

[0058] Example 2: Based on the autonomous driving safety assistance system based on multimodal fusion given in Example 1, this example provides an autonomous driving safety assistance method based on multimodal fusion, which is used to determine whether pedestrians on the roadside intend to cross the road.

[0059] An autonomous driving safety assistance method based on multimodal fusion, combined with Figure 1 , including the following steps:

[0060] S1: Using the vehicle-mounted multimodal sensor (1) to collect visual image data and non-visual image data at a period of 0.5 seconds, and storing them in a data storage unit (7) in a time sequence.

[0061] S2: The preprocessing unit (2) preprocesses the visual image data and the non-visual image data in the data storage unit (7).

[0062] The pretreatment comprises the following steps:

[0063] S201: Identify whether there is a pedestrian in the visual image data at the current moment, if not, wait for the visual image data at the next moment, otherwise continue to execute the following steps;

[0064] S202: performing image segmentation on pedestrians in the visual image data, cropping and resizing the local image of each target pedestrian, and performing normalization processing;

[0065] S203: According to the time period of the visual image data, a fully connected layer is used to perform embedding mapping, dimensionality of different non-visual image data is aligned, and then they are spliced ​​together.

[0066] S3: Using the feature encoder (3), extract features from the preprocessed visual image data and non-visual image data respectively to obtain visual image features and non-visual image features.

[0067] The feature encoder is divided into a visual encoder and a non-visual encoder, and the visual image features of T frames obtained by the visual encoder are recorded as The non-visual image features of T frames obtained by the non-visual encoder are recorded as

[0068] S4: Using the time series merging module (4), the visual image features and the non-visual image features are respectively divided into key events to obtain visual image events and non-visual image events.

[0069] Specifically:

[0070] S401: Calculate the local density of visual image features and non-visual image features of any frame using KNN;

[0071] Specific, visual image features The visual image feature I of the tth frame t The corresponding local density ρ t for: Non-visual image features The non-visual image feature J of the tth frame t The corresponding local density for: Among them, KNN(I t , F v ) indicates that in F v Does not include itself t The K nearest neighbors, KNN (J t , F nv ) is similar to it; K is a positive integer.

[0072] S402: Calculate the shortest distance between the visual image feature and the non-visual image feature of any frame and any visual image feature and non-visual image feature with a higher density;

[0073] Specifically, from the visual image feature I of the tth frame t The shortest distance δ to any visual image feature with higher density t for:

[0074] Among them, m∈[1,T] is the index;

[0075] Similarly, from the non-visual image feature J of the tth frame t The shortest distance to any non-visual image feature with higher density for:

[0076] Among them, m∈[1,T] is the index.

[0077] S403: Select cluster centers from the visual image features and non-visual image features with higher local density and larger shortest distance, and perform clustering on the remaining visual image features and non-visual image features. The obtained cluster sets are respectively visual image events ε = {E1, E2, ..., E M} and non-visual image events Among them, M and is the total number of corresponding events, set to 3.

[0078] S5: Use the context attention module (5) to fuse the features of the visual image events and the non-visual image events to obtain fused features.

[0079] Specifically:

[0080] S501: Using visual image events ε={E1,E2,...,E M}’s own event similarity, and calculate the attention weight ɑ for each event j ; Specifically:

[0081]

[0082] in, Indicates event E i and event E M The similarities between is a trainable weight matrix, M is the total number of corresponding events;

[0083] S502: Calculate the attention weight of each non-visual image event in the same manner as S501

[0084] S503: Fusion of visual image events within a time period to obtain visual image event fusion features H v ; Specifically:

[0085]

[0086] in, is a trainable weight;

[0087] S504: According to the method of S503, the non-visual image events within the time period are fused to obtain the non-visual image event fusion features

[0088] S505: Fusion of visual image events with features H v and non-visual image event fusion features Further fusion is performed to obtain the fusion feature P; specifically:

[0089]

[0090] in, are trainable weights.

[0091] S6: Input the fused features into the classification module (6) to predict the binary classification problem of whether the pedestrian on the roadside intends to cross the road.

[0092] The classification module (6) is a three-layer MLP fully connected layer; its corresponding training loss function is a cross entropy loss function.

[0093] S7: The pedestrian behavior type prediction result is stored in the data storage unit (7), and fed back to the automobile terminal system via the output unit (8).

[0094] S8: The vehicle terminal system decides whether to assist the driver in taking emergency braking or deceleration measures based on whether the pedestrians on the roadside intend to cross the road.

[0095] All models were trained using the RMSProp optimizer with a learning rate of 10. -5 , L2 regularization coefficient λ = 1 × 10 -3 .

[0096] In order to better demonstrate the effectiveness of the method of the present invention in predicting pedestrian crossing intentions, this embodiment conducted a comparative experiment on Hassan, a computing platform equipped with 8 NVIDIA GTX 1080Ti graphics cards, and implemented the model using the Pytorch deep learning framework. The experiment used multiple public pedestrian behavior prediction datasets, including the JAAD dataset, the JAAD-beh dataset, and the PIE dataset. In the experiment, the method was compared with the existing mainstream pedestrian crossing prediction method, and the comparison indicators included AUC, ACC, F1, Precision, and Recall. The results are shown in Tables 1, 2, and 3. As can be seen from the table, the safety assistance method of the present invention is superior to the existing mainstream methods in almost all indicators.

[0097] The references of the mainstream methods are as follows:

[0098] [1].Bhattacharyya A,Fritz M,Schiele B.Long-term on-board prediction of people in traffic scenes under uncertainty[C] / / Proceedings of the IEEEconference on computer vision and pattern recognition.2018:4194-4202.+MultiRNN(2018).

[0099] [2].Kotseruba I,Rasouli A,Tsotsos J K.Do they want to cross? understanding pedestrian intention for behavior prediction[C] / / 2020IEEEIntelligent Vehicles Symposium(IV).IEEE,2020:1688-1693.+SingleRNN(2020).

[0100] [3].Kotseruba I,Rasouli A,Tsotsos J K.Benchmark for evaluatingpedestrian action prediction[C] / / Proceedings of the IEEE / CVF winterconference on applications of computer vision.2021:1258-1268.+PCPA(2021).

[0101] [4].Lorenzo J,Alonso IP,Izquierdo R,et al.Capformer:Pedestriancrossing action prediction using transformer[J].Sensors,2021,21(17):5694.+CAPformer(2021).

[0102] [5].Yang D,Zhang H,Yurtsever E,et al.Predicting pedestrian crossingintention with feature fusion and spatio-temporal attention[J].IEEETransactions on Intelligent Vehicles,2022,7(2):221-230.+Predicting(2022)

[0103] [6].Ham J S,Bae K,Moon J.Mcip:Multi-stream network for pedestriancrossing intention prediction[C] / / European Conference on ComputerVision.Cham:Springer Nature Switzerland,2022:663-679.+MCIP(2022).

[0104] [7].Dong M.Pedestrian Cross Forecasting with Hybrid Feature Fusion[C] / / Asian Conference on Machine Learning.PMLR,2024:327-342.+Hybrid-Group(2024).

[0105] [8].Yao Y,Atkins E,Roberson M J,et al.Coupling intent and action forpedestrian crossing behavior prediction[J].arXiv preprint arXiv:2105.04133,2021.+Coupling(2021).

[0106] [9].Zhou Y,Tan G,Zhong R,et al.Pit:Progressive interactiontransformer for pedestrian crossing intention prediction[J].IEEE Transactionson Intelligent Transportation Systems,2023.+PIT(2023).

[0107]

[10] .Zhang Z,Tian R,Ding Z.Trep:Transformer-based evidentialprediction for pedestrian intention with uncertainty[C] / / Proceedings of theAAAI Conference on Artificial Intelligence.2023,37(3):3534-3542.+TREP(2023).

[0108]

[11] .Rasouli A,Kotseruba I,Tsotsos J K.Pedestrian action anticipationusing contextual feature fusion in stacked rnns[J].arXiv preprint arXiv:2005.06582,2020.+SF-GRU(2020).

[0109]

[12] .Lv N, Huang Y, Zhang H, et al. Pedestrian Crossing Prediction with Pathwise Feature Fusion and Stacked Gate Recurrent Unit [J]. IEEE Sensors Letters, 2024. + PFRN (2024).

[0110] Table 1 Comparative experimental results on the JAAD-beh dataset

[0111] method Encoder AUC ACC F1 Prec Rec MultiRNN (2018) VGG+LSTM 0.5 -- 0.74 -- -- SingleRNN(2020) VGG+GRU 0.52 0.59 0.71 0.94 0.8 PCPA(2021) C3D+GRU 0.5 0.58 0.71 -- -- CAPformer(2021) Transformer 0.55 -- 0.74 -- -- Predicting(2022) VGG+GRU 0.54 0.92 0.74 0.95 0.85 MCIP(2022) VGG+GRU 0.55 0.94 0.78 -- -- Hybrid-Group(2024) VGG+GRU 0.91 0.97 0.79 -- -- Method of the present invention Transformer 0.91 0.74 0.85 0.75 0.98

[0112] Table 2 Comparative experimental results on the JAAD dataset

[0113]

[0114]

[0115] Table 3 Comparative experimental results on the PIE dataset

[0116] method Encoder AUC ACC F1 Prec Rec MultiRNN (2018) VGG+LSTM 0.8 0.83 0.71 0.99 0.73 SingleRNN(2020) VGG+GRU 0.94 0.76 0.45 0.93 0.36 SF-GRU(2020) VGG+GRU 0.83 0.84 0.72 0.96 0.8 PCPA(2021) C3D+GRU 0.86 0.87 0.77 -- -- Coupling(2021) VGG+GRU 0.88 0.84 0.9 0.96 0.84 CAPformer(2021) Transformer 0.85 -- 0.78 -- -- MCIP(2022) VGG+GRU 0.87 0.89 0.81 -- -- PFRN(2024) VGG+GRU 0.85 0.9 0.77 0.81 0.74 Method of the present invention Transformer 0.88 0.87 0.92 0.89 0.96

[0117] In addition, this example also conducts ablation experiments to further prove the effectiveness of the temporal merging module (TMM) and the contextual attention module (CAB). After removing the TMM module or the CAB module, the model performance has decreased, indicating the key role of these two modules in feature extraction and fusion. Detailed comparison data is shown in Table 4, where the check mark corresponds to the selection of the module.

[0118] Table 4 Ablation experiment results of MM module and CAB module

[0119] TMM CAB AUC ACC F1 Precision Recall √ 0.86 0.85 0.91 0.88 0.93 √ 0.84 0.85 0.91 0.86 0.97 √ √ 0.88 0.87 0.92 0.89 0.96

[0120] Embodiment 3: For the scenario of embodiment 1, Figure 3 A schematic diagram of an electronic device (90) that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.

[0121] Electronic devices may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0122] In order to utilize the huge computing power and communication capabilities of the cloud to increase real-time performance, the electronic device (90) is a cloud server equipped with a preprocessing unit (2), a feature encoder (3), a time series merging module (4), a context attention module (5), a classification module (6), a data storage unit (7), and an output unit (8). The vehicle end is equipped with a multimodal sensor (1) to achieve separation of the computing end.

[0123] In order to cope with the situation of poor communication, the autonomous driving safety assistance system based on multimodal fusion can be arranged as a whole on the vehicle side, that is, the multimodal sensor (1), preprocessing unit (2), feature encoder (3), time series merging module (4), context attention module (5), classification module (6), data storage unit (7), and output unit (8) are arranged on the vehicle side.

[0124] like Figure 3 As shown, the electronic device (90) includes at least one processor (91), and a memory connected to the at least one processor (91) in communication, such as a read-only memory (ROM) (92), a random access memory (RAM) (93), etc., wherein the memory stores a computer program that can be executed by the at least one processor, and the processor (91) can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) (92) or the computer program loaded from the storage unit (98) to the random access memory (RAM) (93). In RAM 43, various programs and data required for the operation of the electronic device (90) can also be stored. The processor (91), ROM 42 and RAM 43 are connected to each other through a bus (94). An input / output (I / O) interface (95) is also connected to the bus (94).

[0125] A number of components in the electronic device (90) are connected to the I / O interface (95), including: an input unit (96), such as a keyboard, a mouse, etc.; an output unit (97), such as various types of displays, speakers, etc.; a storage unit (98), such as a disk, an optical disk, etc.; and a communication unit (99), such as a network card, a modem, a wireless communication transceiver, etc. The communication unit (99) allows the electronic device (90) to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0126] The processor (91) may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor (91) include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor (91) executes the various methods and processes described above, such as an autonomous driving safety assistance method based on multimodal fusion.

[0127] In some embodiments, the autonomous driving safety assistance method based on multimodal fusion may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit (98). In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device (90) via the ROM (92) and / or the communication unit (99). When the computer program is loaded into the RAM (93) and executed by the processor (91), one or more steps of the autonomous driving safety assistance method based on multimodal fusion described above may be performed. Alternatively, in other embodiments, the processor (91) may be configured to execute the autonomous driving safety assistance method based on multimodal fusion by any other appropriate means (e.g., by means of firmware).

[0128] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0130] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0133] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0134] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. An autonomous driving safety assistance system based on multimodal fusion, characterized in that: The invention is composed of a multimodal sensor (1), a preprocessing unit (2), a feature encoder (3), a time series merging module (4), a context attention module (5), a classification module (6), a data storage unit (7), and an output unit (8); the multimodal sensor (1) comprises a visual image sensor (11) and a non-visual image sensor (12), which respectively collect visual image data and non-visual image data according to a period; the preprocessing unit (2) is respectively connected to the data storage unit (7) and the feature encoder (3) for preprocessing the visual image data and the non-visual image data; the feature encoder (3) comprises a visual encoder (31) and a non-visual encoder (32); the visual encoder (31) and the non-visual encoder (32) respectively use a deep learning network to realize feature extraction of the corresponding preprocessed visual image data and non-visual image data; the time series merging module (4) is connected to the feature encoder (3), is implemented based on a deep learning network, and is used for event segmentation; the context attention module (5) is connected to the time series merging module (4), is implemented based on a deep learning network, and is used for event feature fusion; The classification module (6) is connected to the context attention module (5), is implemented based on a classifier, and is used to classify pedestrian behavior types; the data storage unit (7) is respectively connected to the multimodal sensor (1), the classification module (6) and the output unit (8) for data storage; the output unit (8) is a communication module, which is used to feed back the pedestrian behavior type to the automobile terminal system.

2. The automatic driving safety assistance system based on multimodal fusion according to claim 1, characterized in that: The visual encoder (31) adopts a pre-trained Video-VIT-base model to extract visual features; the non-visual encoder (32) adopts a Transformer model based on the Video-VIT-small model architecture to extract non-visual features.

3. The automatic driving safety assistance system based on multimodal fusion according to claim 1, characterized in that: The time series merging module (4) is implemented based on the KNN network.

4. The automatic driving safety assistance system based on multimodal fusion according to claim 1, characterized in that: The multimodal sensor, preprocessing unit, feature encoder, time series merging module, context attention module, classification module, data storage unit, and output unit are all arranged on the vehicle side.

5. The automatic driving safety assistance system based on multimodal fusion according to claim 1, characterized in that: The multimodal sensor is arranged on the vehicle-mounted end; the preprocessing unit, feature encoder, timing merging module, context attention module, classification module, data storage unit, and output unit are arranged on the cloud server.

6. The autonomous driving safety assistance method based on multimodal fusion as described in any one of claims 1 to 5, characterized in that: The following steps are involved: S1: using the vehicle-mounted multimodal sensor (1) to periodically collect visual image data and non-visual image data, and storing them in a data storage unit (7) in a time sequence; S2: the preprocessing unit (2) preprocesses the visual image data and the non-visual image data in the data storage unit (7); S3: using the feature encoder (3) to extract features from the preprocessed visual image data and the non-visual image data respectively, to obtain visual image features and non-visual image features; S4: using the temporal merging module (4) to divide the visual image features and the non-visual image features into key events, thereby obtaining the visual image events and the non-visual image events; S5: Using the context attention module (5) to fuse the features of the visual image events and the non-visual image events to obtain fused features; S6: Input the fused features into the classification module (6) to predict the pedestrian behavior type; S7: The pedestrian behavior type prediction result is stored in the data storage unit (7), and fed back to the automobile terminal system via the output unit (8).

7. The autonomous driving safety assistance method based on multimodal fusion according to claim 6, characterized in that: The pre-processing described in step S2 comprises the following steps: S201: Identify whether there is a pedestrian in the visual image data at the current moment, if not, wait for the visual image data at the next moment, otherwise continue to execute the following steps; S202: performing image segmentation on pedestrians in the visual image data, cropping and resizing the local image of each target pedestrian, and performing normalization processing; S203: According to the time period of the visual image data, a fully connected layer is used to perform embedding mapping, dimensionality of different non-visual image data is aligned, and then they are spliced ​​together.

8. The autonomous driving safety assistance method based on multimodal fusion according to claim 6, characterized in that: The step S4 is based on the density peak clustering algorithm, specifically: S401: Calculate the local density of visual image features and non-visual image features of any frame using KNN; Specific, visual image features The visual image feature I of the tth frame t The corresponding local density ρ t for: Non-visual image features The non-visual image feature J of the tth frame t The corresponding local density for: Among them, KNN(I t , F v ) indicates that in F v Does not include itself t The K nearest neighbors, KNN (J t , F nv ) is similar to it; K is a positive integer. S402: Calculate the shortest distance between the visual image feature and the non-visual image feature of any frame and any visual image feature and non-visual image feature with a higher density; Specifically, from the visual image feature I of the tth frame t The shortest distance δ to any visual image feature with higher density t for: Among them, m∈[1,T] is the index; Similarly, from the non-visual image feature J of the tth frame t The shortest distance to any non-visual image feature with higher density for: Among them, m∈[1,T] is the index. S403: Select cluster centers from the visual image features and non-visual image features with higher local density and larger shortest distance, and perform clustering on the remaining visual image features and non-visual image features. The obtained cluster sets are respectively visual image events ε = {E1, E2, ..., E M } and non-visual image events Among them, M and is the total number of corresponding events.

9. The autonomous driving safety assistance method based on multimodal fusion according to claim 6, characterized in that: The step S5 is specifically as follows: S501: Using visual image events ε={E1,E2,...,E M }’s own event similarity, and calculate the attention weight α of each event j ; Specifically: Among them, score(E i , E M )=E i T W c 1 E M , indicating event E i and event E M The similarity between c 1 is a trainable weight matrix, M is the total number of corresponding events; S502: Calculate the attention weight of each non-visual image event in the same manner as S501 S503: Fusing the visual image events within the time period to obtain the visual image event fusion feature H v ; Specifically: in, is a trainable weight; S504: According to the method of S503, the non-visual image events within the time period are fused to obtain the non-visual image event fusion features S505: Fusion of visual image events with features H v and non-visual image event fusion features Further fusion is performed to obtain the fusion feature P; specifically: in, are trainable weights.

10. The method for assisting autonomous driving safety based on multimodal fusion according to claim 6, characterized in that: The classification module (6) is a three-layer MLP fully connected layer; its corresponding training loss function is a cross entropy loss function.

Images