Abnormal driving behavior detection method based on cross-modal image fusion recognition

Abstract

The invention provides an abnormal driving behavior detection method based on cross-modal image fusion recognition. The abnormal driving behavior detection method comprises the steps of performing infrared and depth modal image acquisition on driving behaviors of a driver; the method comprises the following steps: for infrared and depth modal images collected at the same visual angle, executing the following steps on the infrared and depth modal images: performing N times of MT2T processing on the corresponding modal images by adopting an encoder to obtain corresponding modal feature maps; performing cross-modal fusion on the infrared and depth modal feature maps by adopting a cross-modal bilateral multi-head multi-cross proxy converter to obtain an infrared and depth modal feature fusion map; superposing the infrared and depth modal feature fusion images to obtain a modal feature general image, and carrying out N + 1 times of RMT2T processing on the modal feature general image by adopting a decoder to obtain a decoding mark; and according to the decoding mark, the detection mark and the boundary mark, detecting the abnormal driving behavior of the driver under the visual angle. The abnormal driving behavior detection method is relatively simple and relatively high in identification accuracy and efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of safe driving, and in particular relates to an abnormal driving behavior detection method based on cross-modal image fusion recognition. Background Art

[0002] Driving has become an indispensable part of modern life. It provides people with an efficient and convenient way of moving, promotes economic development and social interaction. However, at the same time, with the increase in the number of vehicles and the popularity of road use, the frequency of traffic accidents has also increased significantly. According to statistics from the World Health Organization, about 1.25 million people die in traffic accidents each year, and up to 50 million people are injured. These figures reflect the seriousness of traffic safety issues and their profound impact on global public health. Nearly 90% of these accidents are caused by human factors, such as speeding, drunk driving, distracted driving, etc. In particular, distracted driving has been identified as one of the main causes of about 68% of traffic accidents. Distracted driving can include using mobile phones, adjusting music, drinking water or other activities unrelated to driving. These behaviors will significantly reduce the driver's attention and responsiveness to road conditions, thereby increasing the risk of accidents. Therefore, it is urgent to design a reliable driver's abnormal driving action detection and safety warning algorithm to monitor the driver's performance, alertness and driving intention, which is crucial to preventing human-related road traffic accidents. In order to solve this problem, many teams have focused on technological innovation and application of intelligent recognition of abnormal driving action detection and recognition.

[0003] At present, in order to solve the problem of abnormal driving action detection and safety warning, different teams are committed to using cameras, sensors and other electronic devices to collect videos of the driver's driving process and actions, and use deep learning models to perform posture recognition on the collected video data, or use target detection algorithms to segment the image background and identify the driver's key sub-area images after segmenting the image. However, the existing abnormal driving behavior detection methods still have the problems of high misjudgment rate and inaccurate recognition.

[0004] Specifically, the existing research on abnormal driving action detection and safety warning of drivers mainly focuses on the following two directions: 1. Monitoring methods, devices and electronic equipment for confidence abnormal driving behavior based on deep learning. The system inputs real-time driving video into the action recognition model, and identifies the first confidence of each abnormal driving action of the driver through a variety of preset abnormal driving actions; inputs real-time driving video into the head posture recognition model, and identifies the driver's head posture as the second confidence of the preset abnormal head posture; determines the preset first confidence threshold corresponding to the business type, and selects the target abnormal driving action with a first confidence greater than the preset first confidence threshold in the abnormal driving action, and pushes the corresponding abnormal action prompt information to the driver; judges whether the second confidence is greater than the preset first confidence threshold. If it is greater, it is determined that the driver has a distraction event, and pushes the distraction prompt information to the driver. 2. Verification experimental method for abnormal driving behavior algorithm model under 5G communication. Driver background segmentation, identification and positioning of driver's key sub-area images, identification of driver's behavior action status and time, verification analysis and experimental deployment; this method first designs a driver background image segmentation algorithm based on the improved Mask-RCNN and segments the driver background image, then designs a target detection algorithm based on the improved Yolov3 and identifies the driver's key sub-area image after the segmented image, and then designs a CNN-LSTM fusion classification algorithm applied to the scene of abnormal driving behavior recognition and allows it to identify the driver's action status and time by inputting three images, finally constructs a small car experimental system and conducts systematic comprehensive verification analysis and experimental deployment, and finally realizes the verification experiment of the algorithm model.

[0005] Existing abnormal driving behavior detection methods can indeed identify the driver's abnormal driving behavior to a certain extent, but different studies still have the following shortcomings in terms of abnormal driving action detection and safety warning technology for drivers: 1. The above-mentioned main research direction of this technology is mainly focused on detection and identification after camera data acquisition, so as to detect and identify the driver's abnormal driving actions after training with images in a single mode. However, the accuracy of using images for training in a single mode is not high, which greatly increases the possibility of misjudgment. 2. At present, there are very few studies involving the detection and identification of abnormal driving actions of drivers under multi-modality, but the research plan is too complicated, resulting in a significant increase in hardware costs. Summary of the invention

[0006] The present invention provides an abnormal driving behavior detection method based on cross-modal image fusion recognition, so as to solve the problems that the existing abnormal driving behavior detection methods are relatively complex and have low recognition accuracy and efficiency.

[0007] According to a first aspect of the present invention, a method for detecting abnormal driving behavior based on cross-modal image fusion recognition is provided, comprising:

[0008] Step S100, collecting infrared modality images and depth modality images of the driver's driving behavior;

[0009] Step S200: for the infrared modality image and the depth modality image collected at the same viewing angle, the following steps are performed on both the infrared modality image and the depth modality image: using an encoder to perform N times of MT2T processing on the corresponding modality image to obtain a corresponding modality feature map, where N is an integer greater than 1;

[0010] Step S300, using a cross-modal bilateral multi-head multi-cross proxy converter to perform cross-modal fusion on the infrared modal feature map and the depth modal feature map to obtain an infrared modal feature fusion map and a depth modal feature fusion map;

[0011] Step S400: superimpose the infrared modal feature fusion map and the depth modal feature fusion map to obtain a modal feature total map, and use a decoder to perform N+1 RMT2T processing on the modal feature total map to obtain a decoding mark;

[0012] Step S500: Detect abnormal driving behavior of the driver at the viewing angle according to the decoding mark, the detection mark and the boundary mark.

[0013] In an optional implementation, step S200 includes performing the following steps on both the infrared modality image and the depth modality image:

[0014] Step S210: input the corresponding modality image into the MRT2T module, and the MRT2T module encodes each frame in the modality image into a patch token;

[0015] Step S220, the MRT2T module uses the MHSA layer to perform attention calculation on the patch token, and then converts the patch token into a new token through a multi-layer perceptron MLP, and reshapes the new token into a two-dimensional image, thereby completing the reconstruction of the patch token;

[0016] Step S230, soft segmentation is performed on the two-dimensional image to segment the two-dimensional image into multiple image patches of size k*k, each image patch has s overlapping tokens with its adjacent image patches, thereby completing one MT2T process, obtaining the corresponding modal feature map after the i-th MT2T process, and adding 1 to i, which is used to represent the number of times the MT2T process is completed, wherein k and N are both integers greater than 1, s is an integer greater than 0, and i is an integer less than or equal to N and its initial value is 1;

[0017] Step S240, determine whether i is equal to N. If so, it means that the MRT2T module completes downsampling, obtains the corresponding modal feature map after the Nth MT2T processing, and executes step S300; otherwise, for the corresponding modal feature map obtained after the i-th MT2T processing, the image patch in the modal feature map is used as the patch token, and the process returns to step S220, where the modal feature map includes an infrared modal feature map and a depth modal feature map.

[0018] In another optional implementation, the step S300 includes:

[0019] Step S310, normalizing the infrared modality feature map and the depth modality feature map respectively through the layearNorm layer;

[0020] Step S320: using a bilateral cross-proxy attention network structure to perform cross-modal feature fusion on the normalized infrared modality feature map and the depth modality feature map to obtain infrared modality proxy features and depth modality proxy features;

[0021] Step S330: superimposing the infrared modal feature map with the infrared modal proxy feature to obtain an infrared modal feature superposition map; superimposing the depth modal feature map with the depth modal proxy feature to obtain a depth modal feature superposition map;

[0022] Step S340, normalizing the infrared modality feature overlay map and the depth modality feature overlay map respectively through the layearNorm layer;

[0023] Step S350, the normalized infrared modal feature overlay map and the depth modal feature overlay map are respectively passed through the multi-layer perceptron MLP, and then superimposed with the infrared modal feature overlay map and the depth modal feature overlay map to obtain the infrared modal feature fusion map and the depth modal feature fusion map.

[0024] In another optional implementation, step S320 includes:

[0025] Step S321: The attention of the infrared modality feature map and the depth modality feature map are respectively expanded into a four-tuple (Q r ,A r ,K r ,V r ) and (Q d ,A d ,K d ,V d ), where Q r ,K r ,V rThey are the three groups of tokens in the original Softmax attention of the infrared modality feature map, A r It is by Q r The token from the pool, and Q r ,K r ,V r Q is the corresponding group tag token obtained after the infrared modality feature map passes through three different linear projection layers; d ,K d ,V d They are the three groups of tokens in the original Softmax attention of the deep modality feature map, A d It is by Q d The token from the pool, and Q d ,K d ,V d The corresponding group label token is obtained after the deep modal feature map passes through three different linear projection layers;

[0026] Step S322: for the infrared modal feature map, according to A r , K d and V d Perform the first cross-proxy attention calculation and obtain For this deep modality feature map, according to A d , K r and V r Perform the first cross-proxy attention calculation and obtain

[0027] Step S323: for the infrared modal feature graph, according to Q r , A d and Perform a second cross-proxy attention calculation to obtain For this deep modal feature map, according to Q d , A r and Perform a second cross-proxy attention calculation to obtain

[0028] Step S324: for the infrared modal feature graph, V after deep convolution DWC r Superimpose to obtain infrared modal proxy features; for the depth modal feature map, V after depth convolution d Superposition is performed to obtain deep modal proxy features.

[0029] In another optional implementation, the step S320 further includes:

[0030] Step S325: for the infrared modal feature graph, according to and The bias between r2 , fine-tuning its first cross-proxy attention calculation, according to The offset B between the infrared modal feature map r1 , fine-tune its second cross-proxy attention calculation; for the deep modal feature map, according to and The bias between d2 , fine-tuning its first cross-proxy attention calculation, according to The bias B between the deep modality feature map d1 , and fine-tunes its second cross-proxy attention calculation.

[0031] In another optional implementation, after step S350, the following steps are further included:

[0032] Step S360: Use the MHSA layer to process the marker token sequences in the infrared modality feature fusion map and the depth modality feature fusion map respectively to enhance marker embedding.

[0033] In another optional implementation, the step S400 includes:

[0034] Step S410: superimposing the infrared modal feature fusion map and the depth modal feature fusion map to obtain a modal feature overall map;

[0035] Step S420, perform the j-th RMT2T processing on the modal feature graph: input the modal feature graph into the RMT2T module, map each tag token in the modal feature graph, and expand each tag token into multiple sub-tags; j is an integer greater than 0 and less than or equal to M, and its initial value is 1, M is the total number of RMT2T processing and is equal to N+1, and N is the total number of MT2T processing in the step S200, and is an integer greater than 1;

[0036] Step S430, soft segmenting the sub-tags in the modal feature map to segment each of the tag tokens into multiple image patches of size k*k, each image patch and its adjacent image patch have s overlapping sub-tags, the number of the image patches is equal to the number of image patches in the modal feature map after the Mj-th MT2T processing in step S200, when j=M, the number of the image patches is equal to the number of feature tokens of the modal image in step S100;

[0037] Step S440: Use the MHSA layer to perform attention calculation on the detection mark, the image patch formed by soft segmentation, and the edge mark, and then use a multi-layer perceptron MLP to convert the image patch formed by soft segmentation into a new image patch;

[0038] Step S450, determine whether j is equal to M. If so, it means that the Mth RMT2T processing is completed, the RMT2T module completes upsampling, and obtains the decoder mark after the Mth RMT2T processing Otherwise, get the decoder mark after the jth RMT2T process Execute step S460;

[0039] Step S460: The image patch corresponding to the modal feature map after the Mjth MT2T processing in step S200 is combined with the decoder mark The image patches in the image are spliced ​​and then linearly mapped to complete the j-th level feature fusion. The j used to represent the number of times the RMT2T process is completed is added by 1, and the decoder after feature fusion is marked As a general picture of the modal features, the decoder is marked The image patch in the figure is used as a marker token in the overall feature graph of the modality, and the process returns to execute step S420.

[0040] In another optional implementation, the step S500 includes:

[0041] Step S510: Mark the decoder Perform the first linear mapping to determine the query value Q for attention calculation d , perform a linear mapping on the detection mark to determine the key value for attention calculation and data value V d , the query value and the key value are task-related tags, the data value is a patch tag, and a first set of task-related tags and patch tags is obtained;

[0042] Step S520: Mark the decoder Perform a second linear mapping to determine the query value Q for attention calculation b , perform a linear mapping on the boundary markers to determine the key value for attention calculation and data value V b , obtain the second set of task-related tags and patch tags;

[0043] Step S530: for each set of task-related marks and patch marks, performing patch-task-attention between the task-related marks and the patch marks;

[0044] Step S540: Determine the recognition result of the driver's abnormal driving behavior under the perspective according to the execution results of the two patch-task-attention operations.

[0045] In another optional implementation, the step S100 includes: taking a plurality of different viewing angles, and collecting infrared modality images and depth modality images of the driver's driving behavior from each viewing angle;

[0046] The step S500 further includes: when it is detected that the driving behavior of the driver at each viewing angle is abnormal, determining that the driving behavior of the driver is abnormal.

[0047] In another optional implementation, in step S100, the camera collects video data including the infrared modal image and the depth modal image, and sends the video data to the control module, which sends the video data to the edge device. The edge device detects the driver's abnormal driving behavior under the corresponding viewing angle according to the video data and steps S200 to S500, and returns the detection result to the control module, which sends the detection result and the collected positioning data to the monitoring end.

[0048] The beneficial effects of the present invention are:

[0049] 1. The present invention performs abnormal driving behavior detection based on two modal images, infrared and depth, which can improve the recognition detection accuracy. When performing the detection, the two modal images are firstly processed by MT2T respectively to reduce the size of the modal images while retaining the features of the two modal images. Thereafter, the newly designed cross-modal bilateral multi-head multi-cross proxy converter is used to perform cross-modal fusion on the two modal feature maps. Since the structure and cross-modal fusion method of the cross-modal bilateral multi-head multi-cross proxy converter are simple, and the size of the modal image to be fused is small, the present invention uses the cross-modal bilateral multi-head multi-cross proxy converter to improve the fusion quality while improving the fusion efficiency, that is, to improve the recognition accuracy while improving the recognition efficiency; finally, the present invention first superimposes the two modal feature fusion maps, and then performs RMT2T processing to restore them to the original modal image size, so that the features of the two modal feature fusion maps are retained. On this basis, abnormal driving behavior recognition detection is performed, which can further improve the recognition accuracy;

[0050] 2. The present invention first uses the MHSA layer and the multi-layer perceptron MLP during encoding to convert the patch token in the corresponding modal image into a new token, and then reshapes the new token into a two-dimensional image, so as to realize the reconstruction of the patch token, thereby restoring the spatial structure; after the reconstruction is completed, the two-dimensional image is soft-segmented, which can simulate local information and reduce the length of the token, and when the two-dimensional image is soft-segmented, the two-dimensional image is segmented into a plurality of image patches of size k*k, and each image patch has s overlapping tokens with its adjacent image patches, so as to avoid the loss of information from the image to the token during the soft segmentation process; the present invention performs N MT2T processing on both the infrared modal image and the depth modal image, and each MT2T processing reduces the data to be processed by the converter, thereby realizing the downsampling of the infrared modal image and the depth modal image, so as to improve the cross-modal fusion efficiency of the converter, thereby improving the driving behavior recognition efficiency; and the present invention designs the downsampling process, so as to avoid information loss while reducing the data to be processed by the converter, thereby ensuring the accuracy of abnormal driving behavior recognition;

[0051] 3. The present invention designs a cross-modal bilateral multi-head multi-cross proxy converter, which uses its cross-modal bilateral multi-head multi-cross proxy Transformer layer to first normalize the infrared modal feature map and the deep modal feature map respectively, and then uses the bilateral cross-proxy attention network structure to perform cross-modal feature fusion on the normalized infrared modal feature map and the deep modal feature map to obtain infrared modal proxy features and deep modal proxy features, and superimposes the infrared modal feature map and the deep modal proxy feature with the infrared modal proxy feature and the deep modal proxy feature. The superimposed infrared modal feature superposition map and the deep modal feature superposition map obtained by superposition are normalized respectively, and then passed through the multi-layer perceptron MLP respectively, and then superimposed with the infrared modal feature superposition map and the deep modal feature superposition map to obtain the infrared modal feature fusion map and the deep modal feature fusion map. The cross-modal bilateral multi-head multi-cross proxy converter of the present invention can convert two modal feature maps from the encoder to the decoder space, and fuse features from different modalities to complementarily enhance the required features between different modalities;

[0052] 4. The present invention designs a bilateral cross-proxy attention network structure and uses a two-step cross-modal cross-proxy fusion attention mechanism. The attention mechanism is increased from the traditional triple to the quadruple, and its operation process can be changed by matrix multiplication. Therefore, the designed cross-modal bilateral multi-head multi-cross-proxy converter is a linear attention mechanism, which reduces the computational complexity of the traditional attention mechanism; the two-step cross-proxy fusion mechanism solves the problem of low efficiency of traditional multi-modal fusion and realizes high-quality cross-modal fusion; in addition, the present invention uses bias to fine-tune the cross-proxy attention calculation, which helps to establish long-distance cross-modal dependencies between different features, and the use of deep convolution DWC can retain the diversity of features, thereby further improving the quality of cross-modal fusion;

[0053] 5. After obtaining the infrared modality feature fusion map and the depth modality feature fusion map, the present invention also uses the MHSA layer to enhance the embedding of the marker token sequence in the two, so as to ensure the accuracy of abnormal driving behavior recognition;

[0054] 6. During decoding, the present invention first superimposes the infrared modal feature fusion map and the depth modal feature fusion map to obtain a modal feature total map, and then performs multiple RMT2T processes on the modal feature total map, so that the final decoder mark can be restored to the size of the infrared or depth modal image; and after the corresponding RMT2T process, the image patch after the RMT2T process is spliced ​​with the image patch of the modal feature map after the corresponding MT2T process, and then linear mapping is performed, further realizing multi-level feature fusion, so that the local feature structure information can be accurately retained, thereby further ensuring the quality of feature fusion;

[0055] 7. The present invention obtains the decoder mark After that, the decoder is marked Perform two linear mappings to obtain two query values ​​Q for attention calculation, and mark the decoder While performing the corresponding sublinear mapping, linear mapping is also performed on the detection mark or the boundary mark, and the key value K and data value V when the attention is calculated based on the detection mark, and the key value K and data value V when the attention is calculated based on the boundary mark are obtained respectively. The query value Q and its corresponding key value K and data value V are taken as a group, the query value Q and the key value K are taken as task-related marks, and the data value V is taken as a patch mark to obtain two groups of task-related marks and patch marks, and patch-task-attention is performed on each group of task-related marks and patch marks. Whether the driver's driving behavior is abnormal is determined according to the execution results of the two patch-task-attentions, that is, a multi-task learning method is used to jointly perform abnormal driving behavior detection and recognition, which can improve the accuracy of abnormal driving behavior detection;

[0056] 8. The present invention collects infrared and depth modal images of the driver's driving behavior at multiple viewing angles, and only determines that the driver's driving behavior is abnormal when it is determined that the driver has abnormal driving behavior at each viewing angle, thereby further improving the accuracy of abnormal driving behavior detection;

[0057] 9. The present invention detects the driver's abnormal driving behavior by edge devices and only sends the detection results and vehicle positioning data to the monitoring end, so that real-time remote monitoring of the driver's abnormal driving behavior can be achieved; the 5G network is used for communication and interaction, which can quickly collect data, stably transmit data, and simply process data. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of an embodiment of the abnormal driving behavior detection method based on cross-modal image fusion recognition of the present invention;

[0059] Figure 2 It is a novel cross-modal fusion model designed by the present invention;

[0060] Figure 3 It is a model of the cross-modal bilateral multi-head multi-cross-proxy Transformer layer of the present invention;

[0061] Figure 4 is a schematic diagram of the bilateral cross-proxy attention network structure of the present invention;

[0062] Figure 5 It is a schematic diagram of the framework of an embodiment of the abnormal driving behavior detection system based on cross-modal image fusion recognition of the present invention. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention and to make the above-mentioned purposes, features and advantages of the embodiments of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0064] In the description of the present invention, unless otherwise specified and limited, it should be noted that the term "connection" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two elements. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0065] See also Figure 1 , is a flow chart of an embodiment of the abnormal driving behavior detection method based on cross-modal image fusion recognition of the present invention. Figure 2 As shown, the novel cross-modal fusion model designed by the present invention adopts an encoder-converter-decoder network structure, wherein the encoder Encoder can be a Transformer encoder, which can adopt a dual-stream structure and use the MT2T module for downsampling to preliminarily extract infrared and depth modal feature maps; the converter Convertor can be a cross-modal bilateral multi-head multi-cross proxy converter specially designed by the present invention and can be a Transformer converter, which is responsible for converting the feature map from the encoder to the decoder space, and at the same time, integrating features from different modalities to complement and enhance the required features between different modalities; the decoder Decoder can achieve decoding through the reverse MT2T (RMT2T), complete upsampling and gradually restore to the full scale, and finally complete the detection and recognition of abnormal driver actions. The abnormal driving behavior detection method based on cross-modal image fusion recognition can include the following steps:

[0066] Step S100: collecting infrared modality images and depth modality images of the driver's driving behavior.

[0067] Step S200: for the infrared modality image and the depth modality image collected at the same viewing angle (the infrared modality image and the depth modality image may have the same size), the infrared modality image and the depth modality image are both executed: using an encoder to perform N times of MT2T processing on the corresponding modality image to obtain a corresponding modality feature map, where N is an integer greater than 1, and executing step S300. The step S200 may include executing the following steps on both the infrared modality image and the depth modality image:

[0068] Step S210: input the corresponding modality image into the MRT2T module, and the MRT2T module encodes each frame in the modality image into a patch token;

[0069] Step S220, the MRT2T module uses the MHSA layer to perform attention calculation on the patch token, and then converts the patch token into a new token through a multi-layer perceptron MLP. The process can be expressed by the following formula: i ′ =MLP(MHSA(T i )), T i ′ Indicates the new token, T i Represents the patch token; thereafter, the new token is reshaped into a two-dimensional image, thereby completing the reconstruction of the patch token. The process can be expressed by the following formula: i =Reshape(T i ′ ), I i Represents a two-dimensional image;

[0070] Step S230, soft segmentation is performed on the two-dimensional image to segment the two-dimensional image into multiple image patches of size k*k, each image patch has s overlapping tokens with its adjacent image patches, thereby completing one MT2T process, obtaining the corresponding modal feature map after the i-th MT2T process, and adding 1 to i used to represent the number of times the MT2T process is completed, wherein k and N are both integers greater than 1, s is an integer greater than 0, i is an integer less than or equal to N and its initial value is 1; wherein the zero padding in the image patch is p, and the sequence length of the token in the two-dimensional image can be:

[0071] Where h and w are the height and width dimensions before soft segmentation, respectively. 0 and w 0 They are the height and width dimensions after soft segmentation respectively.

[0072] Step S240, determine whether i is equal to N. If so, it means that the MRT2T module completes downsampling, obtains the corresponding modal feature map after the Nth MT2T processing, and executes step S300; otherwise, for the corresponding modal feature map obtained after the i-th MT2T processing, the image patch in the modal feature map is used as the patch token, and returns to execute step S220, and the modal feature map includes an infrared modal feature map and a depth modal feature map. In one example, the value of N can be 2.

[0073] The present invention first utilizes the MHSA layer and the multi-layer perceptron MLP during encoding to convert the patch token in the corresponding modal image into a new token, and then reshapes the new token into a two-dimensional image, so as to realize the reconstruction of the patch token, thereby restoring the spatial structure; after the reconstruction is completed, the two-dimensional image is soft-segmented, so as to simulate local information and reduce the length of the token, and when the two-dimensional image is soft-segmented, the two-dimensional image is segmented into a plurality of image patches of size k*k, and each image patch has s overlapping tokens with its adjacent image patches, so as to avoid the loss of information from the image to the token during the soft segmentation process; the present invention performs N MT2T processing on both the infrared modal image and the depth modal image, and each MT2T processing reduces the data to be processed of the converter, thereby realizing the downsampling of the infrared modal image and the depth modal image, so as to improve the cross-modal fusion efficiency of the converter, thereby improving the driving behavior recognition efficiency; and the present invention designs the downsampling process, so as to avoid information loss while reducing the data to be processed by the converter, thereby ensuring the accuracy of abnormal driving behavior recognition.

[0074] Step S300, using a cross-modal bilateral multi-head multi-cross proxy transformer to perform cross-modal fusion on the infrared modal feature map and the depth modal feature map to obtain an infrared modal feature fusion map and a depth modal feature fusion map, and then executing step S400. The cross-modal bilateral multi-head multi-cross proxy transformer may include a cross-modal bilateral multi-head multi-cross proxy Transformer layer and an MHSA layer. Based on the cross-modal bilateral multi-head multi-cross proxy Transformer layer, as shown in FIG. Figure 3 As shown, step S300 may include:

[0075] Step S310, normalizing the infrared modality feature map and the depth modality feature map respectively through the layearNorm layer;

[0076] Step S320: Utilize Figure 4 The bilateral cross-agent attention network structure Bilateral Cross-agentAttention shown in the figure performs cross-modal feature fusion on the normalized infrared modality feature map and the deep modality feature map to obtain infrared modality proxy features and deep modality proxy features;

[0077] Step S330: superimposing the infrared modal feature map with the infrared modal proxy feature to obtain an infrared modal feature superposition map; superimposing the depth modal feature map with the depth modal proxy feature to obtain a depth modal feature superposition map;

[0078] Step S340, normalizing the infrared modality feature overlay map and the depth modality feature overlay map respectively through the layearNorm layer;

[0079] Step S350, the normalized infrared modal feature overlay map and the depth modal feature overlay map are respectively passed through the multi-layer perceptron MLP, and then superimposed with the infrared modal feature overlay map and the depth modal feature overlay map to obtain the infrared modal feature fusion map and the depth modal feature fusion map.

[0080] Combination Figure 4 As shown, step S320 may include:

[0081] Step S321: The attention of the infrared modality feature map and the depth modality feature map are respectively expanded into a four-tuple (Q r ,A r ,K r ,V r ) and (Q d ,A d ,K d ,V d ), where Q r ,K r ,V r They are the three groups of tokens in the original Softmax attention of the infrared modality feature map, A r It is by Q r The token from the pool, and Q r ,K r ,V r Q is the corresponding group tag token obtained after the infrared modality feature map passes through three different linear projection layers; d ,K d ,V d They are the three groups of tokens in the original Softmax attention of the deep modality feature map, A d It is by Q d The token from the pool, and Q d ,K d ,V d The corresponding group label token is obtained after the deep modal feature map passes through three different linear projection layers;

[0082] Step S322: for the infrared modal feature map, according to A r , K d and V d Perform the first cross-proxy attention calculation (i.e. the Softmax Attention corresponding to the first column) and obtain For this deep modality feature map, according to A d, K r and V r Perform the first cross-proxy attention calculation and obtain The process of this step can be expressed as:

[0083]

[0084] Where σ(·) is defined as Softmax attention, and the linear attention formula can be simplified as:

[0085]

[0086] The linear attention mechanism changes the calculation order through the mapping function φ(·) to reduce the complexity of Q and K.

[0087] Step S323: for the infrared modal feature graph, according to Q r , A d and Perform the second cross-proxy attention calculation (i.e. the corresponding Softmax Attention in the second column) to obtain For this deep modal feature map, according to Q d , A r and Perform a second cross-proxy attention calculation to obtain The process of this step can be expressed as:

[0088]

[0089] The above formula is equivalent to:

[0090]

[0091] Step S324: for the infrared modal feature graph, V after deep convolution DWC r Superimpose to obtain infrared modal proxy features; for the depth modal feature map, V after depth convolution d Superposition is performed to obtain deep modal proxy features.

[0092] Step S325: for the infrared modal feature graph, according to and The bias between r2 , fine-tuning its first cross-proxy attention calculation, according to The offset B between the infrared modal feature map r1 , fine-tune its second cross-proxy attention calculation; for the deep modal feature map, according to and The bias between d2 , fine-tuning its first cross-proxy attention calculation, according to The bias B between the deep modality feature map d1 , and fine-tunes its second cross-proxy attention calculation.

[0093] At this time, the bilateral cross-agent attention can be described as follows:

[0094]

[0095] The expression before the plus sign in the above formula is the result of the cross-proxy attention calculation after two fine-tuning of the corresponding modal feature map.

[0096] The present invention designs a cross-modal bilateral multi-head multi-cross proxy transformer, which uses its cross-modal bilateral multi-head multi-cross proxy Transformer layer to first normalize the infrared modal feature map and the deep modal feature map respectively, and then uses the bilateral cross-proxy attention network structure to perform cross-modal feature fusion on the normalized infrared modal feature map and the deep modal feature map to obtain infrared modal proxy features and deep modal proxy features, and superimposes the infrared modal feature map and the deep modal proxy feature with the infrared modal proxy feature and the deep modal proxy feature. The infrared modal feature superposition map and the deep modal feature superposition map obtained by superposition are normalized respectively, and then pass through a multi-layer perceptron MLP respectively, and then superimpose with the infrared modal feature superposition map and the deep modal feature superposition map to obtain the infrared modal feature fusion map and the deep modal feature fusion map. The cross-modal bilateral multi-head multi-cross proxy transformer of the present invention can convert two modal feature maps from the encoder to the decoder space, and fuse features from different modalities to complementarily enhance the required features between different modalities.

[0097] The present invention designs a bilateral cross-proxy attention network structure and uses a two-time cross-modal cross-proxy fusion attention mechanism. The attention mechanism is increased from the traditional triple to the quadruple, and its operation process can be changed through matrix multiplication. Therefore, the designed cross-modal bilateral multi-head multi-cross-proxy converter is a linear attention mechanism, which reduces the computational complexity of the traditional attention mechanism; the two-time cross-proxy fusion mechanism solves the problem of low efficiency of traditional multimodal fusion and realizes high-quality cross-modal fusion; in addition, the present invention uses bias to fine-tune the cross-proxy attention calculation, which helps to establish long-distance cross-modal dependencies between different features, and the use of deep convolution DWC can retain the diversity of features, thereby further improving the quality of cross-modal fusion.

[0098] In addition, the cross-modal bilateral multi-head multi-cross proxy converter is based on the MHSA layer, after the step S350, as Figure 2 As shown, it may also include: step S360, C2MBA-TC may use the MHSA layer to process the marked token sequences in the infrared modal feature fusion map and the deep modal feature fusion map respectively to enhance the embedding of the markers. After obtaining the infrared modal feature fusion map and the deep modal feature fusion map, the present invention also uses the MHSA layer to enhance the embedding of the marked token sequences in the two, so as to ensure the accuracy of abnormal driving behavior recognition.

[0099] Step S400: superimpose the infrared modal feature fusion map and the depth modal feature fusion map to obtain a modal feature total map, and use a decoder to perform N+1 RMT2T processing on the modal feature total map to obtain a decoding mark. Figure 2 As shown, step S400 may include the following steps:

[0100] Step S410: superimpose the infrared modal feature fusion map and the depth modal feature fusion map to obtain a modal feature total map T c ;

[0101] Step S420, perform the j-th RMT2T processing on the modal feature graph: input the modal feature graph into the RMT2T module, map each tag token in the modal feature graph, and expand each tag token into multiple sub-tags; j is an integer greater than 0 and less than or equal to M, and its initial value is 1, M is the total number of RMT2T processing and is equal to N+1, and N is the total number of MT2T processing in the step S200, and is an integer greater than 1;

[0102] Step S430, soft segmentation is performed on the sub-tags in the modal feature map to segment each of the tag tokens into multiple image patches of size k*k, each image patch has s overlapping sub-tags with its adjacent image patch, and the number of the image patches is equal to the number of image patches in the corresponding modal feature map (infrared modal feature map or depth modal feature map, the number of image patches of the two can be the same) after the Mj-th MT2T processing in the step S200. When j=M, the number of the image patches is equal to the number of feature tokens of the modal image in the step S100; similarly, the image size can be reversely calculated according to the following formula:

[0103] Where h and w are the height and width dimensions before soft segmentation, respectively. 0 and w 0 They are the height and width dimensions after soft segmentation respectively.

[0104] Step S440: Use the MHSA layer to detect the marker t d , image patches and edge markers formed by soft segmentation b Perform attention calculation, and then pass it through a multi-layer perceptron MLP to convert the image patch formed by soft segmentation into a new image patch;

[0105] Step S450, determine whether j is equal to M. If so, it means that the Mth RMT2T processing is completed, the RMT2T module completes upsampling, and obtains the decoder mark after the Mth RMT2T processing At this time, the decoder marks The size of is restored to the corresponding modality image size (infrared modality image or depth modality image); otherwise, the decoder mark after the jth RMT2T processing is obtained Execute step S460;

[0106] Step S460: The image patch of the deep modal feature map after the Mjth MT2T processing in step S200 is combined with the decoder mark The image patches in the image are spliced ​​and then linearly mapped to complete the j-th level feature fusion and obtain the decoder mark after the j-th RMT2T processing. Add 1 to j, which is used to represent the number of times RMT2T processing is completed, and mark the decoder after feature fusion As the feature fusion map of this modality, the decoder is marked The image patch in the modality feature fusion map is used as the token in the modality feature fusion map, and the process returns to step S420. In one example, M can be 3. Obtain the decoder token The process can be expressed as:

[0107]

[0108] During decoding, the present invention first superimposes the infrared modal feature fusion map and the depth modal feature fusion map to obtain a modal feature total map, and then performs multiple RMT2T processes on the modal feature total map, so that the final decoder mark can be restored to the size of the infrared or depth modal image; and after the corresponding RMT2T process, the image patch after the RMT2T process is spliced ​​with the image patch of the modal feature map after the corresponding MT2T process, and then linear mapping is performed, so as to further realize multi-level feature fusion, so that the local feature structure information can be accurately retained, thereby further ensuring the quality of feature fusion.

[0109] Step S500: Detect abnormal driving behavior of the driver under the viewing angle according to the decoding mark, the detection mark and the boundary mark. The step S500 may include:

[0110] Step S510: Mark the decoder Perform the first linear mapping to determine the query value Q for attention calculation d (i.e. query), the detection tag ( Figure 2 Medium decoder mark The green mark Detection token in the column is linearly mapped to determine the key value for attention calculation (i.e. key) and data value V d (i.e., value), the query value and the key value are task-related tags, the data value is a patch tag, and a first set of task-related tags and patch tags is obtained;

[0111] Step S520: Mark the decoder Perform a second linear mapping to determine the query value Q for attention calculation b , for boundary markers ( Figure 2 Medium decoder mark The red mark in the column is the Boundary token) and a linear mapping is performed to determine the key value for attention calculation. and data value V b , obtain a second set of task-related tags and patch tags;

[0112] Step S530: for each set of task-related marks and patch marks, performing patch-task-attention between the task-related marks and the patch marks;

[0113] Step S540: Determine the abnormal driving behavior recognition result of the driver under the perspective according to the execution results of the two patch-task-attention. The two patch-task-attention can establish the following two formulas to jointly perform abnormal driving behavior recognition:

[0114]

[0115] Where sigmoid() represents the activation function, d is the dimension corresponding to the key value K, and the present invention divides It can be scaled to reduce the sensitivity to the vector length, so that the variance can be controlled to 1, and it is convenient for subsequent calculations. When the execution results of the two patch-task-attentions both indicate that the driver has abnormal driving behavior, it is determined that the driver's driving behavior is abnormal under this perspective.

[0116] The present invention obtains the decoder mark After that, the decoder is marked Perform two linear mappings to obtain two query values ​​Q for attention calculation, and mark the decoder While performing the corresponding sublinear mapping, a linear mapping is also performed on the detection mark or the boundary mark, and the key value K and the data value V when the attention is calculated based on the detection mark, and the key value K and the data value V when the attention is calculated based on the boundary mark are obtained respectively. The query value Q and its corresponding key value K and data value V are taken as a group, the query value Q and the key value K are used as task-related marks, and the data value V is used as a patch mark to obtain two groups of task-related marks and patch marks. Patch-task-attention is performed on each group of task-related marks and patch marks. Whether the driver's driving behavior is abnormal is determined according to the execution results of the two patch-task-attentions, that is, a multi-task learning method is used to jointly perform abnormal driving behavior detection and identification, which can improve the accuracy of abnormal driving behavior detection.

[0117] Among them, combined Figure 2 As shown, the modal characteristic diagram T c The detection token and boundary token in the column (i.e., the green Detection Token and the red Boundary Token) are the detection tokens and boundary tokens obtained through training, and their initial value is 0. They can be obtained through training by using the initial input label mask map and the different level losses of the Token. The detection token and boundary token can be obtained by performing RMT2T processing M times on the detection tokens obtained through training by adopting the steps S420 to S440, so as to obtain the decoder tokens. The green mark Detection token and the red mark Boundary token in the column; when the detection mark and the boundary mark are subjected to RMT2T processing in steps S420 to S440, the modal feature map in steps S420 to S440 is replaced with the detection mark or the boundary mark.

[0118] In addition, the step S100 includes: taking multiple different viewing angles (for example, including a front view and a top view), and collecting infrared modal images and depth modal images of the driver's driving behavior from each viewing angle; and after the step S500, it also includes: when it is detected that the driver's driving behavior is abnormal at each viewing angle, determining that the driver's driving behavior is abnormal. The present invention collects infrared and depth modal images of the driver's driving behavior at multiple viewing angles, and only determines that the driver's driving behavior is abnormal when it is determined that the driver has abnormal driving behavior at each viewing angle, thereby further improving the accuracy of abnormal driving behavior detection.

[0119] like Figure 5As shown, in step S100 of the present invention, the camera collects video data including the infrared modal image and the depth modal image, and sends the video data to the control module, which sends the video data to the edge device. The edge device detects the abnormal driving behavior of the driver under the corresponding viewing angle according to the video data and steps S200 to S500, and returns the detection result to the control module, which sends the detection result and the collected positioning data to the monitoring end. Among them, the positioning data can be obtained by the Beidou module set on the vehicle, and the control module and the monitoring end can be connected through the 5G module, and the monitoring end can be an APP end. The present invention detects the abnormal driving behavior of the driver by the edge device, and only sends the detection result and the vehicle positioning data to the monitoring end, so that the real-time remote monitoring of the driver's abnormal driving behavior can be realized; the 5G network is used for communication interaction, which can quickly collect data, stably transmit, and simply process data.

[0120] It can be seen from the above embodiments that the present invention performs abnormal driving behavior detection based on two modal images of infrared and depth, which can improve the recognition detection accuracy, and firstly performs MT2T processing on the two modal images respectively during the detection, so as to reduce the size of the modal images while retaining the features of the two modal images, and then uses the newly designed cross-modal bilateral multi-head multi-cross proxy converter to perform cross-modal fusion on the two modal feature maps. Since the structure and cross-modal fusion method of the cross-modal bilateral multi-head multi-cross proxy converter are simple, and the size of the modal image to be fused is small, the present invention uses the cross-modal bilateral multi-head multi-cross proxy converter to improve the fusion quality while improving the fusion efficiency, that is, it can improve the recognition efficiency while improving the recognition accuracy; finally, the present invention first superimposes the two modal feature fusion maps, and then performs RMT2T processing to restore them to the original modal image size, so that the features of the two modal feature fusion maps are retained, and on this basis, abnormal driving behavior recognition detection is performed, which can further improve the recognition accuracy.

[0121] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0122] It will be appreciated that the invention is not limited to the precise construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from its scope. The scope of the invention is governed solely by the appended claims.

Claims

1. A method for detecting abnormal driving behavior based on cross-modal image fusion recognition, characterized in that: include: Step S100, collecting infrared modality images and depth modality images of the driver's driving behavior; Step S200: for the infrared modality image and the depth modality image collected at the same viewing angle, the following steps are performed on both the infrared modality image and the depth modality image: using an encoder to perform N times of MT2T processing on the corresponding modality image to obtain a corresponding modality feature map, where N is an integer greater than 1; Step S300, using a cross-modal bilateral multi-head multi-cross proxy converter to perform cross-modal fusion on the infrared modal feature map and the depth modal feature map to obtain an infrared modal feature fusion map and a depth modal feature fusion map; Step S400: superimpose the infrared modal feature fusion map and the depth modal feature fusion map to obtain a modal feature total map, and use a decoder to perform N+1 RMT2T processing on the modal feature total map to obtain a decoding mark; Step S500: Detect abnormal driving behavior of the driver at the viewing angle according to the decoding mark, the detection mark and the boundary mark.

2. The abnormal driving behavior detection method based on cross-modal image fusion recognition according to claim 1 is characterized in that: The step S200 includes performing the following steps on both the infrared modality image and the depth modality image: Step S210: input the corresponding modality image into the MRT2T module, and the MRT2T module encodes each frame in the modality image into a patch token; Step S220, the MRT2T module uses the MHSA layer to perform attention calculation on the patch token, and then converts the patch token into a new token through a multi-layer perceptron MLP, and reshapes the new token into a two-dimensional image, thereby completing the reconstruction of the patch token; Step S230, soft segmentation is performed on the two-dimensional image to segment the two-dimensional image into multiple image patches of size k*k, each image patch has s overlapping tokens with its adjacent image patches, thereby completing one MT2T process, obtaining the corresponding modal feature map after the i-th MT2T process, and adding 1 to i, which is used to represent the number of times the MT2T process is completed, wherein k and N are both integers greater than 1, s is an integer greater than 0, and i is an integer less than or equal to N and its initial value is 1; Step S240, determine whether i is equal to N. If so, it means that the MRT2T module completes downsampling, obtains the corresponding modal feature map after the Nth MT2T processing, and executes step S300; otherwise, for the corresponding modal feature map obtained after the i-th MT2T processing, the image patch in the modal feature map is used as the patch token, and the process returns to step S220, where the modal feature map includes an infrared modal feature map and a depth modal feature map.

3. The abnormal driving behavior detection method based on cross-modal image fusion recognition according to claim 1 is characterized in that: The step S300 includes: Step S310, normalizing the infrared modality feature map and the depth modality feature map respectively through the layearNorm layer; Step S320: using a bilateral cross-proxy attention network structure to perform cross-modal feature fusion on the normalized infrared modality feature map and the depth modality feature map to obtain infrared modality proxy features and depth modality proxy features; Step S330: superimposing the infrared modal feature map with the infrared modal proxy feature to obtain an infrared modal feature superposition map; superimposing the depth modal feature map with the depth modal proxy feature to obtain a depth modal feature superposition map; Step S340, normalizing the infrared modality feature overlay map and the depth modality feature overlay map respectively through the layearNorm layer; Step S350, the normalized infrared modal feature overlay map and the depth modal feature overlay map are respectively passed through the multi-layer perceptron MLP, and then superimposed with the infrared modal feature overlay map and the depth modal feature overlay map to obtain the infrared modal feature fusion map and the depth modal feature fusion map.

4. The abnormal driving behavior detection method based on cross-modal image fusion recognition according to claim 3 is characterized in that: The step S320 includes: Step S321: The attention of the infrared modality feature map and the depth modality feature map are respectively expanded into a four-tuple (Q r ,A r ,K r ,V r ) and (Q d ,A d ,K d ,V d ), where Q r ,K r ,V r They are the three groups of tokens in the original Softmax attention of the infrared modality feature map, A r By Q r The token from the pool, and Q r ,K r ,V r Q is the corresponding group tag token obtained after the infrared modality feature map passes through three different linear projection layers; d ,K d ,V d They are the three groups of tokens in the original Softmax attention of the deep modality feature map, A d By Q d The token from the pool, and Q d ,K d ,V d The corresponding group label token is obtained after the deep modal feature map passes through three different linear projection layers; Step S322: for the infrared modal feature map, according to A r , K d and V d Perform the first cross-proxy attention calculation and obtain For this deep modality feature map, according to A d , K r and V r Perform the first cross-proxy attention calculation and obtain Step S323: for the infrared modal feature graph, according to Q r , A d and Perform a second cross-proxy attention calculation to obtain For this deep modal feature map, according to Q d , A r and Perform a second cross-proxy attention calculation to obtain Step S324: for the infrared modal feature graph, V after deep convolution DWC r Superimpose to obtain infrared modal proxy features; for the depth modal feature map, V after depth convolution d Superposition is performed to obtain deep modal proxy features.

5. The abnormal driving behavior detection method based on cross-modal image fusion recognition according to claim 4 is characterized in that: The step S320 further includes: Step S325: for the infrared modal feature graph, according to and The bias between r2 , fine-tuning its first cross-proxy attention calculation, according to The offset B between the infrared modal feature map r1 , fine-tune its second cross-proxy attention calculation; for the deep modal feature map, according to and The bias between d2 , fine-tuning its first cross-proxy attention calculation, according to The bias B between the deep modality feature map d1 , and fine-tunes its second cross-proxy attention calculation.

6. The abnormal driving behavior detection method based on cross-modal image fusion recognition according to claim 4 is characterized in that: After step S350, the following steps are further included: Step S360: Use the MHSA layer to process the marker token sequences in the infrared modality feature fusion map and the depth modality feature fusion map respectively to enhance marker embedding.

7. The abnormal driving behavior detection method based on cross-modal image fusion recognition according to any one of claims 1 to 6, characterized in that: The step S400 includes: Step S410: superimposing the infrared modal feature fusion map and the depth modal feature fusion map to obtain a modal feature overall map; Step S420, perform the j-th RMT2T processing on the modal feature graph: input the modal feature graph into the RMT2T module, map each tag token in the modal feature graph, and expand each tag token into multiple sub-tags; j is an integer greater than 0 and less than or equal to M, and its initial value is 1, M is the total number of RMT2T processing and is equal to N+1, and N is the total number of MT2T processing in the step S200, and is an integer greater than 1; Step S430, soft segmenting the sub-tags in the modal feature map to segment each of the tag tokens into multiple image patches of size k*k, each image patch and its adjacent image patch have s overlapping sub-tags, the number of the image patches is equal to the number of image patches in the modal feature map after the Mj-th MT2T processing in step S200, when j=M, the number of the image patches is equal to the number of feature tokens of the modal image in step S100; Step S440: Use the MHSA layer to perform attention calculation on the detection mark, the image patch formed by soft segmentation, and the edge mark, and then use a multi-layer perceptron MLP to convert the image patch formed by soft segmentation into a new image patch; Step S450, determine whether j is equal to M. If so, it means that the Mth RMT2T processing is completed, the RMT2T module completes upsampling, and obtains the decoder mark after the Mth RMT2T processing Otherwise, get the decoder mark after the jth RMT2T process Execute step S460; Step S460: The image patch corresponding to the modality feature map after the Mjth MT2T processing in step S200 is combined with the decoder mark The image patches in the image are spliced ​​and then linearly mapped to complete the j-th level feature fusion. The j used to represent the number of times the RMT2T process is completed is added by 1, and the decoder after feature fusion is marked As a general picture of the modal features, the decoder is marked The image patch in the figure is used as a marker token in the overall feature graph of the modality, and the process returns to execute step S420.

8. The abnormal driving behavior detection method based on cross-modal image fusion recognition according to claim 7 is characterized in that: The step S500 includes: Step S510: Mark the decoder Perform the first linear mapping to determine the query value Q for attention calculation d , perform a linear mapping on the detection mark to determine the key value for attention calculation and data value V d , the query value and the key value are task-related tags, the data value is a patch tag, and a first set of task-related tags and patch tags is obtained; Step S520: Mark the decoder Perform a second linear mapping to determine the query value Q for attention calculation b , perform a linear mapping on the boundary markers to determine the key value for attention calculation and data value V b , obtain the second set of task-related tags and patch tags; Step S530: for each set of task-related marks and patch marks, performing patch-task-attention between the task-related marks and the patch marks; Step S540: Determine the recognition result of the driver's abnormal driving behavior under the perspective according to the execution results of the two patch-task-attention operations.

9. The abnormal driving behavior detection method based on cross-modal image fusion recognition according to claim 1 is characterized in that: The step S100 includes: taking a plurality of different viewing angles, and collecting infrared modality images and depth modality images of the driver's driving behavior from each viewing angle; The step S500 further includes: when it is detected that the driving behavior of the driver at each viewing angle is abnormal, determining that the driving behavior of the driver is abnormal.

10. The abnormal driving behavior detection method based on cross-modal image fusion recognition according to claim 1 or 9, characterized in that: In the step S100, the camera collects video data including the infrared modality image and the depth modality image, and sends the video data to the control module, which sends the video data to the edge device. The edge device detects the abnormal driving behavior of the driver under the corresponding viewing angle according to the video data and steps S200 to S500, and returns the detection result to the control module, which sends the detection result and the collected positioning data to the monitoring end.

Images