Abnormal driving behavior detection method based on cross-modal image fusion recognition

Abstract

The present invention provides a method for detecting abnormal driving behavior based on cross-modal image fusion recognition, comprising: collecting infrared and depth modal images of a driver's driving behavior; performing the following steps on each of the infrared and depth modal images collected from the same viewing angle: performing N times of MT2T processing on the corresponding modal images using an encoder to obtain corresponding modal feature maps; performing cross-modal fusion of the infrared and depth modal feature maps using a cross-modal bilateral multi-head multi-cross proxy converter to obtain an infrared and depth modal feature fusion map; superimposing the infrared and depth modal feature fusion maps to obtain a modal feature map; performing N+1 RMT2T processing on the modal feature map using a decoder to obtain a decoding marker; and detecting the driver's abnormal driving behavior from the viewing angle based on the decoding marker, detection marker, and boundary marker. The abnormal driving behavior detection method of the present invention is relatively simple and has high recognition accuracy and efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of safe driving, and specifically relates to a method for detecting abnormal driving behavior based on cross-modal image fusion recognition. Background Art

[0002] Driving has become an integral part of modern life, providing an efficient and convenient means of transportation and fostering economic development and social interaction. However, with the increasing number of vehicles and widespread road use, the frequency of traffic accidents has also increased significantly. According to the World Health Organization, approximately 1.25 million people die in traffic accidents each year, and up to 50 million are injured. These figures reflect the severity of traffic safety issues and their profound impact on global public health. Nearly 90% of these accidents are caused by human error, such as speeding, drunk driving, and distracted driving. Distracted driving, in particular, is identified as a major cause in approximately 68% of traffic accidents. Distracted driving can include using a mobile phone, adjusting music, drinking water, or other unrelated activities. These behaviors significantly reduce a driver's attention and responsiveness to road conditions, thereby increasing the risk of accidents. Therefore, designing reliable algorithms for detecting and warning abnormal driving behaviors and providing safety warnings is an urgent task. Monitoring driver performance, alertness, and driving intentions is crucial for preventing human-related road accidents. To address this issue, many teams are focusing on technological innovation and application in intelligent detection and recognition of abnormal driving behaviors.

[0003] Currently, various teams are working to address the challenges of detecting and providing safety warnings for abnormal driver behavior. They are using cameras, sensors, and other electronic devices to capture video of the driver's driving process and actions. These teams then use deep learning models to perform posture recognition on the captured video data, or they use object detection algorithms to segment the image background and identify key driver sub-regions within the segmented image. However, existing methods for detecting abnormal driving behavior still suffer from high false positive rates and inaccurate recognition.

[0004] Specifically, existing research on abnormal driving action detection and safety warnings for drivers focuses on the following two areas: 1. Deep learning-based confidence-based monitoring methods, devices, and electronic equipment for abnormal driving behavior. This system inputs real-time driving video into a motion recognition model, identifying the first confidence level of each abnormal driving action performed by the driver through a variety of preset abnormal driving actions; inputs real-time driving video into a head posture recognition model, identifying the driver's head posture as a preset abnormal head posture with a second confidence level; determines a preset first confidence threshold corresponding to the business type, filters out target abnormal driving actions with a first confidence level greater than the preset first confidence threshold from the abnormal driving actions, and pushes corresponding abnormal action prompt information to the driver; determines whether the second confidence level is greater than the preset first confidence threshold. If so, it determines that the driver has experienced a distraction event, and pushes a distraction prompt information to the driver. 2. Verification experimental method for abnormal driving behavior algorithm models based on 5G communication. Driver background segmentation, identification and positioning of driver's key sub-area images, identification of driver's behavioral action status and time, verification analysis and experimental deployment; this method first designs a driver background image segmentation algorithm based on the improved Mask-RCNN and segments the driver background image, then designs an improved Yolov3 target detection algorithm and identifies the driver's key sub-area image after the segmented image, and then designs a CNN-LSTM fusion classification algorithm applied to the driving abnormal behavior recognition scenario and allows it to identify the driver's action status and time by inputting three images. Finally, a small car experimental system is constructed and a systematic comprehensive verification analysis and experimental deployment are carried out, ultimately realizing the verification experiment of the algorithm model.

[0005] Existing abnormal driving behavior detection methods can indeed identify abnormal driving behavior to a certain extent, but different studies still have the following shortcomings in the detection and safety warning technology of abnormal driving actions of drivers: 1. The main research direction of this technology is mainly focused on detection and recognition after camera data acquisition, so as to detect and identify abnormal driving actions of drivers after training with images in a single modality. However, the accuracy of training with images in a single modality is not high, which greatly increases the possibility of misjudgment. 2. There are currently very few studies involving the detection and recognition of abnormal driving actions of drivers in multi-modal conditions, but the research solutions are too complex, resulting in a significant increase in hardware costs. Summary of the Invention

[0006] The present invention provides a method for detecting abnormal driving behavior based on cross-modal image fusion recognition, so as to solve the problems that existing abnormal driving behavior detection methods are relatively complex and have low recognition accuracy and efficiency.

[0007] According to a first aspect of the present invention, a method for detecting abnormal driving behavior based on cross-modal image fusion recognition is provided, comprising:

[0008] Step S100: collecting infrared modality images and depth modality images of the driver's driving behavior;

[0009] Step S200: For an infrared modality image and a depth modality image captured at the same viewing angle, perform the following steps on each of the infrared modality image and the depth modality image: using an encoder to perform N times of MT2T processing on the corresponding modality image to obtain a corresponding modality feature map, where N is an integer greater than 1.

[0010] Step S300: using a cross-modal bilateral multi-head multi-cross proxy converter to perform cross-modal fusion on the infrared modal feature map and the depth modal feature map to obtain an infrared modal feature fusion map and a depth modal feature fusion map;

[0011] Step S400: superimpose the infrared modal feature fusion map and the depth modal feature fusion map to obtain a modal feature overall map, and use a decoder to perform N+1 RMT2T processing on the modal feature overall map to obtain a decoding mark;

[0012] Step S500: Detect abnormal driving behavior of the driver at the viewing angle according to the decoding mark, the detection mark and the boundary mark.

[0013] In an optional implementation, step S200 includes performing the following steps on both the infrared modality image and the depth modality image:

[0014] Step S210: Input the corresponding modality image into the MRT2T module, and the MRT2T module encodes each frame in the modality image into a patch token;

[0015] Step S220: The MRT2T module uses the MHSA layer to perform attention calculation on the patch token, and then uses the multi-layer perceptron MLP to convert the patch token into a new token, and reshape the new token into a two-dimensional image, thereby completing the reconstruction of the patch token.

[0016] Step S230: Soft segmentation is performed on the two-dimensional image to divide the two-dimensional image into multiple image patches of size k*k, where each image patch has s overlapping tokens with its adjacent image patches, thereby completing one MT2T process and obtaining a corresponding modal feature map after the i-th MT2T process. i, which represents the number of times the MT2T process is completed, is incremented by 1, where k and N are integers greater than 1, s is an integer greater than 0, and i is an integer less than or equal to N and has an initial value of 1.

[0017] Step S240: Determine whether i is equal to N. If so, it indicates that the MRT2T module completes downsampling and obtains the corresponding modal feature map after the Nth MT2T processing, and executes step S300; otherwise, for the corresponding modal feature map obtained after the i-th MT2T processing, the image patch in the modal feature map is used as the patch token, and the process returns to step S220. The modal feature map includes an infrared modal feature map and a depth modal feature map.

[0018] In another optional implementation, step S300 includes:

[0019] Step S310: normalize the infrared modality feature map and the depth modality feature map respectively through the layearNorm layer;

[0020] Step S320: Using a bilateral cross-proxy attention network structure, cross-modal feature fusion is performed on the normalized infrared modality feature map and the depth modality feature map to obtain infrared modality proxy features and depth modality proxy features.

[0021] Step S330: superimpose the infrared modal feature map and the infrared modal proxy feature to obtain an infrared modal feature superposition map; superimpose the depth modal feature map and the depth modal proxy feature to obtain a depth modal feature superposition map;

[0022] Step S340: normalize the infrared modality feature overlay image and the depth modality feature overlay image respectively through the layearNorm layer;

[0023] In step S350, the normalized infrared modal feature overlay map and the depth modal feature overlay map are respectively passed through the multi-layer perceptron MLP, and then superimposed with the infrared modal feature overlay map and the depth modal feature overlay map to obtain the infrared modal feature fusion map and the depth modal feature fusion map.

[0024] In another optional implementation, step S320 includes:

[0025] Step S321: The attention of the infrared modality feature map and the depth modality feature map are respectively expanded into a quadruple (Q r ,A r ,K r ,V r ) and (Q d ,A d ,K d ,V d ), where Q r ,K r ,V rThey are the three groups of tokens in the original Softmax attention of the infrared modality feature map, A r It is by Q r The token from the pool, and Q r ,K r ,V r Q is the corresponding group tag token obtained after the infrared modality feature map passes through three different linear projection layers; d ,K d ,V d They are the three groups of tokens in the original Softmax attention of the deep modality feature map, A d It is by Q d The token from the pool, and Q d ,K d ,V d The corresponding group label token is obtained after the depth modality feature map passes through three different linear projection layers;

[0026] Step S322: for the infrared modal feature map, according to A r , K d and V d Perform the first cross-proxy attention calculation and obtain For the depth modality feature map, according to A d , K r and V r Perform the first cross-proxy attention calculation and obtain

[0027] Step S323: for the infrared modal feature map, according to Q r 、A d and Perform a second cross-proxy attention calculation to obtain For this deep modal feature map, according to Q d 、A r and Perform a second cross-proxy attention calculation to obtain

[0028] Step S324: for the infrared modal feature map, V after depth convolution DWC r Superimpose to obtain infrared modal proxy features; for the depth modal feature map, V after depth convolution d Superposition is performed to obtain deep modal proxy features.

[0029] In another optional implementation, step S320 further includes:

[0030] Step S325: for the infrared modal feature map, and Bias B r2 , fine-tuning its first cross-proxy attention calculation, according to The offset B between the infrared modal feature map r1 , fine-tune its second cross-proxy attention calculation; for the deep modal feature map, according to and Bias B d2 , fine-tuning its first cross-proxy attention calculation, according to The bias B between the deep modality feature map d1 , and fine-tunes its second cross-proxy attention calculation.

[0031] In another optional implementation, after step S350, the following steps are further included:

[0032] Step S360: Use the MHSA layer to process the token sequences in the infrared modality feature fusion map and the depth modality feature fusion map respectively to enhance the token embedding.

[0033] In another optional implementation, step S400 includes:

[0034] Step S410: superimposing the infrared modal feature fusion map and the depth modal feature fusion map to obtain a modal feature overall map;

[0035] Step S420: Perform the jth RMT2T process on the modal feature graph: input the modal feature graph into the RMT2T module, map each tag token in the modal feature graph, and expand each tag token into multiple sub-tags; j is an integer greater than 0 and less than or equal to M, and its initial value is 1, M is the total number of RMT2T processes and is equal to N+1, and N is the total number of MT2T processes in step S200 and is an integer greater than 1;

[0036] Step S430: soft segmenting the sub-tags in the modal feature map to segment each tag token into multiple image patches of size k*k, where each image patch has s overlapping sub-tags with its adjacent image patches, and the number of image patches is equal to the number of image patches in the modal feature map after the Mj-th MT2T processing in step S200. When j=M, the number of image patches is equal to the number of feature tokens of the modal image in step S100.

[0037] Step S440: Use the MHSA layer to perform attention calculation on the detection mark, the image patch formed by soft segmentation, and the edge mark, and then use the multi-layer perceptron MLP to convert the image patch formed by soft segmentation into a new image patch;

[0038] Step S450: Determine whether j is equal to M. If so, it indicates that the Mth RMT2T process is completed. The RMT2T module completes upsampling and obtains the decoder mark after the Mth RMT2T process. Otherwise, get the decoder mark after the jth RMT2T process Execute step S460;

[0039] Step S460: The image patch corresponding to the modality feature map after the Mj-th MT2T processing in step S200 is combined with the decoder mark The image patches in the image are spliced ​​and then linearly mapped to complete the j-th level feature fusion. The j used to represent the number of times the RMT2T process is completed is added by 1, and the decoder after feature fusion is marked As a general diagram of the modal features, the decoder is marked The image patch is used as a token in the modal feature map, and the process returns to step S420.

[0040] In another optional implementation, step S500 includes:

[0041] Step S510: Mark the decoder Perform the first linear mapping to determine the query value Q for attention calculation d , perform a linear mapping on the detection mark to determine the key value for attention calculation and data value V d , the query value and the key value are task-related tags, the data value is a patch tag, and a first set of task-related tags and patch tags is obtained;

[0042] Step S520: Mark the decoder Perform a second linear mapping to determine the query value Q for attention calculation b , perform a linear mapping on the boundary marker to determine the key value for attention calculation and data value V b , obtain the second set of task-related tags and patch tags;

[0043] Step S530: for each set of task-related tags and patch tags, perform patch-task-attention between the task-related tags and the patch tags;

[0044] Step S540: Determine the recognition result of the driver's abnormal driving behavior under the perspective based on the execution results of the two patch-task-attention operations.

[0045] In another optional implementation, step S100 includes: taking multiple different viewing angles, and collecting infrared modality images and depth modality images of the driver's driving behavior from each viewing angle;

[0046] The step S500 further includes: when it is detected that the driving behavior of the driver at each viewing angle is abnormal, determining that the driving behavior of the driver is abnormal.

[0047] In another optional implementation, in step S100, the camera collects video data including the infrared modal image and the depth modal image, and sends the video data to the control module. The control module sends the video data to the edge device. The edge device detects the driver's abnormal driving behavior under the corresponding perspective based on the video data in accordance with steps S200 to S500, and returns the detection result to the control module. The control module sends the detection result and the collected positioning data to the monitoring end.

[0048] The beneficial effects of the present invention are:

[0049] 1. The present invention performs abnormal driving behavior detection based on infrared and depth modal images, which can improve the recognition and detection accuracy. During the detection, the two modal images are first subjected to MT2T processing respectively to reduce the size of the modal images while retaining the features of the two modal images. Thereafter, the two modal feature maps are cross-modally fused using a newly designed cross-modal bilateral multi-head multi-cross proxy converter. Since the structure and cross-modal fusion method of the cross-modal bilateral multi-head multi-cross proxy converter are simple and the size of the modal images to be fused is small, the present invention uses the cross-modal bilateral multi-head multi-cross proxy converter to improve the fusion quality while improving the fusion efficiency, that is, it can improve the recognition efficiency while improving the recognition accuracy. Finally, the present invention first superimposes the two modal feature fusion maps, and then performs RMT2T processing to restore them to the original modal image size. In this way, the features of the two modal feature fusion maps are retained. On this basis, abnormal driving behavior recognition detection is performed, which can further improve the recognition accuracy.

[0050] 2. During encoding, the present invention first utilizes the MHSA layer and the multi-layer perceptron MLP to convert the patch token in the corresponding modal image into a new token, and then reshapes the new token into a two-dimensional image, thereby realizing the reconstruction of the patch token and restoring the spatial structure. After the reconstruction is completed, the two-dimensional image is soft-segmented to simulate local information and reduce the length of the token. In addition, when soft-segmenting the two-dimensional image, the present invention divides the two-dimensional image into multiple image patches of size k*k, and each image patch has s overlapping tokens with its adjacent image patches, thereby avoiding the loss of information from the image to the token during the soft segmentation process. The present invention performs N MT2T processes on both the infrared modal image and the depth modal image. Each MT2T process reduces the data to be processed by the converter, thereby realizing downsampling of the infrared modal image and the depth modal image, thereby improving the cross-modal fusion efficiency of the converter and thus improving the efficiency of driving behavior recognition. In addition, the present invention designs the downsampling process, thereby reducing the data to be processed by the converter and avoiding information loss, thereby ensuring the accuracy of abnormal driving behavior recognition.

[0051] 3. The present invention designs a cross-modal bilateral multi-head multi-cross proxy converter, which uses its cross-modal bilateral multi-head multi-cross proxy Transformer layer to first normalize the infrared modal feature map and the depth modal feature map respectively, and then uses the bilateral cross-proxy attention network structure to perform cross-modal feature fusion on the normalized infrared modal feature map and the depth modal feature map to obtain infrared modal proxy features and deep modal proxy features. The infrared modal feature map and the depth modal feature map are superimposed with the infrared modal proxy features and the depth modal proxy features respectively. The superimposed infrared modal feature superposition map and the depth modal feature superposition map are normalized respectively, and then passed through the multi-layer perceptron MLP respectively, and then superimposed with the infrared modal feature superposition map and the depth modal feature superposition map respectively to obtain the infrared modal feature fusion map and the depth modal feature fusion map. The cross-modal bilateral multi-head multi-cross proxy converter of the present invention can convert the two modal feature maps from the encoder to the decoder space, and fuse the features from different modalities to complementarily enhance the required features between different modalities;

[0052] 4. This paper designs a bilateral cross-proxy attention network structure and uses a two-step cross-modal cross-proxy fusion attention mechanism. This attention mechanism increases the traditional triple to a quadruple, and its operation process can be changed through matrix multiplication. Therefore, the designed cross-modal bilateral multi-head multi-cross-proxy converter is a linear attention mechanism, which reduces the computational complexity of the traditional attention mechanism. The two-step cross-proxy fusion mechanism solves the problem of low efficiency of traditional multimodal fusion and achieves high-quality cross-modal fusion. In addition, this paper uses bias to fine-tune the cross-proxy attention calculation, which helps to establish long-distance cross-modal dependencies between different features. The use of deep convolution (DWC) can preserve feature diversity, thereby further improving the quality of cross-modal fusion.

[0053] 5. After obtaining the infrared modality feature fusion map and the depth modality feature fusion map, the present invention also uses the MHSA layer to enhance the embedding of the marker token sequences in the two, thereby ensuring the accuracy of abnormal driving behavior recognition;

[0054] 6. During decoding, the present invention first superimposes the infrared modal feature fusion map and the depth modal feature fusion map to obtain a modal feature map, and then performs multiple RMT2T processes on the modal feature map. In this way, the final decoder mark can be restored to the size of the infrared or depth modal image; and after the corresponding RMT2T process, the image patch after the RMT2T process is spliced ​​with the image patch of the modal feature map after the corresponding MT2T process, and then linear mapping is performed, further realizing multi-level feature fusion. In this way, local feature structure information can be accurately retained, thereby further ensuring the quality of feature fusion;

[0055] 7. The present invention obtains the decoder mark After that, mark the decoder Perform two linear mappings to obtain two query values ​​Q for attention calculation, and mark the decoder While performing the corresponding sublinear mapping, a linear mapping is also performed on the detection mark or boundary mark to obtain the key value K and data value V when performing attention calculation based on the detection mark, and the key value K and data value V when performing attention calculation based on the boundary mark, respectively. The query value Q and its corresponding key value K and data value V are grouped together, the query value Q and key value K are used as task-related marks, and the data value V is used as a patch mark to obtain two groups of task-related marks and patch marks. Patch-task-attention is performed on each group of task-related marks and patch marks. Whether the driver's driving behavior is abnormal is determined based on the execution results of the two patch-task-attention methods. That is, a multi-task learning method is used to jointly perform abnormal driving behavior detection and recognition, which can improve the accuracy of abnormal driving behavior detection.

[0056] 8. The present invention collects infrared and depth modal images of the driver's driving behavior from multiple viewing angles, and only determines that the driver's driving behavior is abnormal when it is determined that the driver exhibits abnormal driving behavior from all viewing angles. This can further improve the accuracy of abnormal driving behavior detection.

[0057] 9. The present invention uses edge devices to detect abnormal driving behavior of the driver and only sends the detection results and vehicle positioning data to the monitoring end, so that real-time remote monitoring of the driver's abnormal driving behavior can be achieved; 5G network is used for communication interaction, which can quickly collect data, stably transmit data, and simply process it. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flow chart of an embodiment of a method for detecting abnormal driving behavior based on cross-modal image fusion recognition according to the present invention;

[0059] Figure 2 It is a new cross-modal fusion model designed by the present invention;

[0060] Figure 3 This is the model of the cross-modal bilateral multi-head multi-cross agent Transformer layer of the present invention;

[0061] Figure 4 is a schematic diagram of the bilateral cross-proxy attention network structure of the present invention;

[0062] Figure 5 It is a schematic diagram of the framework of an embodiment of the abnormal driving behavior detection system based on cross-modal image fusion recognition of the present invention. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention and to make the above-mentioned purposes, features and advantages of the embodiments of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0064] In the description of the present invention, unless otherwise specified and limited, it should be noted that the term "connection" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two elements. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meaning of the above terms can be understood according to the specific circumstances.

[0065] See also Figure 1 , is a flow chart of an embodiment of the abnormal driving behavior detection method based on cross-modal image fusion recognition of the present invention. Figure 2 As shown, the novel cross-modal fusion model designed by the present invention adopts an encoder-converter-decoder network structure, wherein the encoder Encoder can be a Transformer encoder, which can adopt a dual-stream structure and use the MT2T module for downsampling to preliminarily extract infrared and depth modal feature maps; the converter Convertor can be a cross-modal bilateral multi-head multi-cross proxy converter specially designed by the present invention and can be a Transformer converter, which is responsible for converting the feature map from the encoder to the decoder space, and at the same time fusing features from different modalities to complement and enhance the required features between different modalities; the decoder Decoder can achieve decoding through inverse MT2T (RMT2T), complete upsampling and gradually restore to the full scale, and finally complete the detection and recognition of abnormal driver actions. The abnormal driving behavior detection method based on cross-modal image fusion recognition can include the following steps:

[0066] Step S100: collecting infrared modality images and depth modality images of the driver's driving behavior.

[0067] Step S200: For an infrared modality image and a depth modality image captured at the same viewing angle (the infrared modality image and the depth modality image may be of the same size), perform the following steps on both the infrared modality image and the depth modality image: perform MT2T processing N times on the corresponding modality image using an encoder to obtain a corresponding modality feature map, where N is an integer greater than 1, and then execute step S300. Step S200 may include performing the following steps on both the infrared modality image and the depth modality image:

[0068] Step S210: Input the corresponding modality image into the MRT2T module, and the MRT2T module encodes each frame in the modality image into a patch token;

[0069] Step S220: The MRT2T module uses the MHSA layer to perform attention calculation on the patch token, and then uses the multi-layer perceptron MLP to convert the patch token into a new token. The process can be expressed by the following formula: i ′ =MLP(MHSA(T i )), T i ′ Indicates the new token, T i Represents the patch token; thereafter, the new token is reshaped into a two-dimensional image, thereby completing the reconstruction of the patch token. The process can be expressed by the following formula: i =Reshape(T i ′ ), I i Represents a two-dimensional image;

[0070] Step S230: Soft segmentation is performed on the two-dimensional image to divide the two-dimensional image into multiple image patches of size k*k. Each image patch has s overlapping tokens with its adjacent image patches, thereby completing one MT2T process and obtaining the corresponding modal feature map after the i-th MT2T process. i, which represents the number of times the MT2T process is completed, is incremented by 1, where k and N are integers greater than 1, s is an integer greater than 0, and i is an integer less than or equal to N and has an initial value of 1. The zero padding in the image patch is p, and the sequence length of the token in the two-dimensional image can be:

[0071] Where h and w are the height and width dimensions before soft segmentation, respectively, and h0 and w0 are the height and width dimensions after soft segmentation, respectively.

[0072] Step S240: Determine whether i is equal to N. If so, the MRT2T module completes downsampling and obtains the corresponding modal feature map after the Nth MT2T process, and then executes step S300. Otherwise, for the corresponding modal feature map obtained after the i-th MT2T process, the image patch in the modal feature map is used as the patch token, and the process returns to step S220. The modal feature map includes an infrared modal feature map and a depth modal feature map. In one example, the value of N can be 2.

[0073] During encoding, the present invention first uses the MHSA layer and the multi-layer perceptron (MLP) to convert the patch token in the corresponding modal image into a new token, and then reshapes the new token into a two-dimensional image, which can realize the reconstruction of the patch token and thus restore the spatial structure. After the reconstruction is completed, the two-dimensional image is soft-segmented to simulate local information and reduce the token length. In addition, when soft-segmenting the two-dimensional image, the present invention divides the two-dimensional image into multiple image patches of size k*k, and each image patch has s overlapping tokens with its adjacent image patches, thereby avoiding the loss of information from image to token during the soft segmentation process. The present invention performs N MT2T processes on both the infrared modal image and the depth modal image. Each MT2T process reduces the data to be processed by the converter, thereby realizing downsampling of the infrared modal image and the depth modal image, thereby improving the cross-modal fusion efficiency of the converter and thus improving the driving behavior recognition efficiency. In addition, the present invention designs the downsampling process, thereby reducing the data to be processed by the converter while avoiding information loss, thereby ensuring the accuracy of abnormal driving behavior recognition.

[0074] Step S300: Use a cross-modal bilateral multi-head multi-cross proxy transformer to perform cross-modal fusion on the infrared modal feature map and the depth modal feature map to obtain an infrared modal feature fusion map and a depth modal feature fusion map, and then execute step S400. The cross-modal bilateral multi-head multi-cross proxy transformer may include a cross-modal bilateral multi-head multi-cross proxy Transformer layer and an MHSA layer. Based on the cross-modal bilateral multi-head multi-cross proxy Transformer layer, as shown in FIG. Figure 3 As shown, step S300 may include:

[0075] Step S310: normalize the infrared modality feature map and the depth modality feature map respectively through the layearNorm layer;

[0076] Step S320: Utilize Figure 4 The bilateral cross-agent attention network structure shown in the figure performs cross-modal feature fusion on the normalized infrared modality feature map and the depth modality feature map to obtain infrared modality proxy features and depth modality proxy features;

[0077] Step S330: superimpose the infrared modal feature map and the infrared modal proxy feature to obtain an infrared modal feature superposition map; superimpose the depth modal feature map and the depth modal proxy feature to obtain a depth modal feature superposition map;

[0078] Step S340: normalize the infrared modality feature overlay image and the depth modality feature overlay image respectively through the layearNorm layer;

[0079] In step S350, the normalized infrared modal feature overlay map and the depth modal feature overlay map are respectively passed through the multi-layer perceptron MLP, and then superimposed with the infrared modal feature overlay map and the depth modal feature overlay map to obtain the infrared modal feature fusion map and the depth modal feature fusion map.

[0080] Combine Figure 4 As shown, step S320 may include:

[0081] Step S321: The attention of the infrared modality feature map and the depth modality feature map are respectively expanded into a quadruple (Q r ,A r ,K r ,V r ) and (Q d ,A d ,K d ,V d ), where Q r ,K r ,V r They are the three groups of tokens in the original Softmax attention of the infrared modality feature map, A r It is by Q r The token from the pool, and Q r ,K r ,V r Q is the corresponding group tag token obtained after the infrared modality feature map passes through three different linear projection layers; d ,K d ,V d They are the three groups of tokens in the original Softmax attention of the deep modality feature map, A d It is by Q d The token from the pool, and Q d ,K d ,V d The corresponding group label token is obtained after the depth modality feature map passes through three different linear projection layers;

[0082] Step S322: for the infrared modal feature map, according to A r , K d and V d Perform the first cross-proxy attention calculation (i.e. the Softmax Attention corresponding to the first column) and obtain For the depth modality feature map, according to A d, K r and V r Perform the first cross-proxy attention calculation and obtain The process of this step can be expressed as:

[0083]

[0084] Where σ(·) is defined as Softmax attention, and the linear attention formula can be simplified as:

[0085]

[0086] The linear attention mechanism changes the calculation order through the mapping function φ(·) to reduce the complexity of Q and K.

[0087] Step S323: for the infrared modal feature map, according to Q r 、A d and Perform the second cross-proxy attention calculation (i.e. the corresponding Softmax Attention in the second column) to obtain For this deep modal feature map, according to Q d 、A r and Perform a second cross-proxy attention calculation to obtain The process of this step can be expressed as:

[0088]

[0089] The above formula is equivalent to:

[0090]

[0091] Step S324: for the infrared modal feature map, V after depth convolution DWC r Superimpose to obtain infrared modal proxy features; for the depth modal feature map, V after depth convolution d Superposition is performed to obtain deep modal proxy features.

[0092] Step S325: for the infrared modal feature map, and Bias B r2 , fine-tuning its first cross-proxy attention calculation, according to The offset B between the infrared modal feature map r1 , fine-tune its second cross-proxy attention calculation; for the deep modal feature map, according to and Bias B d2 , fine-tuning its first cross-proxy attention calculation, according to The bias B between the deep modality feature map d1 , and fine-tunes its second cross-proxy attention calculation.

[0093] At this time, the bilateral cross-agent attention can be described as follows:

[0094]

[0095] The expression before the plus sign in the above formula is the result obtained by calculating the cross-proxy attention after two fine-tuning of the corresponding modal feature map.

[0096] The present invention designs a cross-modal bilateral multi-head multi-cross proxy converter, which uses its cross-modal bilateral multi-head multi-cross proxy Transformer layer to first normalize the infrared modal feature map and the depth modal feature map respectively, and then uses the bilateral cross-proxy attention network structure to perform cross-modal feature fusion on the normalized infrared modal feature map and the depth modal feature map to obtain infrared modal proxy features and deep modal proxy features. The infrared modal feature map and the depth modal feature map are superimposed with the infrared modal proxy features and the depth modal proxy features respectively. The superimposed infrared modal feature superposition map and the depth modal feature superposition map are normalized respectively, and then passed through the multi-layer perceptron MLP respectively, and then superimposed with the infrared modal feature superposition map and the depth modal feature superposition map respectively to obtain the infrared modal feature fusion map and the depth modal feature fusion map. The cross-modal bilateral multi-head multi-cross proxy converter of the present invention can convert the two modal feature maps from the encoder to the decoder space, and fuse the features from different modalities to complementarily enhance the required features between different modalities.

[0097] The present invention designs a bilateral cross-proxy attention network structure and uses a two-step cross-modal cross-proxy fusion attention mechanism. This attention mechanism increases from the traditional triple to the quadruple, and its operation process can be changed through matrix multiplication. Therefore, the designed cross-modal bilateral multi-head multi-cross-proxy converter is a linear attention mechanism, which reduces the computational complexity of the traditional attention mechanism; the two-step cross-proxy fusion mechanism solves the problem of low efficiency of traditional multimodal fusion and achieves high-quality cross-modal fusion; in addition, the present invention uses bias to fine-tune the cross-proxy attention calculation, which helps to establish long-distance cross-modal dependencies between different features, and the use of deep convolution DWC can retain the diversity of features, thereby further improving the quality of cross-modal fusion.

[0098] In addition, the cross-modal bilateral multi-head multi-cross proxy converter is based on the MHSA layer, after the step S350, as shown in FIG. Figure 2 As shown, the process may further include: Step S360, in which the C2MBA-TC may use the MHSA layer to process the marker token sequences in the infrared modal feature fusion map and the depth modal feature fusion map respectively to enhance the embedding of the markers. After obtaining the infrared modal feature fusion map and the depth modal feature fusion map, the present invention further uses the MHSA layer to enhance the embedding of the marker token sequences in both maps, thereby ensuring the accuracy of abnormal driving behavior recognition.

[0099] Step S400: superimpose the infrared modal feature fusion map and the depth modal feature fusion map to obtain a modal feature map, and use a decoder to perform N+1 RMT2T processing on the modal feature map to obtain a decoding mark. Figure 2 As shown, step S400 may include the following steps:

[0100] Step S410: superimpose the infrared modal feature fusion map and the depth modal feature fusion map to obtain a modal feature total map T c ;

[0101] Step S420: Perform the jth RMT2T process on the modal feature graph: input the modal feature graph into the RMT2T module, map each tag token in the modal feature graph, and expand each tag token into multiple sub-tags; j is an integer greater than 0 and less than or equal to M, and its initial value is 1, M is the total number of RMT2T processes and is equal to N+1, and N is the total number of MT2T processes in step S200 and is an integer greater than 1;

[0102] Step S430: soft segmentation is performed on the sub-tags in the modal feature map to segment each tag token into multiple image patches of size k*k. Each image patch has s overlapping sub-tags with its adjacent image patch. The number of image patches is equal to the number of image patches in the corresponding modal feature map (infrared modal feature map or depth modal feature map, the number of image patches of the two can be the same) after the Mj-th MT2T processing in step S200. When j=M, the number of image patches is equal to the number of feature tokens of the modal image in step S100. Similarly, the image size can be reversely calculated according to the following formula:

[0103] Where h and w are the height and width dimensions before soft segmentation, respectively, and h0 and w0 are the height and width dimensions after soft segmentation, respectively.

[0104] Step S440: Use the MHSA layer to detect the tag td , image patches formed by soft segmentation and edge markers t b Perform attention calculations, and then pass them through a multi-layer perceptron (MLP) to convert the soft-segmented image patches into new image patches.

[0105] Step S450: Determine whether j is equal to M. If so, it indicates that the Mth RMT2T process is completed. The RMT2T module completes upsampling and obtains the decoder mark after the Mth RMT2T process. At this time, the decoder marks The size of the image is restored to the corresponding modality image size (infrared modality image or depth modality image); otherwise, the decoder mark after the j-th RMT2T processing is obtained Execute step S460;

[0106] Step S460: The image patch of the depth modality feature map after the Mj-th MT2T processing in step S200 is combined with the decoder mark The image patches in the image are spliced ​​and then linearly mapped to complete the j-th level feature fusion and obtain the decoder label after the j-th RMT2T processing. Add 1 to j, which is used to represent the number of times RMT2T processing is completed, and mark the decoder after feature fusion As the modality feature fusion map, the decoder is marked The image patch is used as the token in the modality feature fusion graph, and the process returns to step S420. In one example, M can be 3. Obtain the decoder token The process can be expressed as:

[0107]

[0108] During decoding, the present invention first superimposes the infrared modal feature fusion map and the depth modal feature fusion map to obtain a modal feature total map, and then performs multiple RMT2T processes on the modal feature total map, so that the final decoder mark can be restored to the size of the infrared or depth modal image; and after the corresponding RMT2T processing, the image patch after the RMT2T processing is spliced ​​with the image patch of the modal feature map after the corresponding MT2T processing, and then linear mapping is performed, further realizing multi-level feature fusion, so that the local feature structure information can be accurately retained, thereby further ensuring the quality of feature fusion.

[0109] Step S500: Detect abnormal driving behavior of the driver under the viewing angle according to the decoding mark, the detection mark and the boundary mark. Step S500 may include:

[0110] Step S510: Mark the decoder Perform the first linear mapping to determine the query value Q for attention calculation d (i.e. query), the detection tag ( Figure 2 Decoder Mark The green mark Detection token in the column is linearly mapped to determine the key value for attention calculation (ie key) and data value V d (i.e., value), the query value and key value are task-related tags, the data value is a patch tag, and a first set of task-related tags and patch tags is obtained;

[0111] Step S520: Mark the decoder Perform a second linear mapping to determine the query value Q for attention calculation b , for boundary markers ( Figure 2 Decoder Mark The red mark Boundary token in the column is used for a linear mapping to determine the key value for attention calculation. and data value V b , obtain the second set of task-related tags and patch tags;

[0112] Step S530: for each set of task-related tags and patch tags, perform patch-task-attention between the task-related tags and the patch tags;

[0113] Step S540: Determine the abnormal driving behavior recognition result of the driver under the perspective based on the execution results of the two patch-task-attention methods. The two patch-task-attention methods can be used to establish the following two formulas to jointly perform abnormal driving behavior recognition:

[0114]

[0115] Where sigmoid() represents the activation function, d is the dimension corresponding to the key value K, and the present invention divides Scaling can be performed to reduce sensitivity to vector length, thereby controlling the variance to 1 and facilitating subsequent calculations. If the results of both patch-task-attention operations indicate abnormal driver behavior, the driver's driving behavior is determined to be abnormal from that perspective.

[0116] The present invention obtains the decoder mark After that, mark the decoder Perform two linear mappings to obtain two query values ​​Q for attention calculation, and mark the decoder While performing the corresponding sublinear mapping, a linear mapping is also performed on the detection mark or boundary mark to obtain the key value K and data value V when calculating attention based on the detection mark, and the key value K and data value V when calculating attention based on the boundary mark, respectively. The query value Q and its corresponding key value K and data value V are grouped together, the query value Q and key value K are used as task-related marks, and the data value V is used as a patch mark to obtain two groups of task-related marks and patch marks. Patch-task-attention is performed on each group of task-related marks and patch marks. Whether the driver's driving behavior is abnormal is determined based on the execution results of the two patch-task-attention processes. That is, a multi-task learning method is used to jointly perform abnormal driving behavior detection and identification, which can improve the accuracy of abnormal driving behavior detection.

[0117] Among them, combined Figure 2 As shown, the modal characteristic diagram T c The detection token and boundary token in the column (i.e., the green Detection Token and the red Boundary Token) are the detection tokens and boundary tokens obtained through training, and their initial value is 0. They can be obtained through training by using the initial input label mask map and the different level losses of the Token. The detection token and boundary token can be obtained by performing RMT2T processing M times on the detection tokens obtained through training using the steps S420 to S440 to obtain the decoder tokens. The green mark Detection token and the red mark Boundary token in the column; when performing RMT2T processing on the detection token and the boundary token in steps S420 to S440, the modal feature map in steps S420 to S440 is replaced with the detection token or the boundary token.

[0118] In addition, step S100 includes: taking multiple different perspectives (for example, including a front view and a top view), and capturing infrared modal images and depth modal images of the driver's driving behavior from each perspective; and step S500 also includes: when the driver's driving behavior is detected to be abnormal at each perspective, determining that the driver's driving behavior is abnormal. The present invention further improves the accuracy of abnormal driving behavior detection by capturing infrared and depth modal images of the driver's driving behavior from multiple perspectives, and only determining that the driver's driving behavior is abnormal when it is determined that the driver exhibits abnormal driving behavior at each perspective.

[0119] like Figure 5As shown, in step S100 of the present invention, the camera collects video data including the infrared modal image and the depth modal image, and sends the video data to the control module, which sends the video data to the edge device. The edge device detects the abnormal driving behavior of the driver under the corresponding viewing angle according to the video data and steps S200 to S500, and returns the detection result to the control module, which sends the detection result and the collected positioning data to the monitoring end. The positioning data can be obtained by the Beidou module set on the vehicle, and the control module and the monitoring end can be connected through a 5G module, and the monitoring end can be an APP end. The present invention detects the abnormal driving behavior of the driver by the edge device, and only sends the detection result and the vehicle positioning data to the monitoring end, so that real-time remote monitoring of the driver's abnormal driving behavior can be achieved; the 5G network is used for communication interaction, which can quickly collect data, stably transmit data, and simply process data.

[0120] As can be seen from the above embodiments, the present invention performs abnormal driving behavior detection based on infrared and depth modal images, which can improve the recognition and detection accuracy. During the detection, the two modal images are first subjected to MT2T processing respectively to reduce the size of the modal images while retaining the features of the two modal images. Thereafter, the two modal feature maps are cross-modally fused using a newly designed cross-modal bilateral multi-head multi-cross proxy converter. Since the structure and cross-modal fusion method of the cross-modal bilateral multi-head multi-cross proxy converter are simple and the size of the modal images to be fused is small, the present invention uses the cross-modal bilateral multi-head multi-cross proxy converter to improve the fusion quality while improving the fusion efficiency, that is, it can improve the recognition accuracy while improving the recognition efficiency. Finally, the present invention first superimposes the two modal feature fusion maps and then performs RMT2T processing to restore them to the original modal image size. In this way, the features of the two modal feature fusion maps are retained. On this basis, abnormal driving behavior recognition and detection are performed, which can further improve the recognition accuracy.

[0121] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the following claims.

[0122] It will be appreciated that the present invention is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and variations can be made without departing from its scope, which is governed solely by the appended claims.

Claims

1. A method for detecting abnormal driving behavior based on cross-modal image fusion recognition, characterized in that: include: Step S100: collecting infrared modality images and depth modality images of the driver's driving behavior; Step S200: For an infrared modality image and a depth modality image captured at the same viewing angle, perform the following steps on each of the infrared modality image and the depth modality image: using an encoder to perform N times of MT2T processing on the corresponding modality image to obtain a corresponding modality feature map, where N is an integer greater than 1. Step S300: using a cross-modal bilateral multi-head multi-cross proxy converter to perform cross-modal fusion on the infrared modal feature map and the depth modal feature map to obtain an infrared modal feature fusion map and a depth modal feature fusion map; Step S400: superimpose the infrared modal feature fusion map and the depth modal feature fusion map to obtain a modal feature overall map, and use a decoder to perform N+1 RMT2T processing on the modal feature overall map to obtain a decoding mark; Step S500: detecting abnormal driving behavior of the driver at the viewing angle according to the decoding mark, the detection mark, and the boundary mark; The step S200 includes performing the following steps on both the infrared modality image and the depth modality image: Step S210: Input the corresponding modality image into the MRT2T module, and the MRT2T module encodes each frame in the modality image into a patch token; Step S220: The MRT2T module uses the MHSA layer to perform attention calculation on the patch token, and then uses the multi-layer perceptron MLP to convert the patch token into a new token, and reshape the new token into a two-dimensional image, thereby completing the reconstruction of the patch token. Step S230: Soft segmentation is performed on the two-dimensional image to divide the two-dimensional image into multiple image patches of size k*k, where each image patch has s overlapping tokens with its adjacent image patches, thereby completing one MT2T process and obtaining a corresponding modal feature map after the i-th MT2T process. i, which represents the number of times the MT2T process is completed, is incremented by 1, where k and N are integers greater than 1, s is an integer greater than 0, and i is an integer less than or equal to N and has an initial value of 1. Step S240: Determine whether i is equal to N. If so, it indicates that the MRT2T module completes downsampling and obtains the corresponding modal feature map after the Nth MT2T processing, and executes step S300; otherwise, for the corresponding modal feature map obtained after the i-th MT2T processing, the image patch in the modal feature map is used as the patch token, and the process returns to step S220. The modal feature map includes an infrared modal feature map and a depth modal feature map. The step S300 includes: Step S310: normalize the infrared modality feature map and the depth modality feature map respectively through the layearNorm layer; Step S320: Using a bilateral cross-proxy attention network structure, cross-modal feature fusion is performed on the normalized infrared modality feature map and the depth modality feature map to obtain infrared modality proxy features and depth modality proxy features. Step S330: superimpose the infrared modal feature map and the infrared modal proxy feature to obtain an infrared modal feature superposition map; superimpose the depth modal feature map and the depth modal proxy feature to obtain a depth modal feature superposition map; Step S340: normalize the infrared modality feature overlay image and the depth modality feature overlay image respectively through the layearNorm layer; Step S350: After the normalized infrared modal feature overlay map and the depth modal feature overlay map are passed through a multi-layer perceptron (MLP), they are superimposed on the infrared modal feature overlay map and the depth modal feature overlay map to obtain an infrared modal feature fusion map and a depth modal feature fusion map. The step S320 includes: Step S321: The attention of the infrared modality feature map and the depth modality feature map are respectively expanded into a quadruple (Q r ,A r ,K r ,V r ) and (Q d ,A d ,K d ,V d ), where Q r ,K r ,V r They are the three groups of tokens in the original Softmax attention of the infrared modality feature map, A r It is by Q r The token from the pool, and Q r ,K r ,V r Q is the corresponding group tag token obtained after the infrared modality feature map passes through three different linear projection layers; d ,K d ,V d They are the three groups of tokens in the original Softmax attention of the deep modality feature map, A d It is by Q d The token from the pool, and Q d ,K d ,V d The corresponding group label token is obtained after the depth modality feature map passes through three different linear projection layers; Step S322: for the infrared modal feature map, according to A r , K d and V d Perform the first cross-proxy attention calculation and obtain For the depth modality feature map, according to A d , K r and V r Perform the first cross-proxy attention calculation and obtain Step S323: for the infrared modal feature map, according to Q r 、A d and Perform a second cross-proxy attention calculation to obtain For this deep modal feature map, according to Q d 、A r and Perform a second cross-proxy attention calculation to obtain Step S324: for the infrared modal feature map, V after depth convolution DWC r Superimpose to obtain infrared modal proxy features; for the depth modal feature map, V after depth convolution d Superposition is performed to obtain deep modal proxy features.

2. The abnormal driving behavior detection method based on cross-modal image fusion recognition according to claim 1 is characterized in that: The step S320 further includes: Step S325: for the infrared modal feature map, and Bias B r2 , fine-tuning its first cross-proxy attention calculation, according to The offset B between the infrared modal feature map r1 , fine-tune its second cross-proxy attention calculation; for the deep modal feature map, according to and Bias B d2 , fine-tuning its first cross-proxy attention calculation, according to The bias B between the deep modality feature map d1 , and fine-tunes its second cross-proxy attention calculation.

3. The abnormal driving behavior detection method based on cross-modal image fusion recognition according to claim 1 is characterized in that: After step S350, the following steps are further included: Step S360: Use the MHSA layer to process the token sequences in the infrared modality feature fusion map and the depth modality feature fusion map respectively to enhance the token embedding.

4. The abnormal driving behavior detection method based on cross-modal image fusion recognition according to any one of claims 1 to 3, characterized in that: The step S400 includes: Step S410: superimposing the infrared modal feature fusion map and the depth modal feature fusion map to obtain a modal feature overall map; Step S420: Perform the jth RMT2T process on the modal feature graph: input the modal feature graph into the RMT2T module, map each tag token in the modal feature graph, and expand each tag token into multiple sub-tags; j is an integer greater than 0 and less than or equal to M, and its initial value is 1, M is the total number of RMT2T processes and is equal to N+1, and N is the total number of MT2T processes in step S200 and is an integer greater than 1; Step S430: soft segmenting the sub-tags in the modal feature map to segment each tag token into multiple image patches of size k*k, where each image patch has s overlapping sub-tags with its adjacent image patches, and the number of image patches is equal to the number of image patches in the modal feature map after the Mj-th MT2T processing in step S200. When j=M, the number of image patches is equal to the number of feature tokens of the modal image in step S100. Step S440: Use the MHSA layer to perform attention calculation on the detection mark, the image patch formed by soft segmentation, and the edge mark, and then use the multi-layer perceptron MLP to convert the image patch formed by soft segmentation into a new image patch; Step S450: Determine whether j is equal to M. If so, it indicates that the Mth RMT2T process is completed. The RMT2T module completes upsampling and obtains the decoder mark after the Mth RMT2T process. Otherwise, get the decoder mark after the jth RMT2T process Execute step S460; Step S460: The image patch corresponding to the modality feature map after the Mj-th MT2T processing in step S200 is combined with the decoder mark The image patches in the image are spliced ​​and then linearly mapped to complete the j-th level feature fusion. The j used to represent the number of times the RMT2T process is completed is added by 1, and the decoder after feature fusion is marked As a general diagram of the modal features, the decoder is marked The image patch is used as a token in the modal feature map, and the process returns to step S420.

5. The abnormal driving behavior detection method based on cross-modal image fusion recognition according to claim 4 is characterized in that: The step S500 includes: Step S510: Mark the decoder Perform the first linear mapping to determine the query value Q for attention calculation d , perform a linear mapping on the detection mark to determine the key value for attention calculation and data value V d , the query value and the key value are task-related tags, the data value is a patch tag, and a first set of task-related tags and patch tags is obtained; Step S520: Mark the decoder Perform a second linear mapping to determine the query value Q for attention calculation b , perform a linear mapping on the boundary marker to determine the key value for attention calculation and data value V b , obtain the second set of task-related tags and patch tags; Step S530: for each set of task-related tags and patch tags, perform patch-task-attention between the task-related tags and the patch tags; Step S540: Determine the recognition result of the driver's abnormal driving behavior under the perspective based on the execution results of the two patch-task-attention operations.

6. The abnormal driving behavior detection method based on cross-modal image fusion recognition according to claim 1 is characterized in that: The step S100 includes: taking multiple different viewing angles and collecting infrared modality images and depth modality images of the driver's driving behavior from each viewing angle; The step S500 further includes: when it is detected that the driving behavior of the driver at each viewing angle is abnormal, determining that the driving behavior of the driver is abnormal.

7. The abnormal driving behavior detection method based on cross-modal image fusion recognition according to claim 1 or 6, characterized in that: In step S100, the camera collects video data including the infrared modality image and the depth modality image, and sends the video data to the control module. The control module sends the video data to the edge device. The edge device detects the driver's abnormal driving behavior under the corresponding perspective based on the video data in accordance with steps S200 to S500, and returns the detection result to the control module. The control module sends the detection result and the collected positioning data to the monitoring end.

Images