Helmet wearing multi-mode detection method fusing semantic syntax tree and reflectivity perception

Abstract

The invention discloses a helmet wearing multi-modal detection method fusing a semantic syntax tree and reflectivity perception. The method comprises the following steps: S1, an input stage: receiving an image and text pair input as a detection target; s2, a HelmetGuard-DINO model processing stage: an input image enters an image backbone network formed by a Swin Transform, and the HelmetGuard-DINO model is subjected to HelmetGuard-DINO model processing stage; outputting a four-level feature map through the layered structure; a dynamic convolution kernel is introduced in the C5 stage, and the convolution weight is dynamically adjusted through text embedding; text encoder processing: processing an input text by a text encoder constructed based on a 12-layer BERT model; feature enhancement layer fusion: using a bidirectional cross-modal attention mechanism to calculate an attention weight from vision to language; the CBAM attention module is used for enhancing the features, so that the model can be focused on a target area more accurately, key features are highlighted, background noise is suppressed, and the detection precision is effectively improved; and the recall rate is enhanced: the recall rate of a small target is improved by introducing a CBAM module, especially in application in a feature pyramid network.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a multimodal helmet wearing detection method that integrates a semantic syntax tree with reflectivity perception. Background Art

[0002] In the field of object detection, the continuous expansion of application scenarios and increasing complexity of requirements place higher demands on the performance of detection models. Traditional object detection algorithms often suffer from insufficient detection accuracy and low recall when faced with complex scenes, small object detection, and multi-category recognition. For example, in some practical applications, such as detecting the wearing of electric vehicle helmets in traffic monitoring, the model must be able to accurately identify helmets from different angles and lighting conditions, and must have high detection speeds to meet real-time requirements. Current mainstream object detection models, such as the YOLO series, have their own strengths and weaknesses in performance, but none of them fully meet the requirements for high-precision and high-recall detection in complex scenarios. Therefore, developing a technical solution that can effectively improve object detection performance is of great practical significance.

[0003] To this end, the present invention proposes a multimodal helmet wearing detection method that integrates semantic syntax tree and reflectivity perception. Summary of the Invention

[0004] The purpose of the present invention is to provide a multimodal helmet wearing detection method that integrates semantic syntax trees and reflectivity perception to solve the problems raised in the above background technology.

[0005] To achieve the above-mentioned object, the present invention provides the following technical solution: a multimodal helmet wearing detection method integrating semantic syntax tree and reflectivity perception, the specific steps comprising: S1: input stage: receiving image and text pair input as detection targets; S2: HelmetGuard-DINO model processing stage: Image backbone network processing: The input image enters the image backbone network composed of Swin Transformer. A spatiotemporal deblurring unit is inserted after the Swin Transformer's Patch Merging layer, and cycling motion trajectory features are learned through 3D convolution: in The features of the current frame are dynamically cached in the video buffer, effectively suppressing motion blur caused by riding speed. The network outputs a 4-level feature map through a hierarchical structure. In the C4 stage, a reflectivity perception branch is added to generate a high-reflectivity area mask using the saturation channel S and brightness channel V of the HSV color space: ; in A learnable helmet reflective template is used. In the C5 stage, dynamic convolution kernels are introduced to dynamically adjust convolution weights using text embedding to improve feature extraction capabilities for different scenes and targets. Text encoder processing: The text encoder built based on the 12-layer BERT model processes the input text; compliant syntax tree embedding: Parse the security standard text into a syntax tree structure and generate rule embedding vectors through graph neural network (GNN) , and the original text features Fusion: is a dynamic weight coefficient learned through the attention mechanism.

[0006] - Dangerous behavior keyword enhancement: Build a vocabulary of dangerous cycling behaviors (such as "not wearing a helmet") and inject keyword bias into the Self-Attention layer of BERT: is the keyword position bias matrix, Dynamically generated by the scenario risk level prediction module; Feature enhancement layer fusion: Using the bidirectional cross-modal attention mechanism, given the visual feature V and text feature L, the calculation process is as follows Among them, Q v =VW q is the query projection of visual features, W q is the learnable weight matrix; K l =LW k is the key projection of text features, W k is the learnable weight matrix; V l =LW v is the value projection of the text feature, W v is the learnable weight matrix; d k Is the dimension of the key vector, used to scale the dot product result to avoid gradient disappearance; Softmax is along the text sequence dimension (ie K l sequence length) normalized attention weights; Calculate the attention weight from vision to language; deeply fuse the visual features extracted by the image backbone network with the text features output by the text encoder to form a cross-modal representation with semantic consistency; Language-guided query selection: mapping text features to initial object queries; Cross-modal decoder processing: The cross-modal decoder consists of 6 cascaded Transformer layers, processing both visual and linguistic features at each layer. During decoding, the similarity score between each predicted box and the text description is calculated, and the prediction with the highest score is finally selected as the output for the REC task. For the OD task, the target box with the corresponding noun phrase is output. S3: CBAM attention module processing stage: Among them, M c and M s Represent channel and spatial attention mapping respectively, represents element-wise multiplication, F in is the input feature map, , where C, H, and W are the number of channels, height, and width respectively; Collaboration with Swin Transformer: When CBAM is combined with Swin Transformer, Swin's windowed self-attention mechanism complements CBAM's local attention; the former captures long-range dependencies, while the latter enhances local details. Cross-modal task optimization: When processing cross-modal tasks with image and text input, the cross-modal CBAM variant introduces text embedding vectors as attention guidance signals in the feature enhancement stage; S4: After the collaborative processing of the HelmetGuard-DINO model and the CBAM attention module, the target detection results are finally output.

[0007] Preferably, in S1, the image serves as the source of visual information to be detected, covering scenes in various actual scenarios; the text contains descriptive information related to the target, which can be a single text composed of all category names in the detection task, used for the target detection task, or a reference text, used for the reference expression understanding task.

[0008] Preferably, in S1, the dataset used is the SCAUHDM dataset, wherein the image sources include photos in real scenes, screenshots from movies, Internet images, and Kaggle datasets.

[0009] Preferably, in S2, the text encoder supports a maximum input of 256 tokens.

[0010] Preferably, the language-guided query selection is specifically performed by selecting feature vectors corresponding to noun phrases in text embeddings and converting them into DINO decoder queries through a learnable projection matrix.

[0011] Preferably, in said S4, in the object detection OD task, a target frame with a corresponding noun phrase is output; in the referential expression understanding REC task, a prediction result with the highest matching degree with the text description is output.

[0012] Compared with the existing technology, the beneficial effects of the present invention are: improving detection accuracy: through the enhancement of features by the CBAM attention module, the model can focus on the target area more accurately, highlight key features, and suppress background noise, thereby effectively improving detection accuracy; in complex scenarios, it can accurately identify and locate targets, reducing the occurrence of false detections and missed detections. Enhanced recall: The introduction of the CBAM module, especially its application in the feature pyramid network, improves the recall of small objects. At the same time, HelmetGuard-DINO's inherent multimodal feature fusion capabilities, combined with CBAM's feature optimization, enable the model to better capture objects of different scales and angles, significantly enhancing the recall rate. Adaptability to complex scenarios: HelmetGuard-DINO's open-set target detection capability enables it to handle targets of different categories. Combined with CBAM's adaptive feature optimization under different tasks and scenarios, this technical solution can better adapt to complex and changing application scenarios, such as target detection tasks in complex environments in fields such as traffic monitoring and industrial inspection. Obvious performance advantages: Comparative experimental results fully demonstrate the advantages of this technical solution in various performance indicators. Compared with the baseline model and other mainstream models, it can achieve higher detection accuracy and recall rate while ensuring a certain consumption of computing resources, providing a more reliable and efficient solution for the practical application of target detection technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 This is a structural diagram of the HelmetGuard-DINO model of the present invention; DETAILED DESCRIPTION

[0014] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention. Example 1

[0015] See also Figure 1-Figure 2The present invention provides a technical solution: a multimodal helmet wearing detection method that integrates semantic syntax trees and reflectivity perception, combining the HelmetGuard-DINO model with the CBAM attention module. HelmetGuard-DINO adopts a dual encoder-single decoder architecture to achieve open set target detection through multimodal feature fusion; it can receive image and text inputs and simultaneously output target boxes and their corresponding noun phrases; the specific steps include: S1: Input stage: The object detection technology of the present invention accepts an image and text pair as input. The image, as the source of visual information to be detected, covers scenes from various real-world scenarios; the text contains descriptive information related to the object. It can be a single text composed of all the category names in the detection task, used for the object detection task, or a reference text, used for the reference expression understanding task.

[0016] The dataset used is the SCAUHDM dataset, whose images come from multiple sources, including real-world photos, movie screenshots, internet images, and Kaggle datasets. The following is a breakdown of the number of images in the SCAUHDM dataset.

[0017] SCAUHDM dataset Number of pictures Wear a helmet Not wearing a helmet Cycling riding an electric motorcycle training set 8000 6853 3053 6252 8642 Test set 2000 1799 453 1471 2509 total 10000 8652 3506 7723 11151

[0018] S2: HelmetGuard-DINO model processing stage: Image backbone network processing: The input image enters the image backbone network composed of Swin Transformer. A spatiotemporal deblurring unit is inserted after the Swin Transformer's Patch Merging layer, and cycling motion trajectory features are learned through 3D convolution: in The features of the current frame are dynamically cached in the video buffer, effectively suppressing motion blur caused by riding speed. The network outputs a 4-level feature map through a hierarchical structure. In the C4 stage, a reflectivity perception branch is added to generate a high-reflectivity area mask using the saturation channel S and brightness channel V of the HSV color space: ; in A learnable helmet reflective template is used. In the C5 stage, dynamic convolution kernels are introduced to dynamically adjust convolution weights using text embedding to improve feature extraction capabilities for different scenes and targets. Text encoder processing: The text encoder built based on the 12-layer BERT model processes the input text; compliant syntax tree embedding: Parse the security standard text into a syntax tree structure and generate rule embedding vectors through graph neural network (GNN) , and the original text features Fusion: is a dynamic weight coefficient learned through the attention mechanism.

[0019] - Dangerous behavior keyword enhancement: Build a vocabulary of dangerous cycling behaviors (such as "not wearing a helmet") and inject keyword bias into the Self-Attention layer of BERT: is the keyword position bias matrix, Dynamically generated by the scenario risk level prediction module; Feature enhancement layer fusion: Using the bidirectional cross-modal attention mechanism, given the visual feature V and text feature L, the calculation process is as follows Among them, Q v =VW q is the query projection of visual features, W q is the learnable weight matrix; K l =LW k is the key projection of text features, W k is the learnable weight matrix; V l =LW v is the value projection of the text feature, W v is the learnable weight matrix; d k Is the dimension of the key vector, used to scale the dot product result to avoid gradient disappearance; Softmax is along the text sequence dimension (ie K l sequence length) normalized attention weights; Calculate the attention weight from vision to language; deeply fuse the visual features extracted by the image backbone network with the text features output by the text encoder to form a cross-modal representation with semantic consistency; Language-guided query selection: mapping text features to initial object queries; Cross-modal decoder processing: The cross-modal decoder consists of 6 cascaded Transformer layers, which process visual and language features simultaneously at each layer. During the decoding process, the similarity score between each predicted box and the text description is calculated, and the prediction with the highest score is finally selected as the output of the REC task. For the OD task, the target box with the corresponding noun phrase is output.

[0020] S3: CBAM attention module processing stage: Feature Pyramid Network Integration: CBAM is embedded in the lateral connection of the Feature Pyramid Network FPN, and its mathematical expression is Among them, M c and M s Represent channel and spatial attention mapping respectively, represents element-wise multiplication, Fin is the input feature map, where C, H, and W represent the number of channels, height, and width, respectively. This integration improves background noise suppression by 37% in the P5 layer feature map, increases small object recall by 4.1 percentage points on the GoldG test set, and significantly enhances the stability of the gradient flow for cross-scale feature fusion.

[0021] Synergy with Swin Transformer: When CBAM is combined with Swin Transformer, Swin's windowed self-attention mechanism complements CBAM's local attention. The former captures long-range dependencies, while the latter enhances local details. In a text-guided detection task, this combination improves the model's localization accuracy for spatial pronouns such as "helmet" in text descriptions by 12.6%.

[0022] Cross-modal task optimization: When processing cross-modal tasks with image and text input pairs, the improved cross-modal CBAM variant introduces text embedding vectors as attention guidance signals during the feature enhancement stage. This design improves the model's visual response to attribute words in text descriptions, such as "red" and "circle," by 28%, and outperforms the baseline method by 4.9 percentage points in the Intersection over Union (IoU) metric on the REC task.

[0023] S4: Output stage: The HelmetGuard-DINO model and the CBAM attention module collaborate to produce the final object detection results. In the object detection (OD) task, the target box with the corresponding noun phrase is output; in the referential expression comprehension (REC) task, the prediction with the highest match to the text description is output. Furthermore, comparative experiments with the baseline model GrundingDINO and the YOLO family models yolov8x and yolov9e demonstrate that the HelmetGuard-DINO + CBAM solution demonstrates superior performance across various metrics, including precision, recall, mAP50 (mean average precision at an IoU threshold of 0.5), and mAP50-95 (mean average precision at IoU thresholds ranging from 0.5 to 0.95). The specific data is shown in the table below.

[0024] Model Precision(%) Recall(%) mAP50(%) mAP50-95(%) Params(M) yolov8x 97.0 96.5 98.4 87.8 68.1 yolov9e 97.2 96.4 98.3 87.4 58.1 GrundingDINO 95.4 97.6 98.2 87.7 172.8 HelmetGuard-DINO+CBAM 97.0 98.2 98.6 88.4 173.2

[0025] Although the embodiments of the present invention have been shown and described, as detailed above, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the invention, and the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal helmet wearing detection method that integrates semantic syntax trees and reflectivity perception is characterized by: The specific steps include: S1: input stage: receiving image and text pair input as detection targets; S2: HelmetGuard-DINO model processing stage: Image backbone network processing: The input image enters the image backbone network composed of Swin Transformer. A spatiotemporal deblurring unit is inserted after the Swin Transformer's Patch Merging layer, and cycling motion trajectory features are learned through 3D convolution: in The features of the current frame are dynamically cached in the video buffer, effectively suppressing motion blur caused by riding speed. The network outputs a 4-level feature map through a hierarchical structure. In the C4 stage, a reflectivity perception branch is added to generate a high-reflectivity area mask using the saturation channel S and brightness channel V of the HSV color space: ; in A learnable helmet reflective template is used. In the C5 stage, dynamic convolution kernels are introduced to dynamically adjust convolution weights using text embedding to improve feature extraction capabilities for different scenes and targets. Text encoder processing: The text encoder built based on the 12-layer BERT model processes the input text; compliant syntax tree embedding: Parse the security standard text into a syntax tree structure and generate rule embedding vectors through graph neural network (GNN) , and the original text features Fusion: is a dynamic weight coefficient, learned through the attention mechanism; - Dangerous behavior keyword enhancement: Build a vocabulary of dangerous cycling behaviors and inject keyword bias into the Self-Attention layer of BERT: is the keyword position bias matrix, Dynamically generated by the scenario risk level prediction module; Feature enhancement layer fusion: Using the bidirectional cross-modal attention mechanism, given the visual feature V and text feature L, the calculation process is as follows Among them, Q v =VW q is the query projection of visual features, W q is the learnable weight matrix; K l =LW k is the key projection of text features, W k is the learnable weight matrix; V l =LW v is the value projection of the text feature, W v is the learnable weight matrix; d k Is the dimension of the key vector, used to scale the dot product result to avoid gradient disappearance; Softmax is along the text sequence dimension (ie K l sequence length) normalized attention weights; Calculate the attention weight from vision to language; deeply fuse the visual features extracted by the image backbone network with the text features output by the text encoder to form a cross-modal representation with semantic consistency; Language-guided query selection: mapping text features to initial object queries; Cross-modal decoder processing: The cross-modal decoder consists of 6 cascaded Transformer layers, processing both visual and linguistic features at each layer. During decoding, the similarity score between each predicted box and the text description is calculated, and the prediction with the highest score is finally selected as the output for the REC task. For the OD task, the target box with the corresponding noun phrase is output. S3: CBAM attention module processing stage: Feature Pyramid Network Integration: CBAM is embedded in the lateral connection of the Feature Pyramid Network FPN, and its mathematical expression is Among them, M c and M s Represent channel and spatial attention mapping respectively, represents element-wise multiplication, F in is the input feature map, (where C, H, and W are the number of channels, height, and width, respectively); Collaboration with Swin Transformer: When CBAM is combined with Swin Transformer, Swin's windowed self-attention mechanism complements CBAM's local attention; the former captures long-range dependencies, while the latter enhances local details. Cross-modal task optimization: When processing cross-modal tasks with image and text input, the cross-modal CBAM variant introduces text embedding vectors as attention guidance signals in the feature enhancement stage; S4: After the collaborative processing of the HelmetGuard-DINO model and the CBAM attention module, the target detection results are finally output.

2. The multimodal helmet wearing detection method integrating semantic syntax tree and reflectivity perception according to claim 1 is characterized by: In S1, images serve as the source of visual information to be detected, covering scenes in various real-world scenarios; text contains descriptive information related to the target, which can be a single text composed of all category names in the detection task, used for the target detection task, or a reference text, used for the reference expression understanding task.

3. The multimodal helmet wearing detection method integrating semantic syntax tree and reflectivity perception according to claim 2 is characterized by: In S1, the dataset used is the SCAUHDM dataset, where the image sources include photos in real scenes, screenshots from movies, Internet images, and Kaggle datasets.

4. The multimodal helmet wearing detection method integrating semantic syntax tree and reflectivity perception according to claim 1 is characterized by: In S2, the text encoder supports a maximum input of 256 tokens.

5. The multimodal helmet wearing detection method integrating semantic syntax tree and reflectivity perception according to claim 1 is characterized by: The specific method of the language-guided query selection is to select the feature vectors corresponding to the noun phrases in the text embedding and convert them into DINO decoder queries through a learnable projection matrix.

6. The multimodal helmet wearing detection method integrating semantic syntax tree and reflectivity perception according to claim 1 is characterized by: In the above S4, in the object detection OD task, the target frame with the corresponding noun phrase is output; in the referential expression understanding REC task, the prediction result with the highest matching degree with the text description is output.

Images