A Multimodal Helmet Wearing Detection Method Integrating Semantic Syntax Tree and Reflectivity Awareness

Abstract

This invention discloses a helmet-wearing multimodal detection method that integrates semantic syntax trees and reflectivity perception. S1: Input stage: Receives image and text pairs as input targets. S2: HelmetGuard-DINO model processing stage: The input image enters the image backbone network composed of Swing Transformers; outputs 4-level feature maps through a hierarchical structure; introduces dynamic convolutional kernels in stage C5, dynamically adjusting convolutional weights using text embedding; text encoder processing: A text encoder based on a 12-layer BERT model processes the input text; feature enhancement layer fusion: Calculates visual-to-language attention weights using a bidirectional cross-modal attention mechanism; improved detection accuracy: The CBAM attention module enhances features, enabling the model to focus more accurately on the target region, highlighting key features and suppressing background noise, thus effectively improving detection accuracy; enhanced recall: The introduction of the CBAM module, especially its application in the feature pyramid network, improves the recall rate for small targets.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a helmet-wearing multimodal detection method that integrates semantic syntax tree and reflectivity perception. Background Technology

[0002] In the field of object detection, with the continuous expansion of application scenarios and the increasing complexity of needs, higher demands are being placed on the performance of detection models. Traditional object detection algorithms often suffer from insufficient detection accuracy and low recall when facing problems such as complex scenes, small object detection, and multi-class recognition. For example, in some practical applications, such as detecting the wearing status of electric bicycle helmets in traffic monitoring, the model needs to be able to accurately identify helmets under different angles and lighting conditions, and also needs to have high detection speed to meet real-time requirements. Current mainstream object detection models, such as the YOLO series, each have their own advantages and disadvantages in performance, but none of them can fully meet the detection requirements of high accuracy and high recall in complex scenes. Therefore, developing a technical solution that can effectively improve object detection performance is of great practical significance.

[0003] To this end, this invention proposes a helmet-wearing multimodal detection method that integrates semantic syntax tree and reflectivity perception. Summary of the Invention

[0004] The purpose of this invention is to provide a helmet-wearing multimodal detection method that integrates semantic syntax tree and reflectivity perception to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a helmet-wearing multimodal detection method that integrates semantic syntax tree and reflectivity perception, the specific steps of which include: S1: Input stage: receiving image and text pairs as input as detection targets;

[0006] S2: HelmetGuard-DINO model processing stage:

[0007] Image backbone processing: The input image enters the image backbone network composed of Swin Transformers; a spatiotemporal deblurring unit is inserted after the Patch Merging layer of the Swin Transformer, and the cycling motion trajectory features are learned through 3D convolution: in The current frame features are used as the basis for dynamic caching of features from adjacent frames through a video buffer, effectively suppressing motion blur caused by cycling speed. The network outputs a 4-level feature map through a hierarchical structure. A reflectivity-sensing branch is added in the C4 stage, using the saturation channel S and the luminance channel V of the HSV color space to generate a mask for highly reflective areas. ; in A learnable helmet reflective template is provided; a dynamic convolution kernel is introduced in the C5 stage, and the convolution weights are dynamically adjusted by text embedding, thereby improving the feature extraction capability for different scenarios and targets;

[0008] Text encoder processing: A text encoder built on a 12-layer BERT model processes the input text; Compliance syntax tree embedding:

[0009] The security standard text is parsed into a syntax tree structure, and rule embedding vectors are generated using a graph neural network (GNN). Features of the original text Fusion: These are dynamic weight coefficients, learned through an attention mechanism.

[0010] - Reinforcement of keywords related to dangerous behaviors:

[0011] Build a vocabulary of dangerous cycling behaviors (such as "not wearing a helmet") and inject keyword bias into BERT's Self-Attention layer: This is the keyword position bias matrix. Dynamically generated by the scenario risk level prediction module;

[0012] Feature enhancement layer fusion: Employing a bidirectional cross-modal attention mechanism, given visual features V and textual features L, the calculation process is as follows. Among them, Q v =VW q It is a query projection of visual features, W q It is a learnable weight matrix; K l =LW k It is the key projection of text features, W k It is a learnable weight matrix; V l =LW v W is the value projection of text features. v It is a learnable weight matrix; d k It is the dimension of the key vector, used to scale the dot product result and avoid gradient vanishing; Softmax follows the dimension of the text sequence (i.e., K). l Normalized attention weights (sequence length);

[0013] Calculate the attention weights from vision to language; deeply fuse the visual features extracted by the image backbone network with the text features output by the text encoder to form a cross-modal representation with semantic consistency;

[0014] Language-guided query selection: Maps text features to initial object queries;

[0015] Cross-modal decoder processing: The cross-modal decoder consists of 6 cascaded Transformer layers, which simultaneously process visual and linguistic features in each layer. During the decoding process, the similarity score between each predicted bounding box and the text description is calculated, and the prediction with the highest score is finally selected as the REC task output. For the OD task, the target bounding box with the corresponding noun phrase is output.

[0016] S3: CBAM Attention Module Processing Stage: Where M c and M s These represent channel and spatial attention mapping, respectively. F represents element-wise multiplication. in For the input feature map, , where C, H, and W are the number of channels, height, and width, respectively;

[0017] Collaboration with Swing Transformer: When CBAM is combined with Swing Transformer, Swing’s window self-attention mechanism complements CBAM’s local attention; the former captures long-distance dependencies, while the latter enhances local detail features.

[0018] Cross-modal task optimization: When dealing with cross-modal tasks with Image and Text inputs, the cross-modal CBAM variant introduces text embedding vectors as attention guidance signals during the feature enhancement stage;

[0019] S4: After collaborative processing by the HelmetGuard-DINO model and the CBAM attention module, the final target detection result is output.

[0020] Preferably, in step S1, the image serves as the source of visual information to be detected, covering images from various real-world scenarios; the text contains descriptive information related to the target, which can be either a single text composed of all category names in the detection task for target detection, or referential text applied to referential expression understanding tasks.

[0021] Preferably, in step S1, the dataset used is the SCAUHDM dataset, whose image sources include photos from real-world scenes, screenshots from movies, internet images, and the Kaggle dataset.

[0022] Preferably, in step S2, the text encoder supports a maximum of 256 tokens as input.

[0023] Preferably, the specific method for language-guided query selection is to select the feature vectors corresponding to noun phrases in the text embedding and convert them into DINO decoder queries through a learnable projection matrix.

[0024] Preferably, in step S4, in the object detection (OD) task, the target bounding box with the corresponding noun phrase is output; in the representation expression comprehension (REC) task, the prediction result with the highest matching degree to the text description is output.

[0025] Compared with the prior art, the beneficial effects of the present invention are: improved detection accuracy: through the enhancement effect of the CBAM attention module on features, the model can more accurately focus on the target area, highlight key features, and suppress background noise, thereby effectively improving detection accuracy; in complex scenes, it can accurately identify and locate targets, reducing the occurrence of false detections and missed detections.

[0026] Enhanced recall: The introduction of the CBAM module, especially its application in the feature pyramid network, improves the recall rate for small targets; at the same time, the multimodal feature fusion capability of HelmetGuard-DINO itself, combined with the feature optimization of CBAM, enables the model to better capture targets of different scales and angles, thereby significantly enhancing the recall rate.

[0027] Adaptable to complex scenarios: HelmetGuard-DINO's open set target detection capability enables it to handle different types of targets. Combined with CBAM's adaptive feature optimization in different tasks and scenarios, this technical solution can better adapt to complex and ever-changing application scenarios, such as target detection tasks in complex environments in traffic monitoring, industrial inspection and other fields.

[0028] Significant performance advantages: Comparative experimental results fully demonstrate the advantages of this technical solution in various performance indicators. Compared with the benchmark model and other mainstream models, it can achieve higher detection accuracy and recall rate while ensuring a certain amount of computing resources are consumed, providing a more reliable and efficient solution for the practical application of target detection technology. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the process of the present invention;

[0030] Figure 2 This is a structural diagram of the HelmetGuard-DINO model of the present invention; Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example 1

[0032] Please see Figures 1-2 This invention provides a technical solution: a helmet-wearing multimodal detection method that integrates semantic syntax trees and reflectivity perception, combining the HelmetGuard-DINO model with a CBAM attention module. HelmetGuard-DINO employs a dual-encoder-single-decoder architecture, achieving open-set target detection through multimodal feature fusion; it can receive both image and text inputs, simultaneously outputting the target bounding box and its corresponding noun phrase; the specific steps include: S1: Input stage:

[0033] The target detection technology of this invention receives image and text pairs as input. The image serves as the source of visual information to be detected, covering various scenes from real-world applications; the text contains descriptive information related to the target. This text can be a single piece of text composed of all category names in the detection task, used for target detection (OD) tasks, or it can be referential text applied to referential expression comprehension (REC) tasks.

[0034] The dataset used is the SCAUHDM dataset, whose images come from multiple sources, including real-world photographs, movie screenshots, internet images, and Kaggle datasets. The following is a breakdown of the number of images in the SCAUHDM dataset.

[0035] SCAUHDM dataset Number of images Wearing a helmet Not wearing a helmet Cycling riding an electric motorcycle training set 8000 6853 3053 6252 8642 test set 2000 1799 453 1471 2509 total 10000 8652 3506 7723 11151

[0036] S2: HelmetGuard-DINO model processing stage:

[0037] Image backbone processing: The input image enters the image backbone network composed of Swin Transformers; a spatiotemporal deblurring unit is inserted after the Patch Merging layer of the Swin Transformer, and the cycling motion trajectory features are learned through 3D convolution: in The current frame features are used as the basis for dynamic caching of features from adjacent frames through a video buffer, effectively suppressing motion blur caused by cycling speed. The network outputs a 4-level feature map through a hierarchical structure. A reflectivity-sensing branch is added in the C4 stage, using the saturation channel S and the luminance channel V of the HSV color space to generate a mask for highly reflective areas. ; in A learnable helmet reflective template is provided; a dynamic convolution kernel is introduced in the C5 stage, and the convolution weights are dynamically adjusted by text embedding, thereby improving the feature extraction capability for different scenarios and targets;

[0038] Text encoder processing: A text encoder built on a 12-layer BERT model processes the input text; Compliance syntax tree embedding:

[0039] The security standard text is parsed into a syntax tree structure, and rule embedding vectors are generated using a graph neural network (GNN). Features of the original text Fusion: These are dynamic weight coefficients, learned through an attention mechanism.

[0040] - Reinforcement of keywords related to dangerous behaviors:

[0041] Build a vocabulary of dangerous cycling behaviors (such as "not wearing a helmet") and inject keyword bias into BERT's Self-Attention layer: This is the keyword position bias matrix. Dynamically generated by the scenario risk level prediction module;

[0042] Feature enhancement layer fusion: Employing a bidirectional cross-modal attention mechanism, given visual features V and textual features L, the calculation process is as follows. Among them, Q v =VW q It is a query projection of visual features, W q It is a learnable weight matrix; K l =LW k It is the key projection of text features, W k It is a learnable weight matrix; V l =LW v W is the value projection of text features. v It is a learnable weight matrix; d k It is the dimension of the key vector, used to scale the dot product result and avoid gradient vanishing; Softmax follows the dimension of the text sequence (i.e., K). l Normalized attention weights (sequence length);

[0043] Calculate the attention weights from vision to language; deeply fuse the visual features extracted by the image backbone network with the text features output by the text encoder to form a cross-modal representation with semantic consistency;

[0044] Language-guided query selection: Maps text features to initial object queries;

[0045] Cross-modal decoder processing: The cross-modal decoder consists of 6 cascaded Transformer layers, which simultaneously process visual and linguistic features in each layer. During the decoding process, the similarity score between each predicted bounding box and the text description is calculated, and the prediction with the highest score is finally selected as the REC task output. For the OD task, the target bounding box with the corresponding noun phrase is output.

[0046] S3: CBAM Attention Module Processing Stage:

[0047] Feature Pyramid Network Ensemble: CBAM is embedded at the lateral connections of the Feature Pyramid Network (FPN), and its mathematical expression is as follows: Where M c and M s These represent channel and spatial attention mapping, respectively. F represents element-wise multiplication. in For the input feature map, Where C, H, and W represent the number of channels, height, and width, respectively. Through this integration method, the background noise suppression effect of the P5 layer feature map is improved by 37%, the small target recall rate increases by 4.1 percentage points on the GoldG test set, and the gradient flow stability of cross-scale feature fusion is significantly enhanced.

[0048] Collaboration with the Swin Transformer: When CBAM is combined with the Swin Transformer, the window self-attention mechanism of Swin complements the local attention of CBAM. The former captures long-distance dependencies, while the latter enhances local detail features. In text-guided detection tasks, this combination improves the model's accuracy in locating spatial pronouns such as "helmet" in text descriptions by 12.6%.

[0049] Cross-modal task optimization: When handling cross-modal tasks with Image and Text inputs, the improved cross-modal CBAM variant introduces text embedding vectors as attention-guiding signals during the feature enhancement stage. This design improves the model's visual response intensity to attribute words in the text description, such as "red" and "circle," by 28%, and outperforms the baseline method by 4.9 percentage points in IoU on the REC task.

[0050] S4: Output Phase

[0051] After collaborative processing by the HelmetGuard-DINO model and the CBAM attention module, the final target detection result is output. In the target detection OD task, the target box with the corresponding noun phrase is output; in the representation expression understanding REC task, the prediction result with the highest matching degree to the text description is output. Furthermore, through comparative experiments with the benchmark model GrundingDINO and YOLO series models yolov8x and yolov9e, the HelmetGuard-DINO + CBAM scheme of this invention performs excellently in performance metrics such as precision, recall, mAP50 (mean accuracy, with an IoU threshold of 0.5), and mAP50-95 (mean accuracy, with an IoU threshold from 0.5 to 0.95). Specific data are shown in the table below.

[0052] Model Precision (%) Recall(%) mAP50(%) mAP50-95% (%) Params(M) yolov8x 97.0 96.5 98.4 87.8 68.1 yolov9e 97.2 96.4 98.3 87.4 58.1 GrundingDINO 95.4 97.6 98.2 87.7 172.8 HelmetGuard-DINO+CBAM 97.0 98.2 98.6 88.4 173.2

[0053] Although embodiments of the invention have been shown and described in detail above, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method of fusing semantic parse trees with reflectance-aware headgear multi-modal detection, characterized by: Specific steps include: S1: input stage: receiving image and text pair input as detection target; S2: HelmetGuard-DINO model processing stage: Image backbone network processing: the input image enters the image backbone network composed of Swin Transformer; after the Patch Merging layer of Swin Transformer, the space-time deblurring unit is inserted, and the riding motion trajectory features are learned through 3D convolution: ; wherein is the current frame feature, the adjacent frame feature is dynamically cached through a video buffer, effectively suppressing motion blur caused by riding speed; Conv3D is a 3D convolution: sliding on a video or a sequence of consecutive frames 3D volume: time T x height H x width W, extracting spatio-temporal features; the convolution kernel itself is also three-dimensional depth x height x width, which also extends in the time dimension; LSTM is a special kind of recurrent neural network, whose core is to solve the "long-term dependence forgetting" problem encountered by standard RNN when processing long sequences; it is achieved through a sophisticated "gate mechanism" and "cell state"; cell state: it runs through the entire timeline, responsible for carrying information over long distances in the sequence; its change is slow and controlled; Gate mechanism: a "switch" that controls the flow of information in, retention, and out, realized by Sigmoid function and point-wise multiplication; Forget gate: decides which old information to discard from the cell state; Input gate: decides which new information to store in the cell state; Output gate: based on the current input and cell state, decides what information to output to the next time step and as the current hidden state; The network outputs 4-level feature maps through a hierarchical structure; in the C4 stage, the reflectivity perception branch is added, and the saturation channel S and the brightness channel V of the HSV color space are used to generate a high-reflective area mask: ; ; wherein is a learnable helmet reflectivity template; F is the original feature that needs to be "fixed" or "enhanced"; MLP is the abbreviation of Multi-Layer Perceptron, which is the most classic and basic artificial neural network structure; its core composition is: Fully connected layer: each neuron in the layer is connected to all neurons in the previous layer; Nonlinear activation function: nested between layers, enables the network to learn and model complex nonlinear relationships; in the C5 stage, dynamic convolution kernels are introduced, and the convolution weights are dynamically adjusted using text embedding to improve the feature extraction ability for different scenes and targets; Specific process: for each spatial position (i,j) on the feature map, the two-dimensional vector [s,v] is processed independently by the MLP; the following calculations are performed: First layer: map the 2D input to a higher dimension, introduce nonlinearity, and learn the primary interaction features between S and V; Intermediate layer: further combine these features to form higher-level representations, identify that "bright but saturated" may not be reflective, and "bright and not saturated" is likely to be reflective; Output layer: map the high-dimensional features to a single-valued scalar representing the original score of the "high-reflective area" for that pixel point; Output and activation: σ(...) σ refers to the Sigmoid activation function; it compresses the original score output by the MLP to the interval (0,1); The final M reflect is a mask with the same spatial dimensions [H, W] as the input image, where the value of each pixel represents the probability that the location belongs to a high-reflectance region; the closer the value is to 1, the more likely the location is to reflect light. Text encoder processing: the text encoder based on a 12-layer BERT model processes the input text; compliance syntax tree embedding: The security standard text is parsed into a syntax tree structure, and a rule embedding vector is generated through a graph neural network GNN , and the original text features are fused: ; Dynamic weight coefficients are learned through attention mechanisms; LayerNorm is short for "layer normalization," which adjusts data to a standard distribution with a mean of 0 and a variance of 1 in the feature dimension of a single sample; - Dangerous behavior keyword reinforcement: Build a riding dangerous behavior keyword library and inject keyword bias in the Self-Attention layer of BERT: ; biasing the matrix for keyword position, generated dynamically by a scene risk level prediction module; QK T is the most core calculation in the Self-Attention layer of BERT, representing the matrix multiplication between the query matrix Q and the transpose of the key matrix K; Feature enhancement layer fusion: use a bidirectional cross-modal attention mechanism, given visual features V and text features L, the calculation process is as follows ; where Q v = VW q is the query projection of visual features, W q is a learnable weight matrix; K l = LW k is the key projection of textual features, W k is a learnable weight matrix; V l = LW v is the value projection of textual features, W v is a learnable weight matrix; d k is the dimension of key vectors, used to scale the dot product result to avoid gradient vanishing; Softmax normalizes the attention weights along the sequence length of K l , i.e., the sequence length of textual features. The attention weight of visual-to-language is calculated, and the visual features extracted by the image backbone network are deeply fused with the text features output by the text encoder to form a cross-modal representation with semantic consistency. Language-guided query selection: mapping text features to initial object queries. Cross-modal decoder processing: a cross-modal decoder composed of 6 cascaded Transformer layers processes visual and language features simultaneously at each layer; during decoding, the similarity score of each predicted box with the text description is calculated, and the prediction with the highest score is finally selected as the output of the REC task; for the OD task, the target box with the corresponding noun phrase is output. S3: CBAM attention module processing stage: Feature pyramid network integration: CBAM is embedded in the lateral connection of the feature pyramid network FPN, which is mathematically expressed as ; where M c and M s represent channel and spatial attention mapping, respectively, denotes element-wise multiplication, F in is the input feature map, where C, H, W are the number of channels, height and width, respectively. Collaboration with Swin Transformer: when CBAM is combined with Swin Transformer, the window self-attention mechanism of Swin and the local attention of CBAM form a complement; the former captures long-distance dependencies, and the latter emphasizes local detail features. Cross-modal task optimization: when processing cross-modal tasks with Image, Text input, the cross-modal CBAM variant introduces text embedding vectors as attention guide signals in the feature enhancement stage. S4: After the HelmetGuard-DINO model and the CBAM attention module are collaboratively processed, the final target detection result is output.

2. The method of claim 1, wherein the method further comprises: In S1, images serve as the source of visual information to be detected, covering pictures in real scenes; texts contain description information related to targets, which are single texts composed of all class names in the detection task and are also referred to as texts applied to reference expression understanding tasks.

3. The method of claim 2, wherein the method further comprises: In S1, the dataset used is the SCAUHDM dataset, which contains real scene photos, movie screenshots, internet images, and Kaggle datasets.

4. The method of claim 1, wherein the method further comprises: In S2, the text encoder supports a maximum of 256 tokens input.

5. The method of claim 1, wherein the method further comprises: The specific method of language-guided query selection is to select the feature vector corresponding to the noun phrase in the text embedding and convert it into a DINO decoder query through a learnable projection matrix.

6. The method of claim 1, wherein the method further comprises: In S4, in the target detection OD task, the target box with the corresponding noun phrase is output; in the reference expression understanding REC task, the prediction with the highest matching degree with the text description is output.

Images