Bridge side falling behavior recognition method and system in haze environment

Abstract

The application discloses a bridge-side falling behavior recognition method and system in a fog environment, and the method comprises the following steps: based on the bridge-side falling behavior data set in the fog environment and the model loss function, a pre-constructed RA-M fusion coding and decoding recognition network model is trained; the trained RA-M fusion coding and decoding recognition network model is used to recognize the bridge-side falling behavior in the fog environment; wherein the RA-M fusion coding and decoding recognition network model comprises an RA-M encoder module and an RA-M decoder module, the RA-M encoder module is used for multi-scale feature extraction and fusion of an image, and the RA-M decoder module is used for pedestrian mask and position recovery based on a reverse attention mechanism. The application can accurately recognize the position and behavior state of the falling pedestrian in the fog environment, realize the bridge-side falling behavior monitoring under the adverse weather conditions, provide early warning information for the rescue personnel in time, and improve the rescue response speed and success rate.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method and system for recognizing bridge fall behavior in hazy environments. Background Technology

[0002] In recent years, computer vision-based behavior recognition technology has been increasingly applied to public safety monitoring scenarios. Mainstream recognition methods typically rely on clear video streams captured by high-definition cameras, utilizing algorithms such as target detection and temporal action analysis to identify specific abnormal behaviors. However, when applied to open outdoor environments such as bridges, the performance of this technology is severely limited by weather conditions.

[0003] Haze is a common adverse weather condition where suspended particulate matter in the atmosphere degrades images, manifesting as decreased contrast, color distortion, and blurred details. This image quality degradation makes it difficult for traditional visual algorithms to reliably extract the effective appearance and motion features of moving targets, leading to a significant decrease or even failure in the accuracy of bridge fall detection. Existing research mostly utilizes image dehazing preprocessing to improve visibility; however, the dehazing process may introduce noise or artifacts, making it difficult for conventional algorithms to effectively extract pedestrian appearance and motion features, affecting the reliability and accuracy of subsequent recognition and detection.

[0004] Therefore, there is an urgent need for a fall behavior recognition method optimized for severe working conditions such as smog and suitable for bridge monitoring scenarios. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for identifying bridge fall behavior in hazy environments. This method and system can accurately identify the location and behavior of falling pedestrians in hazy environments, realize bridge fall behavior monitoring under severe weather conditions, provide timely early warning information for rescuers, and improve the speed and success rate of rescue response.

[0006] This invention provides the following technical solution:

[0007] Firstly, a method for recognizing bridge-side fall behavior in smoggy environments is provided, including:

[0008] We obtained real bridge fall behavior data under smog conditions, and after preprocessing, we obtained a bridge fall behavior dataset under smog conditions.

[0009] Based on the bridge fall behavior dataset and model loss function under hazy conditions, a pre-constructed RA-M fusion codec recognition network model is trained to obtain the trained RA-M fusion codec recognition network model.

[0010] The trained RA-M fusion encoder-decoder recognition network model was used to identify bridge fall behavior in hazy environments;

[0011] The RA-M fusion encoding and decoding recognition network model includes an RA-M encoder module and an RA-M decoder module. The RA-M encoder module is used for multi-scale feature extraction and fusion of images, and the RA-M decoder module is used for pedestrian masking and location recovery based on the inverse attention mechanism.

[0012] Furthermore, the method for preprocessing the real bridge fall behavior data under the hazy environment includes data augmentation, data annotation, and dataset partitioning;

[0013] The data augmentation includes: applying physical degradation operations to the image by randomly adjusting atmospheric light values ​​and randomly perturbing the scattering coefficient to generate an image sequence with continuously changing visibility; and increasing the number of samples by employing spatial and pixel-level enhancement strategies, including random cropping, multi-scale scaling, rotation, and adding motion blur.

[0014] The data annotation includes: performing binarization segmentation on the region of the fallen pedestrian in the image to generate a binarized segmentation mask image containing only the mask of the fallen pedestrian; annotating the target bounding boxes of the fallen person to generate a target bounding box annotation file; and obtaining a dataset sample that matches the original image, the binarized segmentation mask image, and the target bounding box annotation file.

[0015] The dataset partitioning includes dividing the dataset samples into a training set, a validation set, and a test set.

[0016] Furthermore, the RA-M encoder module is used to perform multi-scale feature extraction and fusion on the image, including:

[0017] A multi-layer Res2Net feature extraction layer is used to perform global feature extraction on the image, generating multiple sets of feature extraction maps with different semantic depth scales. Then, downsampling is performed to obtain multiple sets of original feature maps.

[0018] Each set of original feature maps is input into the alignment enhancement module, and the original feature maps are subjected to scale unification, layer normalization, and serialization flattening to obtain multiple sets of serialized features.

[0019] Multiple sets of sequential features are concatenated and input into the core feature fusion module. The Mamba module integrates multi-scale features into a unified feature representation at the same spatial scale. The sliding window structure of the SwinT module captures local structural features, enabling the fine-grained contour features of the human body and action semantics to be preserved simultaneously, thus achieving multi-scale feature fusion at different semantic depths.

[0020] Furthermore, the alignment enhancement module is used to perform scale unification, layer normalization, and serialization flattening on the original feature map, specifically including:

[0021] Input the original feature map The convolutional layers are scaled, and then the attention output feature map is generated through the CBAM module.

[0022] After adding the attention output feature map to the original feature map, layer normalization is performed.

[0023] The features output by the layer normalization process are serialized and flattened to obtain serialized features.

[0024] Furthermore, the core feature fusion module is used to achieve multi-scale feature fusion at different semantic depths, specifically including:

[0025] The concatenated serialized features are first non-linearized using linear layer mapping and the Gelu activation function; then, the sequence is divided into positive and negative sub-branches through reverse processing.

[0026] For the forward branch, the sequence is processed by layer normalization and then fed into the Mamba module. The Mamba module dynamically and selectively memorizes or ignores information based on the input content to obtain the forward sequence.

[0027] For the reverse branch, the sequence is fed into the SwinT module after being processed by layer normalization. Inside the SwinT module, the sequence is first reshaped into a two-dimensional feature map, and then the local structural features are captured through the sliding window attention mechanism to obtain the reverse sequence.

[0028] The forward and reverse sequences are input into the cross-attention fusion module for processing. The resulting sequences are then normalized, segmented, and reshaped to obtain multiple feature maps and attention guidance maps.

[0029] Furthermore, the input of the forward and reverse sequences into the cross-attention fusion module for processing includes:

[0030] The forward sequence X + After normalization, multiply by the parameter matrix of the appropriate size to obtain the query matrix Q;

[0031] Reverse sequence X - After normalization, multiply by the parameter matrix of the corresponding size to obtain the key matrix K and the value matrix V;

[0032] The query matrix Q and the key matrix K are multiplied, and the fused attention matrix is ​​obtained by applying the Softmax activation function;

[0033] The extraction matrix U is obtained by multiplying the fusion attention matrix by the value matrix V;

[0034] Extract the matrix U and the forward sequence X + The sequences are added together and then normalized to obtain the fused output sequence X. out ;

[0035] The processing procedure can be expressed by the following formula:

[0036] ;

[0037] ;

[0038] ;

[0039] ;

[0040] Among them, W Q W K W V d represents the parameter matrix, d represents the input dimension, and Norm represents the layer normalization operation.

[0041] Furthermore, the pedestrian masking and location recovery based on the reverse attention mechanism of the RA-M decoder module includes:

[0042] The multi-layer segmentation-guided fusion module uses reverse attention to guide the recovery of image details step by step, generating multiple intermediate mask images.

[0043] The intermediate mask image corresponding to the first-layer segmentation-guided fusion module is activated by the sigmoid activation function and upsampled to generate a mask image;

[0044] The mask image is processed by a self-attention module to generate weight information. The weight information is then multiplied by the mask image and passed through an anchorless detection head to generate the corresponding bounding box position information and category information.

[0045] Furthermore, the step-by-step restoration of image details via the multi-layer segmentation-guided fusion module based on reverse attention guidance includes:

[0046] feature map h m And attention-guided diagram p m The deepest segmentation-guided fusion module is input and processed in two ways: an attention-guided branch and a feature map-guided branch.

[0047] In the attention-guided branch, the feature map h m And attention-guided diagram p m Input the inverse attention fusion module to generate a new feature map through convolution;

[0048] In the feature map-guided branch, the feature map h m And attention-guided diagram pm Input the core feature fusion module, multiply it element-wise and serialize it to obtain a new attention guidance map;

[0049] The new feature map and the new attention guidance map are used as new inputs to the attention guidance branch and the feature map guidance branch to continue a new round of alternating optimization, and dense connections are added to each branch after each round of alternating optimization.

[0050] After two rounds of alternating optimization, the attention-guided map and feature map are fused through a reverse attention fusion module. After one convolutional layer, the output feature map r of the deepest segmentation-guided fusion module is obtained. m ;

[0051] The deepest segmentation guides the output feature map r of the fusion module. m With attention guidance diagram p m After addition, it is convolved and then combined with the feature map h from the previous layer. m-1 By matching dimensions, a new attention guidance graph p is obtained. m-1 Guide new attention to graph p m-1 With the previous layer feature map h m-1 Input the previous layer segmentation and fusion module to guide the enhancement of shallower textures;

[0052] The output feature maps of each layer segmentation guidance fusion module are added to the attention guidance maps input to the corresponding layer segmentation guidance fusion module, and multiple intermediate mask images are generated by convolutional dimensionality reduction.

[0053] Furthermore, the formula for the model loss function is as follows:

[0054] ;

[0055] ;

[0056] ;

[0057] ;

[0058] in, For the total loss, To divide the loss, To detect the loss, For attention monitoring loss, , , These represent the weight coefficients of the segmentation loss, detection loss, and attention supervision loss, respectively. and These represent the pixel value at the j-th position of the predicted mask and the pixel value at the j-th position of the real mask, respectively, where j represents the index of all pixels in the image. Represents the smoothing term; Represents classification loss, Represents the target bounding box regression loss. Dice(*) represents the balance coefficient between classification loss and bounding box regression loss; Function, i.e., calculation Loss coefficient, where G represents the attention guidance map output by the RA-M encoder. Represents the real mask image, i represents the layer sequence of the segmentation-guided fusion module, and m represents the number of layers in the segmentation-guided fusion module. This represents the Sigmoid activation function. Indicates an upsampling operation. This represents the intermediate mask image generated by the i-th layer segmentation-guided fusion module.

[0059] Secondly, a bridge fall behavior recognition system is provided for use in smoggy environments, including:

[0060] The dataset construction module is used to obtain real bridge fall behavior data under hazy conditions. After preprocessing, a bridge fall behavior dataset under hazy conditions is obtained.

[0061] The model training module is used to train a pre-built RA-M fusion codec recognition network model based on a dataset of bridge fall behavior under haze conditions and a model loss function, so as to obtain the trained RA-M fusion codec recognition network model.

[0062] The behavior recognition module is used to identify bridge-side fall behavior in hazy environments using a trained RA-M fusion encoding and decoding recognition network model.

[0063] The RA-M fusion encoding and decoding recognition network model includes an RA-M encoder module and an RA-M decoder module. The RA-M encoder module is used for multi-scale feature extraction and fusion of images, and the RA-M decoder module is used for pedestrian masking and location recovery based on the inverse attention mechanism.

[0064] Compared with the prior art, the beneficial effects of the present invention are:

[0065] (1) This invention obtains real bridge fall behavior data under haze conditions, and after preprocessing, obtains a bridge fall behavior dataset under haze conditions. The dataset includes a dataset sample that matches the original image, the binarized segmentation mask image and the target box annotation file. This fills the data gap in the field of bridge fall behavior recognition under haze conditions and improves the robustness of the model in haze environment training.

[0066] (2) This invention uses a trained RA-M fusion encoder-decoder recognition network model to identify bridge fall behavior in hazy environments. The RA-M fusion encoder-decoder recognition network model includes an RA-M encoder module and an RA-M decoder module. The RA-M encoder module is used for multi-scale feature extraction and fusion of images. The state space modeling capability of the Mamba module and the window attention mechanism of the SwinT module are complemented by the cross-attention fusion module, which can better perceive the target and enhance details in hazy environments. The RA-M decoder module is used for pedestrian masking and position recovery based on the reverse attention mechanism. The segmentation-guided fusion module adopts an alternating optimization structure of attention guidance map and feature map, which enhances the interaction between information, avoids information loss in a single path, and the added dense links can effectively prevent the gradient vanishing problem.

[0067] (3) The method provided by the present invention can effectively focus on the prominent area of ​​a fallen pedestrian in a foggy environment with visibility of not less than 150 meters, accurately identify the location and behavior of the fallen pedestrian, realize the monitoring of bridge-side fall behavior under severe weather conditions, provide timely early warning information for rescuers, and improve the speed and success rate of rescue response. Attached Figure Description

[0068] Figure 1 This is a flowchart illustrating the method for identifying bridge-side fall behavior in a hazy environment, as described in this embodiment of the invention.

[0069] Figure 2 This is a schematic diagram of the overall structure of the RA-M fusion encoding and decoding recognition network model in an embodiment of the present invention;

[0070] Figure 3 This is a schematic diagram of the alignment enhancement module in an embodiment of the present invention;

[0071] Figure 4 This is a schematic diagram of the core feature fusion module in an embodiment of the present invention;

[0072] Figure 5 This is a schematic diagram of the structure of the segmentation guidance fusion module in an embodiment of the present invention. Detailed Implementation

[0073] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0074] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0075] Example 1

[0076] like Figure 1 As shown, this embodiment provides a method for recognizing bridge-side fall behavior in a hazy environment, including:

[0077] We obtained real bridge fall behavior data under smog conditions, and after preprocessing, we obtained a bridge fall behavior dataset under smog conditions.

[0078] Based on the bridge fall behavior dataset and model loss function under hazy conditions, a pre-constructed RA-M fusion codec recognition network model is trained to obtain the trained RA-M fusion codec recognition network model.

[0079] The trained RA-M fusion encoder-decoder recognition network model was used to identify bridge fall behavior in hazy environments;

[0080] The RA-M fusion encoding and decoding recognition network model includes an RA-M encoder module and an RA-M decoder module. The RA-M encoder module is used for multi-scale feature extraction and fusion of images, and the RA-M decoder module is used for pedestrian masking and location recovery based on the inverse attention mechanism.

[0081] Example 2

[0082] This embodiment provides a method for recognizing bridge fall behavior in hazy environments, the steps of which are as follows:

[0083] Step 1: Obtain real bridge fall behavior data under smog conditions, and preprocess it to obtain a bridge fall behavior dataset under smog conditions. Preprocessing methods include data augmentation, data annotation, and dataset partitioning.

[0084] Step 1.1, Data Acquisition.

[0085] High-definition cameras were installed on the sides of the bridge to capture images of pedestrians falling in smoggy conditions, thereby collecting the necessary dataset of images.

[0086] Step 1.2, Data Augmentation.

[0087] Because the bridge fall is sudden and the image sampling scale is small, the dataset first needs to be augmented to improve the model's generalization ability and robustness.

[0088] On the one hand, in order to simulate the dynamic changes of haze under different concentrations and lighting conditions, physical degradation operations such as randomly adjusting atmospheric light values ​​and randomly perturbing scattering coefficients are applied to the images to generate an image sequence with continuously changing visibility.

[0089] On the other hand, spatial and pixel-level enhancement strategies, including random cropping, multi-scale scaling, small-amplitude rotation, and adding motion blur, were employed to increase the number and richness of samples.

[0090] Step 1.3, Data labeling.

[0091] To meet the data annotation requirements during model training and accurately capture the behavioral characteristics of falling individuals, the region of the fallen pedestrian in the image is binarized and segmented, generating a binarized segmentation mask image containing only the fallen pedestrian's mask. Furthermore, to accurately provide the fallen pedestrian's location information, a bounding box is annotated for the fallen person, generating a bounding box annotation file. The location information in the bounding box annotation file includes four data points: the x-coordinate and y-coordinate of the bounding box's center, and the length and width of the bounding box. This yields a dataset sample that matches the original image, the binarized segmentation mask image, and the bounding box annotation file.

[0092] Step 1.4: Dataset partitioning.

[0093] To ensure the fairness and effectiveness of model evaluation, all data was randomly divided into three parts: training set, validation set, and test set. The training set, which accounts for a large proportion (70%), is used for learning model parameters; the validation set (20%) is used to monitor performance and adjust hyperparameters during training; and the test set (10%) serves as the standard for finally evaluating the model's generalization ability, simulating its performance in real-world smog environments.

[0094] Step 2: Construct the RA-M fusion encoding and decoding recognition network model.

[0095] The RA-M fusion encoder-decoder recognition network model includes an RA-M encoder module and an RA-M decoder module. The RA-M encoder module is used for multi-scale feature extraction and fusion of images, and the RA-M decoder module is used for pedestrian masking and location recovery based on a reverse attention mechanism. Figure 2 The diagram shows the overall structure of the RA-M fusion encoding / decoding recognition network model.

[0096] Step 2.1: Use the RA-M encoder module to perform multi-scale feature extraction and fusion on the image.

[0097] Step 2.1.1: Use a multi-layer Res2Net feature extraction layer to perform global feature extraction on the image, generating multiple sets of feature extraction maps with different semantic depth scales. Then, perform downsampling to unify the scale and obtain multiple sets of original feature maps.

[0098] In the specific implementation of this embodiment, it is assumed that the size of the input original image is H×W×3 (where H represents the image height and W represents the image width, both in pixels), and the size of the input binary mask image is H×W×1. After passing through the Res2Net feature extraction layer, three feature maps f1, f2, and f3 are output, with sizes of H×W×C (C represents the number of channels in the feature map), H / 2×W / 2×2C, and H / 4×W / 4×4C, respectively. To facilitate subsequent serialization and merging, the feature maps are downsampled, finally obtaining three original feature maps f1', f2', and f3' of the same scale, with sizes of H / 4×W / 4×C, H / 4×W / 4×2C, and H / 4×W / 4×4C, respectively.

[0099] Step 2.1.2: Input each set of original feature maps into the alignment enhancement module. Perform scale unification, layer normalization, and serialization flattening on the original feature maps to obtain multiple sets of serialized features. This will enable preliminary attention guidance to suppress haze background interference and further highlight the effective areas related to pedestrians.

[0100] The alignment enhancement module performs scale unification, layer normalization, and sequential flattening on the original feature maps, which helps alleviate internal variable shifts and makes deep network training more stable. Figure 3 As shown, the specific implementation steps are as follows:

[0101] (1) Input the original feature map The convolutional layers are scaled and then the attention output feature map is generated through the CBAM module; where CBAM is a lightweight attention mechanism module that combines channel and spatial attention.

[0102] (2) After adding the attention output feature map to the original feature map, perform layer normalization to ensure more stable network characteristics and faster convergence speed during data training;

[0103] (3) The features output by the layer normalization process are serialized and flattened to obtain serialized features; where serialization is to flatten the two-dimensional tensor of length and width in the feature output into a one-dimensional tensor.

[0104] In the specific implementation of this embodiment, after inputting the original feature maps f1', f2', and f3' into the alignment enhancement module, serialized feature f1" with a size of (H / 4·W / 4)×C, serialized feature f2" with a size of (H / 4·W / 4)×2C, and serialized feature f3" with a size of (H / 4·W / 4)×4C can be generated.

[0105] The processing procedure of the alignment enhancement module is expressed by the following formula:

[0106] ;

[0107] Where F is the original feature map of the input alignment enhancement module, and S is the serialized feature map of the output alignment enhancement module. denoted as 1×1 convolution, CBAM is the convolutional block attention module operation, Norm function represents the layer normalization operation, and Flatten function represents flattening the two-dimensional feature map into a one-dimensional sequence.

[0108] Step 2.1.3: After concatenating multiple sets of serialized features, input them into the core feature fusion module. The Mamba module integrates multi-scale features into a unified feature representation under the same spatial scale. The sliding window structure of the SwinT module captures local structural features, so that the fine-grained contour features of the human body and action semantics can be preserved at the same time, realizing multi-scale feature fusion under different semantic depths.

[0109] In the specific implementation process, the serialized features f1”, f2”, and f3” are concatenated to generate a serialized feature f of size (H / 4·W / 4)×7C. The core feature fusion module's role is to further fuse the fused sequence f through the Mamba module, the SwinT module, the cross-attention fusion module, and several normalization processes. Finally, it reconstructs output feature maps of sizes H / 4×W / 4×C, H / 4×W / 4×2C, and H / 4×W / 4×4C through segmentation, and generates an attention guidance map of size H / 4×W / 4×4C based on the internal attention distribution. Figure 4 As shown, the specific steps are as follows:

[0110] (1) The concatenated serialized feature f is first nonlinearized by linear layer mapping and Gelu activation function; then the sequence is divided into positive and negative sub-branches by reverse processing:

[0111] For the positive branch, the sequence is processed by layer normalization and then fed into the Mamba module. The Mamba module dynamically and selectively memorizes or ignores information based on the input content, which effectively suppresses noise interference related to haze and concentrates the information of small objects falling pedestrians, resulting in a positive sequence X of size (H / 4·W / 4)×11C. + ;

[0112] For the reverse branch, the sequence is processed by layer normalization and then fed into the SwinT module. Inside the SwinT module, the sequence is first reshaped into a two-dimensional feature map, and then a sliding window attention mechanism is used to capture local structural features, thereby enhancing the feature and improving the model's ability to perceive details, resulting in a reverse sequence X of size (H / 4·W / 4)×11C. - .

[0113] (2) Transform the forward sequence X+ and reverse sequence X - The input is processed by the cross-attention fusion module. The specific implementation process is as follows:

[0114] The forward sequence X + After normalization, multiply by a parameter matrix W of size 11C×11C. Q The query matrix Q is obtained;

[0115] Reverse sequence X - After normalization, each is multiplied by a parameter matrix W of size 11C×11C. K W V This yields the key matrix K and the value matrix V;

[0116] The query matrix Q and the key matrix K are multiplied, and the fused attention matrix is ​​obtained by applying the Softmax activation function;

[0117] The extraction matrix U is obtained by multiplying the fused attention matrix with the value matrix V. The extraction matrix U mainly reflects the edge information and subtle action features of the feature sequence.

[0118] Extract the matrix U and the forward sequence X + The sequences are added together and then normalized to obtain the fused output sequence X. out The dimensions are (H / 4·W / 4)×11C.

[0119] The processing procedure of the cross-attention fusion module is expressed by the following formula:

[0120] ;

[0121] ;

[0122] ;

[0123] ;

[0124] Where d represents the input dimension and Norm represents the layer normalization operation.

[0125] Furthermore, the processing procedure of the cross-attention fusion module can be simplified as follows:

[0126] = ;

[0127] CAF stands for Cross Attention Fusion Module Operation.

[0128] (3) The sequence X obtained by the cross-attention fusion module outAfter layer normalization, segmentation, and reshaping, multiple feature maps and attention guidance maps are obtained.

[0129] In the specific implementation of this embodiment, feature maps h1, h2, and h3 with dimensions of H / 4×W / 4×C, H / 4×W / 4×2C, and H / 4×W / 4×4C, respectively, and an attention guidance map p3 with a size of H / 4×W / 4×4C are obtained. The processing procedure of the core feature fusion module is then expressed by the following formula:

[0130] ;

[0131] ;

[0132] ;

[0133] ;

[0134] ;

[0135] Where X represents the input to the core feature fusion module, and Linear represents the linear mapping layer. is the Gaussian error linear unit activation function, and S0 represents the intermediate feature sequence after the input sequence X is processed by linear mapping and Gelu activation function; , Represent Module, The input sequence of the module, Representative output The forward sequence of modules, Representative output The reverse sequence of the module; Indicates the reverse operation; Mamba indicates Mamba module operation, and SwinT indicates SwinT module operation. and These represent the segmentation operation and the reshaping operation, respectively.

[0136] Step 2.2: Pedestrian masking and location recovery based on the reverse attention mechanism using the RA-M decoder module.

[0137] The RA-M decoder module is mainly responsible for outputting the predicted mask image and predicted bounding box of the falling person. The predicted mask image is mainly used to train the model, while the predicted bounding box includes location information and behavior category. The behavior categories include three states: out of bounds, falling, and falling into water.

[0138] The RA-M decoder module, based on the features of the RA-M encoder, enhances the detail recovery of small human targets and the action contour information of fallen persons through a reverse-guided attention mechanism. The RA-M decoder first uses the attention guidance map from the RA-M encoder to reverse-guide multi-scale features, enabling the model to find the true boundaries of the human body region and suppress irrelevant background even under severe fog and haze obscuration. Then, through multiple segmentation-guided fusion modules, it achieves a bottom-up, progressive fusion and upsampling structure, effectively combining deep semantics with shallow texture information to reconstruct a more accurate feature map. Finally, the processed features are input into the anchorless bounding box detector to achieve bounding box regression and behavior category prediction for fallen pedestrians. The specific steps are as follows:

[0139] Step 2.2.1: The image details are restored step by step by the multi-layer segmentation guided fusion module based on reverse attention guidance, generating multiple intermediate mask images.

[0140] This step takes multiple feature maps and attention guidance maps output by the RA-M encoder module as input, and decodes layer by layer in the RA-M decoder module in the direction from deep to shallow through multiple segmentation guidance fusion modules.

[0141] In this embodiment, the feature maps h1, h2, h3 and the attention guidance map p3 output by the RA-M encoder module are used as input, such as Figure 5 As shown, the deepest segmentation-guided fusion module is taken as an example.

[0142] (1) Input the feature map h3 with size H / 4×W / 4×4C and the attention guidance map p3 with size H / 4×W / 4×4C into the deepest segmentation guidance fusion module, and process them into attention guidance branch and feature map guidance branch.

[0143] (2) In the attention-guided branch, feature map h3 and attention-guided map p3 are input into the reverse attention fusion module to generate a new feature map through convolution.

[0144] Within the reverse attention fusion module, the attention guidance map p3 is combined and compressed using a combination of convolution + Gelu activation function + convolution to generate an attention guidance map of size H / 4×W / 4×4. This generated attention guidance map is then divided into four sub-attention guidance maps of size H / 4×W / 4×1 according to channels, representing different attention guidance modes. Feature map h3 is divided into four sub-feature maps of size H / 4×W / 4×C according to channels. The single-channel attention guidance map is inverted and inserted between every two adjacent sub-feature maps. Channel concatenation generates a fused feature map of size H / 4×W / 4×(4C+4). A new feature map of size H / 4×W / 4×4C is generated through a convolutional layer. This new feature map is then used as the new input to the feature map guidance branch to assist in attention weight optimization.

[0145] (3) In the feature map guidance branch, the feature map h3 and the attention guidance map p3 are input into the core feature fusion module, and they are multiplied element by element and serialized to obtain a new attention guidance map.

[0146] (4) The new feature map obtained in step (2) and the new attention guidance map obtained in step (3) are used as new inputs to the attention guidance branch and the feature map guidance branch to continue a new round of alternating optimization. After each round of alternating optimization, dense connections are added to each branch to prevent the loss of semantic information and gradient vanishing.

[0147] (5) The attention guidance map and feature map after two rounds of alternating optimization are fused by the reverse attention fusion module. After one convolution, the output feature map r3 of the deepest segmentation guidance fusion module is obtained, with a size of H / 4×W / 4×4C.

[0148] (6) After adding the output feature map r3 of the deepest segmentation guidance fusion module to the attention guidance map p3, it is convolved and matched with the dimension of the previous layer feature map h2 to obtain a new attention guidance map p2 with size H / 4×W / 4×2C; then the new attention guidance map p2 and the previous layer feature map h2 are input into the previous layer segmentation guidance fusion module to guide the shallower texture enhancement.

[0149] (7) Add the output feature map of each layer segmentation guidance fusion module to the attention guidance map input to the corresponding layer segmentation guidance fusion module, and generate multiple intermediate mask images through convolution dimensionality reduction. In this embodiment, three intermediate mask images are finally generated, namely the intermediate mask images C1, C2, and C3 corresponding to the first, second, and third layer segmentation guidance fusion modules.

[0150] The processing procedure of the segmentation-guided fusion module is expressed by the following formula:

[0151] ;

[0152] ;

[0153] ;

[0154] ;

[0155] ;

[0156] in, , These represent the feature map and attention guidance map input to the i-th layer segmentation guidance fusion module, respectively. The feature map representing the output of the i-th layer segmentation-guided fusion module; These represent the feature maps obtained from the first and second rounds of alternating optimization in the attention-guided branch. These represent the attention weights obtained from the first and second rounds of alternating optimization of the feature map-guided branch, respectively. This indicates element-wise multiplication. This represents the convolution operation. This indicates a channel splicing operation. For linear correction unit activation function, The core feature fusion module operation is represented by ; RAI represents the reverse attention fusion module operation, expressed by the following formula:

[0157] ;

[0158] ;

[0159] ;

[0160] in, express List of sub-feature maps segmented by channel express After dimensionality reduction using a non-linear combination of convolution + Gelu activation function + convolution, the attention guidance graph list is obtained by segmenting by channel and then inverting. Let Gaussian error be the activation function of the linear unit. This indicates a splitting operation. This indicates the reverse operation.

[0161] Step 2.2.2: Apply the sigmoid activation function and upsampling to the intermediate mask image C1 corresponding to the first layer segmentation guidance fusion module to generate the mask image C_out.

[0162] Step 2.2.3: The mask image C_out is processed by a self-attention module to generate weight information. The weight information is then multiplied with the mask image to give higher weight to the area where the fallen pedestrian is located and lower weight to the area with haze and noise, thereby further increasing the accuracy of the predicted bounding box generation. Finally, the predicted bounding box location information and category information are generated by the anchorless detection head.

[0163] Step 3: Construct the model loss function.

[0164] To address the characteristic of RA-M fusion encoder-decoder recognition network models that simultaneously output binary segmentation masks and target detection boxes, a multi-task loss function fusing segmentation and detection losses was designed. This loss function aims to guide the network to not only accurately segment human contours under haze interference but also accurately regress target locations and determine their behavioral states. The total loss function consists of three parts: segmentation loss, detection loss, and attention supervision loss. The formula for the total loss function is as follows:

[0165] ;

[0166] in, For the total loss, To divide the loss, To detect the loss, For attention monitoring loss, , , These represent the weight coefficients of the segmentation loss, detection loss, and attention supervision loss, respectively.

[0167] The segmentation loss measures the difference between the predicted mask and the true mask. It is obtained by calculating the Dice coefficients based on the predicted mask and the true mask, as shown in the following calculation process:

[0168] ;

[0169] in, and These represent the pixel value at the j-th position of the predicted mask and the pixel value at the j-th position of the real mask, respectively, where j represents the index of all pixels in the image. This represents the smoothing term to prevent the denominator from being zero; the predicted mask image is the mask image C_out from step 2.

[0170] The detection loss includes the bounding box regression loss and the classification loss, mainly derived from the cross-entropy between the predicted class information and the ground truth information, and the Interchange of Value (IOU) between the predicted box position and the ground truth position. The calculation process is shown below:

[0171] ;

[0172] ;

[0173] ;

[0174] in, Represents classification loss, Represents the target bounding box regression loss. K represents the balance coefficient between classification loss and bounding box regression loss. K=3 represents the number of categories: "out of bounds," "falling," and "falling into water." This represents the true probability distribution of the k-th category. This represents the predicted probability of the k-th category. Represents the number of target boxes. and These represent the t-th predicted bounding box and the t-th ground truth bounding box, respectively. This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. This represents the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box. This represents the diagonal length of the smallest bounding box.

[0175] The attention-supervised loss, used to constrain the consistency between the attention guidance map and the ground truth mask, is obtained by a weighted average of the Dice coefficients between the predicted and ground truth masks output from each layer of the decoder. The calculation process is as follows:

[0176] ;

[0177] Where Dice(*) represents Function, i.e., calculation Loss coefficient, where G represents the attention guidance map output by the RA-M encoder. Represents the real mask image, i represents the layer sequence of the segmentation-guided fusion module, and m represents the number of layers in the segmentation-guided fusion module. Indicates an upsampling operation. This represents the Sigmoid activation function. This represents the intermediate mask image generated by the i-th layer segmentation-guided fusion module.

[0178] Step 4: Based on the bridge fall behavior dataset and model loss function under hazy conditions, train the pre-built RA-M fusion codec recognition network model to obtain the trained RA-M fusion codec recognition network model.

[0179] A phased training strategy was employed. Initially, the focus was on extracting basic features, thus assigning greater weight to the segmentation loss. Later, the accuracy of location and category information was emphasized, resulting in a higher weight for the detection loss. During training, a validation set was used to continuously monitor model performance and prevent overfitting. Hyperparameters such as the learning rate were adjusted to ensure stable convergence. Finally, iterative optimization continuously reduced the total loss function, resulting in a network model capable of accurately identifying bridge-side falls under smog conditions.

[0180] Step 5: Use the trained RA-M fusion encoder-decoder recognition network model to identify bridge-side fall behavior in hazy environments.

[0181] Verified, the RA-M fusion codec recognition network model can quickly and accurately identify the location and behavior of a fallen pedestrian even in foggy or hazy conditions with visibility of at least 150 meters, enabling real-time monitoring and early warning, and providing rescue personnel with immediate intervention and rescue information. Furthermore, the entire process can be linked with a high-efficiency alarm system; when a fall is detected, the system immediately triggers an alarm mechanism and locates the event, significantly improving the response speed and accuracy of bridge public safety monitoring.

[0182] Example 3

[0183] Based on the same inventive concept as Embodiment 1 or Embodiment 2, this embodiment provides a bridge fall behavior recognition system in a hazy environment, comprising:

[0184] The dataset construction module is used to obtain real bridge fall behavior data under hazy conditions. After preprocessing, a bridge fall behavior dataset under hazy conditions is obtained.

[0185] The model training module is used to train a pre-built RA-M fusion codec recognition network model based on a dataset of bridge fall behavior under haze conditions and a model loss function, so as to obtain the trained RA-M fusion codec recognition network model.

[0186] The behavior recognition module is used to identify bridge-side fall behavior in hazy environments using a trained RA-M fusion encoding and decoding recognition network model.

[0187] The RA-M fusion encoding and decoding recognition network model includes an RA-M encoder module and an RA-M decoder module. The RA-M encoder module is used for multi-scale feature extraction and fusion of images, and the RA-M decoder module is used for pedestrian masking and location recovery based on the inverse attention mechanism.

[0188] The specific functions of each module described above are explained in the relevant content of the method in Embodiment 2, and will not be repeated here.

[0189] All parts not covered in this invention are the same as or implemented using existing technologies.

[0190] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0191] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0192] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0193] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0194] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for recognizing bridge-side fall behavior in a hazy environment, characterized in that, include: We obtained real bridge fall behavior data under smog conditions, and after preprocessing, we obtained a bridge fall behavior dataset under smog conditions. Based on the bridge fall behavior dataset and model loss function under hazy conditions, a pre-constructed RA-M fusion codec recognition network model is trained to obtain the trained RA-M fusion codec recognition network model. The trained RA-M fusion encoder-decoder recognition network model was used to identify bridge fall behavior in hazy environments; The RA-M fusion encoding and decoding recognition network model includes an RA-M encoder module and an RA-M decoder module. The RA-M encoder module is used for multi-scale feature extraction and fusion of images, and the RA-M decoder module is used for pedestrian masking and location recovery based on the reverse attention mechanism. The pedestrian masking and location recovery based on the inverse attention mechanism using the RA-M decoder module includes: The multi-layer segmentation-guided fusion module uses reverse attention to guide the recovery of image details step by step, generating multiple intermediate mask images. The intermediate mask image corresponding to the first-layer segmentation-guided fusion module is activated by the sigmoid activation function and upsampled to generate a mask image; The mask image is processed by a self-attention module to generate weight information. The weight information is then multiplied by the mask image and passed through an anchorless detection head to generate the corresponding bounding box position information and category information. The method of restoring image details through a multi-layer segmentation-guided fusion module based on reverse attention guidance includes: feature map h m And attention-guided diagram p m The deepest segmentation-guided fusion module is input and processed in two ways: an attention-guided branch and a feature map-guided branch. In the attention-guided branch, the feature map h m And attention-guided diagram p m Input the inverse attention fusion module to generate a new feature map through convolution; In the feature map-guided branch, the feature map h m And attention-guided diagram p m Input the core feature fusion module, multiply it element-wise and serialize it to obtain a new attention guidance map; The new feature map and the new attention guidance map are used as new inputs to the attention guidance branch and the feature map guidance branch to continue a new round of alternating optimization, and dense connections are added to each branch after each round of alternating optimization. After two rounds of alternating optimization, the attention-guided map and feature map are fused through a reverse attention fusion module. After one convolutional layer, the output feature map r of the deepest segmentation-guided fusion module is obtained. m ; The deepest segmentation guides the output feature map r of the fusion module. m With attention guidance diagram p m After addition, it is convolved and then combined with the feature map h from the previous layer. m-1 By matching dimensions, a new attention guidance graph p is obtained. m-1 Guide new attention to graph p m-1 With the previous layer feature map h m-1 Input the previous layer segmentation and fusion module to guide the enhancement of shallower textures; The output feature maps of each layer segmentation guidance fusion module are added to the attention guidance maps input to the corresponding layer segmentation guidance fusion module, and multiple intermediate mask images are generated by convolutional dimensionality reduction.

2. The method for identifying bridge fall behavior in hazy environments according to claim 1, characterized in that, Methods for preprocessing real bridge fall behavior data under smog conditions include data augmentation, data annotation, and dataset partitioning; The data augmentation includes: applying physical degradation operations to the image by randomly adjusting atmospheric light values ​​and randomly perturbing the scattering coefficient to generate an image sequence with continuously changing visibility; and increasing the number of samples by employing spatial and pixel-level enhancement strategies, including random cropping, multi-scale scaling, rotation, and adding motion blur. The data annotation includes: performing binarization segmentation on the region of the fallen pedestrian in the image to generate a binarized segmentation mask image containing only the mask of the fallen pedestrian; annotating the target bounding boxes of the fallen person to generate a target bounding box annotation file; and obtaining a dataset sample that matches the original image, the binarized segmentation mask image, and the target bounding box annotation file. The dataset partitioning includes dividing the dataset samples into a training set, a validation set, and a test set.

3. The method for identifying bridge fall behavior in hazy environments according to claim 1, characterized in that, The RA-M encoder module is used to perform multi-scale feature extraction and fusion on the image, including: A multi-layer Res2Net feature extraction layer is used to perform global feature extraction on the image, generating multiple sets of feature extraction maps with different semantic depth scales. Then, downsampling is performed to obtain multiple sets of original feature maps. Each set of original feature maps is input into the alignment enhancement module, which performs scale unification, layer normalization, and serialization flattening on the original feature maps to obtain multiple sets of serialized features. Multiple sets of sequential features are concatenated and input into the core feature fusion module. The Mamba module integrates multi-scale features into a unified feature representation at the same spatial scale. The sliding window structure of the SwinT module captures local structural features, enabling the fine-grained contour features of the human body and action semantics to be preserved simultaneously, thus achieving multi-scale feature fusion at different semantic depths.

4. The method for identifying bridge fall behavior in hazy environments according to claim 3, characterized in that, The alignment enhancement module is used to perform scale unification, layer normalization, and serialization flattening on the original feature map, specifically including: Input the original feature map The convolutional layers are scaled, and then the attention output feature map is generated through the CBAM module. After adding the attention output feature map to the original feature map, layer normalization is performed. The features output by the layer normalization process are serialized and flattened to obtain serialized features.

5. The method for identifying bridge fall behavior in hazy environments according to claim 3, characterized in that, The core feature fusion module is used to achieve multi-scale feature fusion at different semantic depths, specifically including: The concatenated serialized features are first non-linearized using linear layer mapping and the Gelu activation function; then the sequence is divided into positive and negative sub-branches through reverse processing. For the forward branch, the sequence is processed by layer normalization and then fed into the Mamba module. The Mamba module dynamically and selectively memorizes or ignores information based on the input content to obtain the forward sequence. For the reverse branch, the sequence is fed into the SwinT module after being processed by layer normalization. Inside the SwinT module, the sequence is first reshaped into a two-dimensional feature map, and then the local structural features are captured through the sliding window attention mechanism to obtain the reverse sequence. The forward and reverse sequences are input into the cross-attention fusion module for processing. The resulting sequences are then normalized, segmented, and reshaped to obtain multiple feature maps and attention guidance maps.

6. The method for identifying bridge fall behavior in hazy environments according to claim 5, characterized in that, The step of inputting the forward and reverse sequences into the cross-attention fusion module for processing includes: The forward sequence X + After normalization, multiply by the parameter matrix of the appropriate size to obtain the query matrix Q; Reverse sequence X - After normalization, multiply by the parameter matrix of the corresponding size to obtain the key matrix K and the value matrix V; The query matrix Q and the key matrix K are multiplied, and the fused attention matrix is ​​obtained by applying the Softmax activation function; The extraction matrix U is obtained by multiplying the fusion attention matrix by the value matrix V; Extract the matrix U and the forward sequence X + The sequences are added together and then normalized to obtain the fused output sequence X. out ; The processing procedure can be expressed by the following formula: ； ； ； ； Among them, W Q W K W V d represents the parameter matrix, d represents the input dimension, and Norm represents the layer normalization operation.

7. The method for identifying bridge fall behavior in hazy environments according to claim 1, characterized in that, The formula for the model loss function is as follows: ； ； ； ； in, For the total loss, To divide the loss, To detect the loss, For attention monitoring loss, , , These represent the weight coefficients of the segmentation loss, detection loss, and attention supervision loss, respectively. and These represent the pixel value at the j-th position of the predicted mask and the pixel value at the j-th position of the real mask, respectively, where j represents the index of all pixels in the image. Represents the smoothing term; Represents classification loss, Represents the target bounding box regression loss. Dice(*) represents the balance coefficient between classification loss and bounding box regression loss; Function, i.e., calculation The loss coefficient, G, represents the attention guidance map output by the RA-M encoder. Represents the real mask image, i represents the layer sequence of the segmentation-guided fusion module, and m represents the number of layers in the segmentation-guided fusion module. This represents the Sigmoid activation function. Indicates an upsampling operation. This represents the intermediate mask image generated by the i-th layer segmentation-guided fusion module.

8. A bridge fall behavior recognition system in a smoggy environment, characterized in that, To implement the method according to any one of claims 1 to 7, comprising: The dataset construction module is used to obtain real bridge fall behavior data under smog conditions. After preprocessing, a bridge fall behavior dataset under smog conditions is obtained. The model training module is used to train a pre-built RA-M fusion codec recognition network model based on a dataset of bridge fall behavior under haze conditions and a model loss function, so as to obtain the trained RA-M fusion codec recognition network model. The behavior recognition module is used to identify bridge-side fall behavior in hazy environments using a trained RA-M fusion encoding and decoding recognition network model. The RA-M fusion encoding and decoding recognition network model includes an RA-M encoder module and an RA-M decoder module. The RA-M encoder module is used for multi-scale feature extraction and fusion of images, and the RA-M decoder module is used for pedestrian masking and location recovery based on the inverse attention mechanism.

Images