A method and system for bridge side fall behavior perception suitable for low-quality visual conditions

Abstract

The application discloses a bridge-side falling behavior perception method and system suitable for low-quality visual conditions, and belongs to the technical field of behavior perception. The method comprises the following steps: determining whether a bridge-side scene image exists in weak light and / or blur according to a TBV index of the calculated bridge-side scene image; inputting the bridge-side scene image existing in weak light into a D2A-DCE model, inputting the bridge-side scene image existing in blur into an improved NAFNet model, inputting the bridge-side scene image existing in weak light and blur into the D2A-DCE model and the improved NAFNet model in sequence, and outputting a bridge-side scene enhanced image; inputting the bridge-side scene enhanced image into an improved CMFF model, and outputting a fusion feature map; and inputting the fusion feature map into a pre-constructed Bridge-SIDE Net model, and outputting a bridge-side falling behavior perception result. The application can solve the problems of poor bridge-side behavior recognition accuracy, false positives and false negatives in an image quality degradation environment.

Description

Technical Field

[0001] This invention relates to the field of behavior perception technology, and in particular to a method and system for perceiving behavior during bridge falls under low-quality visual conditions. Background Technology

[0002] In real-world bridge monitoring scenarios, behaviors such as jumping off bridges and falling into water are characterized by their suddenness, small size, complex actions, and variable environmental conditions. These events typically occur within a short period, involve a limited area, and involve rapid and subtle movements, posing a significant challenge to real-time identification by video surveillance systems. Furthermore, the surrounding environment of bridges is often affected by factors such as fog, low light at night, rain, snow, and backlighting, which blur the edges of visible targets and severely damage texture information. Traditional visible light-based video surveillance methods are prone to false alarms and missed alarms under these low-quality visual conditions, seriously affecting public safety response efficiency and failing to meet actual safety protection needs.

[0003] While existing methods can identify bridge-crossing and falling behaviors, they lack quantification processing for image quality fluctuations and adaptive enhancement mechanisms, resulting in a significant drop in recognition accuracy under low light, fog, and blurry conditions. Other methods have improved recognition accuracy in certain scenarios, but their algorithms are complex, lack real-time performance, and lack cross-modal information fusion and small target detection mechanisms, making it difficult to address issues such as small targets, rapid movements, and complex environments in bridge-side falling behaviors. Existing methods generally suffer from insufficient image enhancement, severe loss of brightness, contrast, and edge details under low-quality visual conditions, limited small target detection capabilities, underutilization of multimodal information, insufficient global and local feature modeling, difficulty in accurately locating the action area, and insufficient real-time performance and robustness, making it difficult to achieve efficient and stable operation in edge computing or field deployment environments. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a bridge-side fall behavior perception method and system suitable for low-quality visual conditions. This method can solve the problems of poor accuracy, false alarms and missed alarms in bridge-side behavior recognition under degraded image quality conditions.

[0005] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0006] On the one hand, this invention provides a method for perceiving bridge-side fall behavior under low-quality visual conditions, including:

[0007] Acquire images of the bridge side scene;

[0008] Calculate the TBV index of the bridge-side scene image, and determine whether the bridge-side scene image has weak light and / or blur based on the TBV index;

[0009] The model inputs a pre-built D2A-DCE model with a low-light bridge-side scene image to output an enhanced bridge-side scene image; it inputs a pre-built improved NAFNet model with a blurred bridge-side scene image to output an enhanced bridge-side scene image; it then inputs both a low-light and blurred bridge-side scene image into the pre-built D2A-DCE model and the pre-built improved NAFNet model sequentially to output an enhanced bridge-side scene image. The D2A-DCE model is constructed by replacing the single-branch structure of the Zero-DCE++ framework with a dual-branch structure where each branch embeds a CBAM module, and by adding an amplitude controller to the parameter prediction unit of the Zero-DCE++ framework. The improved NAFNet model is constructed by adding a non-local attention module between the encoder and decoder of the NAFNet model.

[0010] The enhanced image of the bridge-side scene is input into a pre-constructed improved CMFF model, and a fused feature map is output. The improved CMFF model is constructed by adding a bidirectional context transfer path in the cross-modal interaction stage of the CMFF model and a modal residual signal compensation mechanism in the fusion embedding stage.

[0011] The fused feature map is input into the pre-built Bridge-SIDE Net model, and the bridge-side fall behavior perception result is output. The Bridge-SIDE Net model is constructed by embedding a SwinConv module in the last layer of each residual block of the backbone network in the RT-DETR architecture and adding an FPN module after the backbone network.

[0012] Optionally, the TBV index of the bridge-side scene image is expressed as:

[0013] ;

[0014] ;

[0015] ;

[0016] in, The TBV (Total Volume Value) metric represents the image of the bridge side scene. These represent the sub-indicators of ambiguity and visual information loss, respectively. All represent weighting coefficients; These represent the length and width of the image of the scene on the bridge side, respectively. These represent the lengths of the bridge-side scene images, respectively. ,Width At Gaussian kernel scale Gaussian function under Gaussian kernel scale The Gaussian function under the following conditions; This represents an image of the scene on the side of the bridge. This represents the convolution operation; Image representing the scene on the side of the bridge Entropy; This represents the ideal state where the image information is most abundant; This represents the average image contrast. This represents the ideal mean maximum contrast. These represent the contributions of adjusting image entropy to visual information loss and adjusting contrast to visual information loss, respectively.

[0017] Optionally, the step of inputting the image of the bridge-side scene with low light into a pre-built D2A-DCE model and outputting an enhanced image of the bridge-side scene includes:

[0018] The bridge-side scene image is inverted to obtain an inverted bridge-side scene image;

[0019] The bridge-side scene image and the bridge-side scene inverted image are respectively input into the dual-branch structure of the D2A-DCE model. In each branch, the multi-scale feature map of the image is extracted through the first continuous convolutional layer. The channel attention module and spatial attention module in the CBAM module are used to reweight the multi-scale feature map with channel attention and spatial attention to obtain the enhanced feature map. The enhanced feature map is then dimensionality-reduced and mapped through the second continuous convolutional layer to obtain the mapped feature map.

[0020] The mapped feature map is input into the parameter prediction unit of the D2A-DCE model, and the iterator and amplitude controller work together to process the mapped feature map. An iterative enhancement strategy is used to obtain an enhanced image of the bridge-side scene; the enhancement amplitude and brightness offset are corrected in real time by an amplitude controller.

[0021] Optionally, the step of performing the mapping of the feature map together through an iterator and an amplitude controller is described. A multi-level iterative enhancement strategy is used to obtain enhanced images of the bridge-side scene, including:

[0022] The mapping feature map is executed jointly by iterator and amplitude controller. The iterative enhancement strategy involves an iterator generating enhancement parameters and an amplitude controller generating adjustment coefficients. Each iteration satisfies the following control logic:

[0023] ;

[0024] Based on the enhancement parameters and adjustment coefficients, the enhancement amplitude coefficient is generated. Level Iteration Enhancement Amplitude Coefficient ;

[0025] An enhanced image of the bridge-side scene is generated based on the enhancement amplitude coefficient. Level-by-level bridge-side scene enhancement images ;

[0026] in, Indicates the first Level iteration; This represents a nonlinear curve enhancement operator; This represents an image of the scene on the side of the bridge. Indicated in the red channel of the mapped feature map Green Channel Blue Channel The following is the prediction of the iterator. Level iteration enhancement parameters; All represent control coefficients; They represent the first Level iteration enhancement parameters and adjustment coefficients; Indicates the first Enhanced images of the bridge-side scene through iterative processing.

[0027] Optionally, the step of inputting the blurred bridge-side scene image into a pre-built improved NAFNet model and outputting an enhanced bridge-side scene image includes:

[0028] In the encoder, hierarchical information extraction is performed on the bridge-side scene image to obtain features at each level;

[0029] In the non-local attention module, similarity calculation and weighted fusion are performed on the features at each level to obtain an enhanced feature map;

[0030] In the decoder, the enhanced feature map is decoded and reconstructed, and the decoded and reconstructed features are fused with the features at each level to obtain an enhanced image of the bridge side scene.

[0031] Optionally, the enhanced feature map is represented as:

[0032] ;

[0033] ;

[0034] ;

[0035] in, This indicates that the enhanced feature map is located at... eigenvectors; Indicates the location of features at each level. Feature vectors, at position eigenvectors; These represent the number of feature channels, feature height, and feature width, respectively. Indicates the location of features at each level. eigenvectors In position eigenvectors The similarity function; Indicates the location of features at each level. eigenvectors The characteristic transformation function; Represents the learnable weight matrix; This indicates the matrix transpose.

[0036] Optionally, the step of inputting the enhanced image of the bridge-side scene into a pre-constructed improved CMFF model and outputting a fused feature map includes:

[0037] The enhanced image of the bridge-side scene is input into the visible light mode branch and the infrared mode branch respectively, and the corresponding backbone features are extracted to obtain the visible light mode intermediate representation and the infrared mode intermediate representation.

[0038] The intermediate representation of visible light modes is divided into visible light residual branch features and visible light interactive branch features, and the intermediate representation of infrared modes is divided into infrared residual branch features and infrared interactive branch features;

[0039] A bidirectional context transfer path is constructed between the visible light modal intermediate representation and the infrared modal intermediate representation, enabling the visible light modal intermediate representation and the infrared modal intermediate representation to mutually perceive semantic information in a global scope, thereby obtaining visible light attention and infrared attention;

[0040] By connecting the intermediate representation of the visible light mode with the visible light attention residual, we obtain the visible light enhanced representation; by connecting the intermediate representation of the infrared mode with the infrared attention residual, we obtain the infrared enhanced representation.

[0041] After feature fusion of infrared enhancement representation and visible light interactive branch features, feature splicing is performed with visible light residual branch features to obtain visible light compressed features. Channel projection, residual connection and normalization are then performed on the visible light compressed features in sequence to obtain visible light fused features.

[0042] After fusing the visible light enhanced representation with the infrared interactive branch features, it is then spliced ​​with the infrared residual branch features to obtain the infrared compressed features. The infrared compressed features are then subjected to channel projection, residual connection and normalization in sequence to obtain the infrared fused features.

[0043] Visible light fusion features and infrared fusion features are spliced ​​together to obtain spliced ​​features. The spliced ​​features are then subjected to feature fusion, local context enhancement, residual connection and normalization in sequence to obtain hybrid features.

[0044] Calculate the intermediate representation of visible light modes Intermediate representation of infrared modes residuals The residual term Residual enhancement weights are obtained through the SE attention module. Weight the residuals Incorporating hybrid features , to obtain the fused feature map .

[0045] Optionally, the visible light attention is represented as:

[0046] ;

[0047] The infrared attention is represented as:

[0048] ;

[0049] in, These represent visible light attention and infrared attention, respectively. These represent the visible light query vector, key vector, and value vector, respectively. These represent the infrared query vector, key vector, and value vector, respectively. The scaling dimension representing attention; This indicates the matrix transpose.

[0050] Optionally, the fused feature map is input into a pre-built Bridge-SIDE Net model to output bridge-side fall behavior perception results, including:

[0051] The fused feature map is input into the backbone network. After convolution processing, the fused feature map is sequentially processed through the first residual block, the second residual block, the third residual block, and the fourth residual block for feature extraction, resulting in the final stage feature map, the second stage feature map output by the second residual block, and the third stage feature map output by the third residual block. In each residual block, the feature map is divided into non-overlapping local windows by the SwinConv module. Multi-head self-attention is calculated in each local window to capture local and contextual relationships, while extracting local texture and boundary information.

[0052] The feature map of the last stage is converted into a one-dimensional sequence through flattening and linear projection operations. Spatial location encoding is used to retain the spatial domain location information of the one-dimensional sequence to obtain the encoded features.

[0053] After the encoded features are globally modeled using a multi-head self-attention mechanism, they are residually connected and normalized with a one-dimensional sequence to obtain processed features. The processed features are then refined using a feedforward network and residually connected and normalized with the processed features to obtain normalized features.

[0054] The normalized features are input into the FPN module along with the second-stage and third-stage feature maps to perform cross-scale information interaction and obtain interactive features.

[0055] The interaction features are used to focus on the target region and predict the target category to obtain the bridge side fall behavior perception results.

[0056] On the other hand, the present invention provides a bridge fall behavior perception system suitable for low-quality visual conditions, comprising:

[0057] The image acquisition module is used to: acquire images of the bridge side scene;

[0058] The quality judgment module is used to: calculate the TBV index of the bridge side scene image, and determine whether the bridge side scene image has weak light and / or blur based on the TBV index of the bridge side scene image;

[0059] An image enhancement module is used to: input a bridge-side scene image with low light into a pre-built D2A-DCE model and output an enhanced bridge-side scene image; input a bridge-side scene image with blurriness into a pre-built improved NAFNet model and output an enhanced bridge-side scene image; input a bridge-side scene image with both low light and blurriness sequentially into the pre-built D2A-DCE model and the pre-built improved NAFNet model and output an enhanced bridge-side scene image; the D2A-DCE model is constructed by replacing the single-branch structure of the Zero-DCE++ framework with a dual-branch structure in which each branch is embedded in a CBAM module, and adding an amplitude controller to the parameter prediction unit of the Zero-DCE++ framework; the improved NAFNet model is constructed by adding a non-local attention module between the encoder and decoder of the NAFNet model;

[0060] The image fusion module is used to: input the enhanced image of the bridge-side scene into a pre-constructed improved CMFF model and output a fused feature map; the improved CMFF model is constructed by adding a bidirectional context transfer path in the cross-modal interaction stage of the CMFF model and adding a modal residual signal compensation mechanism in the fusion embedding stage;

[0061] The behavior perception module is used to: input the fused feature map into the pre-built Bridge-SIDE Net model and output the bridge-side fall behavior perception result; the Bridge-SIDE Net model is constructed by embedding a SwinConv module in the last layer of each residual block of the backbone network in the RT-DETR architecture and adding an FPN module after the backbone network.

[0062] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0063] This invention performs quality quantification analysis on acquired bridge-side images. Through adaptive brightness, color adjustment, and detail restoration strategies, it optimizes low-quality images, providing high-quality image data for subsequent analysis. It achieves deep feature fusion from different visual modalities, enhancing the system's semantic understanding and structural perception capabilities of targets in complex environments. It accurately detects small, featureless targets in bridge-side scenes, enabling real-time perception of bridge-side fall behavior. It achieves high-precision, real-time, cross-modal fusion bridge-side fall behavior perception under low-quality visual conditions, adaptively improving image quality, accurately detecting small targets, combining global and local feature modeling, and ensuring stability and real-time response capabilities in complex environments. This significantly improves the accuracy and reliability of bridge-side behavior recognition, effectively reducing the probability of false alarms and false negatives. Attached Figure Description

[0064] Figure 1 The diagram shown is a flowchart of one embodiment of the bridge fall behavior perception method applicable to low-quality visual conditions of the present invention.

[0065] Figure 2 The diagram shown is a structural schematic of the D2A-DCE model of the present invention in one embodiment;

[0066] Figure 3 The diagram shown is a structural schematic of the improved NAFNet model of the present invention in one embodiment;

[0067] Figure 4 The diagram shown is a structural schematic of the improved CMFF model of the present invention in one embodiment;

[0068] Figure 5 The diagram shown is a structural schematic of the Bridge-SIDE Net model of the present invention in one embodiment. Detailed Implementation

[0069] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0070] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0071] Example 1

[0072] like Figure 1 As shown in the figure, this embodiment introduces a method for perceiving bridge fall behavior under low-quality visual conditions, including the following steps:

[0073] Step 1: Acquire images of the bridge side scene and perform image quality assessment, specifically:

[0074] To acquire images of the bridge side scene, the acquisition resolution, frame rate, and modal type can be adjusted according to actual needs to adapt to different environmental conditions. The resolution of the acquired images is 1920×1080, and the sampling frequency is 25 to 30 frames per second to ensure the continuity of the video stream and the capture of small target motion.

[0075] Image quality assessment is the first step in addressing poor visual conditions. Its core function is to quantify image quality by calculating the Total Bounded Variation (TBV) index, thereby determining whether the image enhancement process needs to be triggered. In the TBV index calculation process, the analysis is mainly carried out from two dimensions: the degree of image blurring and the loss of visual information. Based on the TBV index, it is determined whether there is weak light and / or blurring in the bridge side scene image.

[0076] For calculating the image blur level sub-index, the Difference of Gaussian (DoG) algorithm is used. This algorithm convolves the image with Gaussian filters of different scales to obtain edge information at multiple scales. The greater the difference in edge information across different scales, the higher the degree of image blur. For example, in images of bridge scenes taken under smoggy conditions, the scattering of light by suspended particles in the air blurs the image's edge details. The DoG algorithm can effectively capture this blurring effect.

[0077] The visual information loss sub-index is based on two key parameters: image entropy and contrast. Image entropy measures the richness of information in an image; the lower the entropy value, the less effective information the image contains and the worse the visual quality. The contrast parameter reflects the image's clarity by calculating the brightness difference between different regions in the image; the lower the contrast, the more blurred the image appears. The image entropy and contrast are weighted and summed according to certain weights (based on experimental results from actual bridge monitoring images, the image entropy weight is 0.6 and the contrast weight is 0.4) to obtain the sub-index reflecting visual information loss.

[0078] After normalizing the blur level sub-index and the visual information loss sub-index, the TBV index of the bridge-side scene image is calculated by linear combination, and is expressed as:

[0079] ;

[0080] in, The TBV (Total Volume Value) metric represents the image of the bridge side scene. These represent the sub-indicators of ambiguity and visual information loss, respectively. All represent weighting coefficients, reflecting the degree of impact of blurriness on image quality and the degree of impact of visual information loss on image quality, respectively. In this embodiment... = 0.6, = 0.4;

[0081] The blur level sub-index measures the degree of blur in an image by calculating the differences between images after Gaussian blurring at different scales. A higher value indicates a more blurred image. Represented as:

[0082] ;

[0083] The visual information loss sub-index assesses the loss of visual information in an image from two dimensions: image entropy and contrast. A higher value indicates a more severe loss of visual information. Represented as:

[0084] ;

[0085] in, These represent the length and width of the image of the scene on the bridge side, respectively. The dimensions of the scene image on the bridge side; These represent the lengths of the bridge-side scene images, respectively. ,Width At Gaussian kernel scale Gaussian function under Gaussian kernel scale The Gaussian function under the given conditions, =1.0, =3.0, which measures the degree of blur by multi-scale convolutional difference, representing Gaussian blur at different scales; This represents an image of the scene on the side of the bridge. This represents the convolution operation; Image representing the scene on the side of the bridge Entropy is used to measure the information richness of an image; This represents the ideal state where the image information is most abundant; It represents the average image contrast, reflecting the brightness differences between different areas of the image; This represents the ideal mean maximum contrast. These represent the contributions of adjusting image entropy to visual information loss and adjusting contrast to visual information loss, respectively.

[0086] To accurately determine image quality, the TBV (Total Product Value) index is divided into four levels: Excellent (TBV < 0.3), Good (0.3 ≤ TBV < 0.5), Average (0.5 ≤ TBV < 0.7), and Poor (TBV ≥ 0.7). When the TBV index of the bridge-side scene image reaches the Average or Poor level, the image quality is deemed substandard. In this case, the image enhancement process is immediately triggered, and the original image and related information such as the TBV index are transmitted together.

[0087] Two types of enhancement networks are configured: a Dual Attention & Dynamic Adaptive Deep Curve Estimation (D2A-DCE) model and an improved Non-local Attention Fusion Network (NAFNet) model, to address brightness and structural fuzziness issues respectively. The enhancement path is determined based on the specific values ​​of each sub-index in the TBV (Total Tolerance Value) metric: the D2A-DCE model is used when brightness is insufficient; the improved NAFNet model is used when fuzziness is dominant; if both problems exist simultaneously, the D2A-DCE model and the improved NAFNet model are used sequentially for cascaded enhancement.

[0088] Step 2: Enhance the image based on its quality, specifically as follows:

[0089] As the core component for improving image quality, the image enhancement module introduces the D2A-DCE model and the improved NAFNet model to achieve adaptive enhancement for images under different low-quality visual conditions such as low light and blur, providing high-quality image data support for subsequent multimodal fusion and object detection.

[0090] The D2A-DCE model is an adaptive image enhancement network proposed based on the Zero-Reference Deep Curve Estimation (Zero-DCE++) framework. It constructs a depth curve estimation system with dynamic adjustment capabilities for low-quality visual environments on the bridge side. Its core improvements are reflected in three dimensions: dual-branch parallel enhancement architecture, dynamic amplitude adjustment, and multi-dimensional attention enhancement.

[0091] like Figure 2As shown, the D2A-DCE model includes a two-branch structure and a parameter prediction unit connected in sequence. Each branch of the two-branch structure embeds a Convolutional Block Attention Module (CBAM), which includes a channel attention module and a spatial attention module. Therefore, each branch in the two-branch structure includes a first consecutive convolutional layer, a channel attention module, a spatial attention module, and a second consecutive convolutional layer connected in sequence. The output of the second consecutive convolutional layer in each branch is connected to a parallel iterator and amplitude controller in the parameter prediction unit. The iterator and amplitude controller jointly execute a multi-level iterative enhancement strategy. This embodiment is... The model employs a multi-level iterative enhancement strategy. Specifically, unlike the single-branch input mode of the traditional Zero-DCE++ framework, it synchronously constructs a dual-branch structure, achieving mathematically complementary modeling of global brightness enhancement and local detail mining. This effectively solves the detail loss problem that a single branch is prone to cause under extreme low-light conditions. Furthermore, CBAM modules are embedded in each branch of the dual-branch structure, enabling the model to model the dual importance of channel and spatial dimensions. This allows the network to automatically focus on the boundary of the falling target and key textures in the dark areas, rather than indiscriminately enhancing background noise. Simultaneously, the model introduces an amplitude controller in the parameter prediction unit of the Zero-DCE++ framework. This amplitude controller can dynamically output adjustment coefficients based on real-time image features, allowing the enhancement intensity to adaptively adjust with changes in ambient light. This overcomes the limitation of relatively fixed enhancement parameters in the Zero-DCE++ framework, thereby preventing color cast and noise amplification caused by over-enhancement.

[0092] Input a low-light bridge-side scene image into a pre-built D2A-DCE model, and output an enhanced bridge-side scene image:

[0093] like Figure 2 As shown, the image I of the bridge side scene with weak light is received, and the bridge side scene image I is inverted to obtain the bridge side scene inverted image I' = 1-I. In the processing logic, the dark area of ​​the original image is transformed into the bright area through the inversion operation. The inversion feature is used to help the model to more sensitively capture the structured information in the dark area of ​​the original image, thereby forming semantic complementarity with the original image branch and providing more accurate mapping parameters for the final iterative curve estimation.

[0094] The bridge-side scene image and the inverted bridge-side scene image are respectively input into the dual-branch structure of the D2A-DCE model. The branch containing the bridge-side scene image I is used to model the global brightness curve, while the branch containing the inverted bridge-side scene image I' is used to enhance the local dark details.

[0095] Within each branch, multi-scale feature maps of the image are extracted through the first continuous convolutional layer. The channel attention module and spatial attention module in the CBAM module reweight the multi-scale feature maps with channel attention and spatial attention to obtain enhanced feature maps. This enables the network to automatically identify and focus on key feature regions such as dark areas, human portrait boundaries and target contours in the image. The enhanced feature maps are then dimensionality-reduced and mapped through the second continuous convolutional layer to obtain mapped feature maps.

[0096] The mapped feature map is input into the parameter prediction unit of the D2A-DCE model, and the iterator and amplitude controller work together to process the mapped feature map. The iterative enhancement strategy yields enhanced images of the bridge-side scene:

[0097] In each iteration, an enhancement iterator is used to predict the parameters of the nonlinear enhancement curve of the current image, while the enhancement intensity is dynamically adjusted by an amplitude controller. The iterator generates enhancement parameters, and the amplitude controller generates adjustment coefficients. Each iteration satisfies the following control logic:

[0098] ;

[0099] Based on the enhancement parameters and adjustment coefficients, the enhancement amplitude coefficient is generated, and the enhancement amplitude coefficient for the t-th iteration is obtained. ;

[0100] Generate an enhanced image of the bridge-side scene based on the enhancement amplitude coefficient. The t-th iteration bridge-side scene enhancement image In this formula, The process constructs a non-linear enhancement response to ensure higher gain for low-brightness pixels, thereby effectively suppressing overexposure in bright areas while improving shadow details. The image starts from the first iteration and goes through... Secondary cascade optimization involves multiple enhancement iterations, gradually increasing image brightness and restoring details, ultimately generating an output image with adaptive brightness enhancement.

[0101] in, Indicates the first Level iteration; This represents a nonlinear curve enhancement operator that performs depth curve estimation calculations using an input set of parameters (image features, iteration parameters, and control coefficients). This represents an image of the scene on the side of the bridge. Indicated in the red channel of the mapped feature map Green Channel Blue Channel The following is the prediction of the iterator. Level iteration enhancement parameters; Indicates the control coefficient; They represent the first Level iteration enhancement parameters and adjustment coefficients; Indicates the first Enhanced images of the bridge-side scene through iterative processing.

[0102] The first-level iteration satisfies the control logic:

[0103] ;

[0104] in, This represents the first iteration, which is the initial stage of the entire enhancement process. The input is the mapped feature map. This represents the first-level iterative enhancement parameters predicted by the iterator; based on the first-level iterative enhancement parameters... Level 1 Iteration Adjustment Coefficient Generate the first-level iterative enhancement amplitude coefficient Enhanced image 1 is generated based on the first-level iterative enhancement amplitude coefficient.

[0105] Level 2 iteration satisfies the control logic:

[0106] ;

[0107] in, This indicates the second iteration, which is the subsequent stage of cascaded enhancement, and its input is the enhanced image generated in the previous stage; This represents the second-level iterative enhancement parameters predicted by the iterator; based on the second-level iterative enhancement parameters... Level 2 Iteration Adjustment Coefficient Generate the second-level iterative enhancement amplitude coefficient Enhanced image 2 is generated based on the enhancement amplitude coefficient of the second-level iteration.

[0108] No. Level iteration satisfies control logic:

[0109] ;

[0110] in, Indicates the first This is the final stage of the entire enhancement process, and its output is the final enhanced image of the bridge side scene (output image). Represents the first prediction obtained by the iterator. Level iteration enhancement parameters; based on the first Level Iteration Enhancement Parameters , No. Level Iteration Adjustment Coefficient , generate the first Level Iteration Enhancement Amplitude Coefficient According to the first The enhanced amplitude coefficients of the step-by-step enhancement process generate the enhanced image of the bridge-side scene, which is the output image.

[0111] The improved NAFNet model focuses on solving the image blur problem, effectively restoring image details through a non-local attention mechanism and feature fusion strategy.

[0112] like Figure 3 As shown, the improved NAFNet model includes an encoder, a non-local attention module, and a decoder connected in sequence. The encoder includes a first convolutional layer and three downsampling blocks connected in sequence. The three downsampling blocks are the first downsampling block, the second downsampling block, and the third downsampling block, respectively. The decoder includes three upsampling blocks and a second convolutional layer connected in sequence. The three upsampling blocks are the first upsampling block, the second upsampling block, and the third upsampling block, respectively.

[0113] The output of the third downsampling block is connected to the non-local attention module, and the output of the non-local attention module is connected to the first upsampling block. There is a one-to-one skip connection between each downsampling block of the encoder and each upsampling block of the decoder. That is, there is a skip connection between the first downsampling block and the third upsampling block, and there is a skip connection between the second downsampling block and the second upsampling block. The three upsampling blocks and the three downsampling blocks are all sampling modules with the same structure, including a layer normalization layer, a third convolutional layer, a depthwise separable convolutional layer, a simple gating unit, and a skip connection layer connected in sequence.

[0114] Specifically, the improved NAFNet model builds upon the existing model by introducing a non-local attention module to construct long-range dependencies. This effectively restores image detail degradation caused by factors such as haze and device vibration. The non-local attention module is added between the encoder and decoder of the NAFNet model, within the feature extraction process. After extracting multi-scale features using a convolutional neural network, the non-local attention module calculates the similarity between features at different locations across the entire image and weights and fuses these similar features. This enhances key details in the image that have been blurred, such as the outlines of bridge railings and people crossing them. The similarity function of the non-local attention module uses dot product calculations and transforms the feature vectors through a learnable weight matrix, enabling the model to transcend local limitations, capture similar features from a global perspective, and suppress background noise. Furthermore, the improved NAFNet model employs a hierarchical feature fusion structure, organically combining low-level texture information with high-level semantic information to further improve the clarity and sharpness of images under low-quality visual conditions.

[0115] Input a blurred bridge-side scene image into a pre-built improved NAFNet model, and output an enhanced bridge-side scene image:

[0116] like Figure 3 The diagram shows the overall encoder-decoder architecture and its internal block structure of the improved NAFNet model. First, the improved NAFNet model uses the encoder to extract hierarchical information from the bridge-side scene image, obtaining features at each level. This step corresponds to... Figure 3 The encoder consists of a hierarchical structure consisting of a first convolutional layer and three downsampling blocks. It obtains hierarchical information from the bottom texture to the high semantic level by downsampling layer by layer.

[0117] Then, a non-local attention module is used to calculate the similarity and perform weighted fusion of features at each level to obtain an enhanced feature map. The similarity between features at different locations across the entire image is calculated, and these similar features are weighted and fused to enhance the blurred details in the image. Through the non-local attention module, the model can capture long-distance dependencies between different locations in the image, further enhancing the image's feature representation ability, highlighting target objects, and suppressing background noise. The enhanced feature map is represented as follows:

[0118] ;

[0119] ;

[0120] ;

[0121] in, This indicates that the enhanced feature map is located at... eigenvectors; Indicates the location of features at each level. Feature vectors, at position eigenvectors; These represent the number of feature channels, feature height, and feature width, respectively. Indicates the location of features at each level. eigenvectors In position eigenvectors The similarity function; Indicates the location of features at each level. eigenvectors The characteristic transformation function; Represents the learnable weight matrix; Indicates matrix transpose;

[0122] Finally, the enhanced feature map is decoded and reconstructed using a decoder; this step corresponds to... Figure 3The NAFNet model employs a hierarchical structure consisting of a second convolutional layer and three upsampling blocks within the decoder. It then performs hierarchical feature fusion on the decoded and reconstructed features with multi-level features to obtain an enhanced image of the bridge-side scene. Furthermore, the hierarchical feature fusion structure used in the NAFNet model corresponds to... Figure 3 The skip connection between the encoder and decoder further enhances the clarity and sharpness of the image by organically combining the output of upsampling blocks at different levels with encoder features.

[0123] like Figure 3 As shown, in each upsampling and downsampling block, the input features are sequentially processed through layer normalization, convolution, depthwise separable convolution, simple gating units, and skip connections, thereby achieving accurate restoration of blurred and degraded features. For example, for key details such as bridge railings and the outlines of people crossing the railings, the nonlocal attention mechanism can overcome local regional limitations and capture similar features from a global perspective, thus enhancing details.

[0124] Upsampling and downsampling blocks have the same structure. This design aims to maintain consistency in the logic for repairing blurred features at different resolutions, thereby ensuring alignment between the underlying texture and high-level semantics in terms of computational dimension. However, their physical functions and parameter settings are fundamentally different: the downsampling block is located at the encoder end, using stride convolution to reduce the image resolution step by step, primarily compressing the spatial resolution of the image to extract deep semantic features; while the upsampling block is located at the decoder end, using transposed convolution or interpolation to increase the feature map size step by step, progressively restoring the spatial dimension of the image and fusing the detailed information retained by the encoder with skip connections, ultimately restoring the feature map back to its original size to output an enhanced image. The consistent structure is for the stability of feature flow, but they handle different feature map sizes and spatial purposes—one is responsible for compression, and the other for restoration—therefore they must appear in pairs and symmetrically in the model.

[0125] The image of the bridge side scene, which is in low light and blurry, is sequentially input into the pre-built D2A-DCE model and the pre-built improved NAFNet model, and the output is an enhanced image of the bridge side scene.

[0126] If the image is in low light (judged by the visual information loss sub-index in the TBV index), the D2A-DCE model is used first for processing; if the image is blurry (judged by the blur degree sub-index in the TBV index), the improved NAFNet model is mainly used for enhancement; for complex images with both low light and blur problems, the cascaded processing method of the D2A-DCE model and the improved NAFNet model is used in sequence. First, the D2A-DCE model is used to improve the image brightness and color, and then the improved NAFNet model is used to restore the image details, thereby achieving comprehensive optimization of low-quality images.

[0127] After the enhancement process is completed, the resulting image is output, and the curve parameters of the D2A-DCE model and the attention weights of the improved NAFNet model are recorded simultaneously for analysis of the controllability of the enhancement effect and subsequent model optimization.

[0128] Step 3: Fuse the enhanced images of the bridge side scene, specifically as follows:

[0129] The improved Cross-Modality Feature Fusion (CMFF) model is a feature-level deep fusion structure designed for the characteristics of visible light and infrared images in a bridge fall behavior perception system. Its core improvements are reflected in two dimensions: cross-modal interaction mechanism and structural compensation strategy.

[0130] like Figure 4 As shown, the improved CMFF model includes a parallel bi-branch cross-modal interaction module, a first feature concatenation layer, a feature fusion layer, a local context enhancement layer, and a first residual connection and normalization layer connected in sequence. Each branch in the bi-branch cross-modal interaction module has a bidirectional context transfer path. Each branch after this path includes a second feature concatenation layer, a channel projection layer, and a second residual connection and normalization layer connected in sequence. The features output by the second residual connection and normalization layer in each branch are fed into the first feature concatenation layer for feature concatenation.

[0131] Specifically, the improved CMFF model addresses the information redundancy and mode shift issues present in traditional CMFF modal splicing methods. It introduces a bidirectional context transfer path during the cross-modal interaction stage, utilizing a multi-head cross-attention mechanism to provide supplementary information from the infrared mode to the visible light mode, while simultaneously enabling semantic transfer from the visible light mode to the infrared mode, allowing both modes to perceive each other's semantic information globally. Furthermore, the improved CMFF model introduces a modal residual signal compensation mechanism during the CMFF model fusion and embedding stage. This mechanism calculates the residual term by determining the distribution difference between the visible light and infrared modes. And use the attention module to learn residual augmentation weights. The residual enhancement weight This design enhances the system's ability to perceive structural boundaries and ambiguous regions, achieving bidirectional complementarity and unified expression of contextual information while ensuring feature alignment.

[0132] The enhanced image of the bridge-side scene is input into a pre-built improved CMFF model, and the output is a fused feature map:

[0133] like Figure 4As shown, the improved CMFF model adopts a two-stage fusion strategy, namely the cross-modal interaction stage and the fusion embedding stage. The core objective of the first stage is to achieve information complementarity and semantic alignment between modalities. The second stage is used to unify the fusion output for subsequent image enhancement, detection, or recognition tasks.

[0134] like Figure 4 As shown, firstly, the enhanced image of the bridge-side scene is input into the visible light mode branch and the infrared mode branch respectively, and the corresponding backbone features are extracted to obtain the intermediate representation of the visible light mode. Infrared mode intermediate representation The two modalities are located at the same spatial resolution. To facilitate cross-modal interactive operation, the system flattens them into a sequence and embeds channels into them.

[0135] Then, to achieve explicit interaction between modalities, two sub-branches are constructed for each modality: one is a residual branch that preserves the structural information of the modality, and the other is a semantic branch for interaction, which is the intermediate representation of the visible light modality. Dividing the infrared modes into visible light residual branch features and visible light interactive branch features, the intermediate representation is... It is divided into infrared residual branch features and infrared interactive branch features;

[0136] In the first stage, the improved CMFF model performs bidirectional cross-modal interaction modeling, and represents the intermediate visible light modalities through a multi-head cross-attention mechanism. Intermediate representation of infrared modes A bidirectional context passing path is established between them to transmit the intermediate representation of the visible light modes. Intermediate representation of infrared modes Semantic information is mutually perceived globally to obtain visible light attention. and infrared attention Specifically, the intermediate representation of visible light modes Constructing visible light query vectors Infrared mode intermediate representation Constructing infrared bond vectors Infrared value vector Visible light attention Represented as:

[0137] ;

[0138] This operation means that the infrared mode provides supplementary information to the RGB; similarly, the reverse operation is performed to provide intermediate representations of the infrared mode. Constructing Infrared Query Vectors The visible light mode represents the middle part. Construct visible light bond vectors Visible light value vector Infrared attention Represented as:

[0139] ;

[0140] in, The scaling dimension representing attention; Indicates matrix transpose;

[0141] Through this mechanism, two modalities can perceive each other's semantic information globally, alleviating the problems of information redundancy and modality shift in traditional modality splicing methods;

[0142] The intermediate representation of visible light modes With visible light attention Residual connections yield visible light enhancement representations. Infrared mode intermediate representation With infrared attention Residual connection yields infrared-enhanced representation. ;

[0143] In the second stage, the improved CMFF model performs unified fusion coding to enhance the infrared representation. After feature fusion with visible light interactive branch features, feature splicing with visible light residual branch features is performed to obtain visible light compressed features. Channel projection, residual connection and normalization are performed on the visible light compressed features in sequence to obtain visible light fused features, realizing dimension alignment and preliminary fusion.

[0144] Visible light enhancement representation After feature fusion with infrared interactive branch features, feature splicing with infrared residual branch features is performed to obtain infrared compressed features. Channel projection, residual connection and normalization are performed on the infrared compressed features in sequence to obtain infrared fused features, realizing dimension alignment and preliminary fusion.

[0145] Visible light fusion features and infrared fusion features are stitched together to obtain stitched features. These stitched features are then subjected to feature fusion, local context enhancement, residual connection, and normalization to obtain hybrid features. Local context enhancement can use depthwise separable convolutions, lightweight residual modules, or spatial pyramid structures (such as parallel 3×3 / 5×5 convolutions) to model edge, texture, and detail information in local regions.

[0146] Considering the differences in the distribution of visible light modes and infrared modes, the system introduces a residual signal compensation mechanism to calculate the intermediate representation of visible light modes. Intermediate representation of infrared modes residuals , the residual term Residual enhancement weights are obtained through a squeeze-and-excitation (SE) attention module. Weight the residuals Incorporating hybrid features , to obtain the fused feature map .

[0147] In the process of information interaction and fusion, not only is modality specificity maintained, but semantic consistency is also enhanced. Furthermore, residual signals are introduced to enhance the ability to express structural details. The final output is a fused feature map after semantic alignment, structural compensation, and context enhancement.

[0148] Employing a bidirectional cross-modal interaction mechanism, it achieves accurate information transfer and complementarity between different modalities. During the fusion process, modal residual paths are preserved, enhancing the system's ability to perceive structural boundaries and ambiguous regions. A lightweight contextual convolutional structure is introduced to effectively control the number of parameters and inference time, making it suitable for edge deployment and applicable to multimodal fusion tasks in various scenarios, including low-light image enhancement, infrared-assisted recognition, and small target detection in complex scenes. It achieves high-quality feature interaction and unified expression under different visual conditions and modal combinations.

[0149] Step 4: Perform bridge-side fall behavior perception on the fused feature map, specifically:

[0150] Small target detection is a key step in achieving real-time perception of bridge fall behavior. Its core function is to accurately detect small targets with indistinct features in the bridge scene, including people jumping over the railing and water splashes generated at the moment of fall. In bridge monitoring scenarios, these targets are usually small in size, have sparse features, and are easily affected by low-quality visual conditions such as low light, fog, and rain. Traditional detection algorithms often have significant shortcomings in terms of detection accuracy and real-time performance.

[0151] In this embodiment, the Bridge Side-view Integrated Detection and Enhancement Network (Bridge-SIDE Net) model is an improvement on the Real-Time Detection Transformer (RT-DETR) architecture. Each feature extraction module in the backbone network of the RT-DETR architecture is replaced with a window self-attention convolution (Swin Convolution, SwinConv) module. At the same time, a Feature Pyramid Network (FPN) module is added after the backbone network in the RT-DETR architecture. Combined with the multi-scale feature fusion strategy of the FPN module, the global information modeling and local feature enhancement are organically combined, thereby significantly improving the detection performance and robustness of small targets.

[0152] like Figure 5 As shown, the Bridge-SIDE Net model includes the backbone network, FPN module, encoder, and decoder of the RT-DETR architecture connected in sequence. The backbone network of the RT-DETR architecture includes a 3×3 convolutional layer, the first residual block in stage 1, the second residual block in stage 2, the third residual block in stage 3, and the fourth residual block in stage 4 connected in sequence. Each residual block has the same structure. The SwinConv module is the last layer embedded in each residual block. The SwinConv module includes a first 1×1 convolutional layer, a feature segmentation layer, a window self-attention block and a residual convolutional block connected in parallel, a third feature concatenation layer, and a second 1×1 convolutional layer connected in sequence.

[0153] The encoder in the RT-DETR architecture includes a flattening and linear projection layer, a spatial location encoding layer, a third residual connection and normalization layer, a feedforward network, and a fourth residual connection and normalization layer, connected in sequence. The backbone network and the encoder output in the RT-DETR architecture are connected to the FPN module, and the output of the FPN module is connected to the decoder in the RT-DETR architecture. The decoder interacts with the detection head to complete target category prediction and bounding box regression.

[0154] The fused feature map is input into the pre-built Bridge-SIDE Net model, and the output is the bridge-side fall behavior perception result:

[0155] like Figure 5As shown, firstly, the fused feature map is input into the backbone network. After performing 3×3 convolution on the fused feature map, it is sequentially processed through the first residual block, the second residual block, the third residual block, and the fourth residual block for feature extraction, resulting in the final stage feature map S5, the second stage feature map S3 output from the second residual block, and the third stage feature map S4 output from the third residual block. In each residual block, the feature map is divided into non-overlapping local windows w×w by the SwinConv module. Multi-head self-attention (W-MHSA) is calculated within each local window. The local window self-attention of the SwinConv module is represented as follows:

[0156] ;

[0157] ;

[0158] in, This represents the intermediate features after passing through the window self-attention block and the residual convolution block; This indicates the output of the previous residual block; Presentation layer normalization operation; This indicates the focus of the bulls; Indicates the output of the current residual block; Indicates a feedforward network;

[0159] like Figure 5 As shown, each SwinConv module includes a first 1×1 convolution, feature segmentation, window self-attention block and residual convolution block connected in parallel, feature concatenation, and a second 1×1 convolution. The data processing steps are detailed in the prior art. Through in-window attention, the SwinConv module can capture local small targets and their surrounding context. At the same time, the convolution operation further extracts local texture and boundary information, so that the features of small targets are enhanced while preserving local details.

[0160] Then, the final stage feature map S5 is converted into a one-dimensional sequence through flattening and linear projection operations. The spatial domain position information of the one-dimensional sequence is preserved through spatial position encoding to obtain the encoded features.

[0161] Next, the encoded features are globally modeled using a multi-head self-attention mechanism, then residually connected and normalized with a one-dimensional sequence to obtain processed features. These processed features are then refined using a feedforward network, and residually connected and normalized again to obtain normalized features. This step is handled by the encoder in the RT-DETR architecture. The encoder performs global feature modeling based on a multi-head self-attention mechanism. Through this mechanism, the encoder can capture long-distance dependencies between small targets and other regions globally, thereby enhancing the expressive power of small targets in complex scenes. Subsequently, the decoder in the RT-DETR architecture further focuses on candidate target regions through cross-attention and interacts with the detection head to complete target category prediction and bounding box regression. In this process, the global modeling capability of the RT-DETR architecture and the local detail enhancement of the SwinConv module complement each other: the SwinConv module is responsible for preserving and enhancing the subtle texture information of local small targets, while the encoder in the RT-DETR architecture allocates attention weights in the global context, enabling the model to accurately distinguish targets from the background in complex bridge-side environments.

[0162] Then, the normalized features are input into the FPN module along with the second-stage feature map S3 and the third-stage feature map S4 to perform cross-scale information interaction and obtain interactive features. The first-stage feature map output by the first residual block is a high-resolution shallow feature that mainly serves as the foundation. If it is directly introduced into the FPN module loop, it will significantly increase the computational burden of the system. Therefore, it is not introduced into the FPN module to ensure the real-time response performance of the system at the edge.

[0163] Finally, the interaction features are used to focus on the target region and predict the target category to obtain the bridge fall behavior perception results. This step is processed by the decoder in the RT-DETR architecture, and the bridge fall behavior perception results are output by the detection head.

[0164] One of the core improvements of the Bridge-SIDE Net model is the embedding of a SwinConv module in the last layer of each residual block in the backbone network of the RT-DETR architecture. A residual block refers to a hierarchical processing module in the backbone network used to extract features at different scales, such as... Figure 5As shown, the backbone network of the RT-DETR architecture contains a hierarchical feature extraction structure consisting of stages 1, 2, 3, and 4. After the input image undergoes initial 3×3 convolution processing, it sequentially passes through the residual blocks in each stage for multi-scale feature extraction. The SwinConv module is specifically embedded in the last layer of the residual block in each stage. It enhances local small targets and contextual relationships by calculating multi-head self-attention windows, while extracting local texture and boundary information. This achieves the goal of enhancing the expressive power of the feature maps in each stage while maintaining computational efficiency. The output of stage 2 is the second-stage feature map S3, the output of stage 3 is the third-stage feature map S4, and the output of stage 4 is the final-stage feature map S5.

[0165] In the RT-DETR architecture, an FPN module is introduced after the backbone network to perform multi-scale feature fusion. Through upsampling and lateral connections, cross-scale information interaction between feature maps of different levels (such as S3, S4, and S5) is achieved to solve the problem that small targets are easily lost during feature downsampling.

[0166] In addition, the query selection mechanism of the decoder was improved by adopting the Intersection over Union (IoU)-aware query selection strategy and combining it with a comprehensive optimization scheme including a small target weighted loss mechanism, a multi-scale training strategy, and focus loss and IoU-aware loss. This enables accurate detection and location regression of minute targets such as people jumping over barriers and splashes caused by falling behavior in complex and low-quality environments.

[0167] In this embodiment, the first and second consecutive convolutional layers have the same structure, the first, second, and third convolutional layers have the same structure, the first, second, and third feature splicing layers have the same structure, the first, second, and third feature splicing layers have the same structure, the first, second, third, and fourth residual connection and normalization layers have the same structure, and the first and second 1×1 convolutional layers have the same structure.

[0168] In this embodiment, the Bridge-SIDE Net model is used to preserve small target information in high-resolution shallow features and enhance semantic expression in deep features, thus solving the problem of small targets being easily lost during downsampling. In terms of training and optimization, this invention adopts low-quality image enhancement strategies, including low-light, haze, and blurred data enhancement; improves the weight of small target samples through a small target weighted loss mechanism; improves the robustness of targets of different sizes by adopting a multi-scale training strategy; and optimizes the detection effect of difficult samples by combining focus loss and cross-union ratio perception loss, thereby reducing false negatives and false positives. Through global context modeling of the RT-DETR architecture, local detail enhancement of the SwinConv module, and multi-scale fusion of the FPN module, it can quickly and accurately complete the detection of small targets such as people crossing the fence and water splashes in complex and low-quality environments on the bridge side, providing reliable data support for subsequent behavior chain recognition, while meeting real-time requirements in terms of computational efficiency.

[0169] Example 2

[0170] Based on Example 1, this example presents an experimental case study of a bridge-side fall behavior perception method suitable for low visual conditions:

[0171] To acquire images of the bridge side scene, the acquisition resolution, frame rate, and modal type can be adjusted according to actual needs to adapt to different environmental conditions. The resolution of the acquired images is 1920×1080, and the sampling frequency is 25 to 30 frames per second to ensure video continuity and small target motion capture.

[0172] The acquired images are quantitatively analyzed to determine whether the image enhancement process needs to be triggered. The image enhancement process optimizes low-quality images through adaptive brightness, color adjustment, and detail restoration strategies, providing high-quality image data for subsequent analysis.

[0173] Multimodal fusion enables deep fusion of features from visible light and infrared modes, enhancing the system's semantic understanding and structural perception of targets in complex environments.

[0174] Small target detection is based on the Bridge-SIDE Net model, which accurately detects small and inconspicuous targets (such as people jumping over the railing and water splashes at the moment of falling) in the bridge side scene, thereby realizing real-time perception of falling behavior on the bridge side.

[0175] When the TBV index of the bridge side scene image is ≥0.3, that is, in the general or poor level, the system judges the image as a low-quality image, automatically triggers image enhancement processing, and inputs the TBV index of the bridge side scene image and the current bridge side scene image together for image enhancement.

[0176] like Figure 2As shown, the D2A-DCE model is an image enhancement network with dynamic control capabilities proposed based on the Zero-DCE++ framework. Its network structure consists of two enhancement branches, which model the global brightness curve and the detail enhancement requirements, respectively. To improve the enhancement effect, the network introduces a CBAM module in each branch to increase the enhancement weight of key image regions (such as dark areas and portrait boundaries) from both channel and spatial dimensions.

[0177] In the specific enhancement process, the system performs the following steps on the image: The process involves multiple iterations. In each iteration, the iterator generates enhancement parameters based on the current state of the image and combines them with the adjustment coefficients output by the amplitude controller to generate the final enhancement amplitude coefficient, thereby generating an enhanced image of the bridge-side scene. The enhancement process is generally completed within 4 to 6 iterations.

[0178] In addition to brightness issues, the images also exhibit blurring and degradation (such as in the TBV index of bridge-side scene images). If the value is greater than 0.5, then after processing with the D2A-DCE model, the improved NAFNet model is called for blur restoration. The improved NAFNet model introduces a non-local attention mechanism to build long-distance dependencies, finds similar feature regions across the entire image, and fuses their contextual information for detail restoration. This structure allows weak edge regions in the image to obtain information about structurally similar regions across the entire image, thereby achieving cross-regional feature enhancement and significantly improving the recognizability of details such as bridge railings, diving action contours, and water splash deformation.

[0179] The image suffers from a combined degradation problem of brightness and blur. A cascaded enhancement path is adopted, which first uses a nonlinear curve to improve brightness and contrast, and then uses a non-local attention mechanism to restore texture and edges, ultimately outputting a clear enhanced image of the bridge side scene.

[0180] The enhancement results are returned to the image quality assessment for re-examination. When the TBV index of the enhanced image of the bridge side scene drops below 0.3, the enhancement is considered successful. If it still does not meet the standard, the system will automatically adjust the next round of enhancement strategy based on the parameter feedback recorded in the enhancement log, including increasing the enhancement amplitude, re-initializing curve parameters, or optimizing the attention region, and will perform a maximum of one second enhancement to prevent entering an infinite loop.

[0181] The input feature summary, enhancement path selection, output image quality, and processing time throughout the entire image enhancement process will be recorded in the log. Combined with the performance indicators of downstream detection modules (such as the recall rate of detecting people crossing the fence), an enhancement-detection mapping table will be generated to support the self-learning and dynamic strategy adjustment of the subsequent enhancement network.

[0182] The enhanced images of the bridge side scene are then fed into the improved CMFF model and the Bridge-SIDE Net model in sequence for feature fusion and target detection, outputting the bridge side fall behavior perception results.

[0183] During training, the Bridge-SIDE Net model employs a comprehensive strategy to improve its ability to detect small targets on the bridge side under low-quality visual conditions. Input images are randomly scaled to different sizes to ensure the network can adapt to small targets of varying sizes, improving scale invariance. An optimizer combined with a cosine annealing scheduler is used for learning rate decay. The loss for small targets is closely monitored in each training epoch to ensure stable convergence. Based on hardware resource configuration (e.g., four 2080Ti GPUs), a multi-GPU training strategy with a batch size of 16 per GPU is used, along with a gradient accumulation mechanism to address memory limitations. To prevent overfitting, detection accuracy is evaluated on the validation set every 10 training epochs. If the validation metrics do not improve for five consecutive epochs, an early stopping mechanism is triggered, ensuring both efficient and robust model training.

[0184] Through the above collaborative efforts, real-time perception of bridge side fall behavior can be achieved under low-quality visual conditions, significantly improving the accuracy of target recognition, reducing false alarms and missed alarms, and providing reliable technical support for bridge safety monitoring.

[0185] Example 3

[0186] This embodiment introduces a bridge fall behavior perception system suitable for low-quality visual conditions, including:

[0187] The image acquisition module is used to: acquire images of the bridge side scene;

[0188] The quality judgment module is used to: calculate the TBV index of the bridge side scene image, and determine whether the bridge side scene image has weak light and / or blur based on the TBV index of the bridge side scene image;

[0189] An image enhancement module is used to: input a bridge-side scene image with low light into a pre-built D2A-DCE model and output an enhanced bridge-side scene image; input a bridge-side scene image with blurriness into a pre-built improved NAFNet model and output an enhanced bridge-side scene image; input a bridge-side scene image with both low light and blurriness sequentially into the pre-built D2A-DCE model and the pre-built improved NAFNet model and output an enhanced bridge-side scene image; the D2A-DCE model is constructed by replacing the single-branch structure of the Zero-DCE++ framework with a dual-branch structure in which each branch embeds a CBAM module, and adding an amplitude controller in the parameter prediction stage of the Zero-DCE++ framework; the improved NAFNet model is constructed by adding a non-local attention module between the encoder and decoder of the NAFNet model;

[0190] The image fusion module is used to: input the enhanced image of the bridge-side scene into a pre-constructed improved CMFF model and output a fused feature map; the improved CMFF model is constructed by adding a bidirectional context transfer path in the cross-modal interaction stage of the CMFF model and adding a modal residual signal compensation mechanism in the fusion embedding stage;

[0191] The behavior perception module is used to: input the fused feature map into the pre-built Bridge-SIDE Net model and output the bridge-side fall behavior perception result; the Bridge-SIDE Net model is constructed by embedding the SwinConv module in each feature extraction stage of the backbone network in the RT-DETR architecture and adding the FPN module after the backbone network.

[0192] The specific functions of each module described above are explained in the relevant content of Embodiment 1 or 2, and will not be repeated here.

[0193] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0194] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0195] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0196] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0197] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for perceiving bridge fall behavior under low-quality visual conditions, characterized in that, include: Acquire images of the bridge side scene; Calculate the TBV index of the bridge-side scene image, and determine whether the bridge-side scene image has weak light and / or blur based on the TBV index; The image of the bridge side scene with low light is input into the pre-built D2A-DCE model, and the enhanced image of the bridge side scene is output. The blurred bridge-side scene image is input into a pre-built improved NAFNet model, and the output is an enhanced bridge-side scene image; The image of a bridge-side scene with low light and blurriness is sequentially input into a pre-built D2A-DCE model and a pre-built improved NAFNet model, and the output is an enhanced image of the bridge-side scene. The D2A-DCE model is constructed by replacing the single-branch structure of the Zero-DCE++ framework with a dual-branch structure in which each branch embeds a CBAM module, and adding an amplitude controller to the parameter prediction unit of the Zero-DCE++ framework. The improved NAFNet model is constructed by adding a non-local attention module between the encoder and decoder of the NAFNet model. The enhanced image of the bridge-side scene is input into a pre-constructed improved CMFF model, and a fused feature map is output. The improved CMFF model is constructed by adding a bidirectional context transfer path in the cross-modal interaction stage of the CMFF model and a modal residual signal compensation mechanism in the fusion embedding stage. The fused feature map is input into the pre-built Bridge-SIDE Net model, and the bridge-side fall behavior perception result is output. The Bridge-SIDE Net model is constructed by embedding a SwinConv module in the last layer of each residual block of the backbone network in the RT-DETR architecture and adding an FPN module after the backbone network. The process of inputting a low-light bridge-side scene image into a pre-constructed D2A-DCE model and outputting an enhanced bridge-side scene image includes: The bridge-side scene image is inverted to obtain an inverted bridge-side scene image; The bridge-side scene image and the bridge-side scene inverted image are respectively input into the dual-branch structure of the D2A-DCE model. In each branch, the multi-scale feature map of the image is extracted through the first continuous convolutional layer. The channel attention module and spatial attention module in the CBAM module are used to reweight the multi-scale feature map with channel attention and spatial attention to obtain the enhanced feature map. The enhanced feature map is then dimensionality-reduced and mapped through the second continuous convolutional layer to obtain the mapped feature map. The mapped feature map is input into the parameter prediction unit of the D2A-DCE model, and the iterator and amplitude controller work together to process the mapped feature map. An iterative enhancement strategy is used to obtain an enhanced image of the bridge-side scene; the enhancement amplitude and brightness offset are corrected in real time by an amplitude controller.

2. The bridge fall behavior perception method applicable to low-quality visual conditions according to claim 1, characterized in that, The TBV index of the bridge-side scene image is expressed as: ； ； ； in, The TBV (Total Volume Value) metric represents the image of the bridge side scene. These represent the sub-indicators of ambiguity and visual information loss, respectively. All represent weighting coefficients; These represent the length and width of the image of the scene on the bridge side, respectively. These represent the lengths of the bridge-side scene images, respectively. ,Width At Gaussian kernel scale Gaussian function under Gaussian kernel scale The Gaussian function under the following conditions; This represents an image of the scene on the side of the bridge. This represents the convolution operation; Image representing the scene on the side of the bridge Entropy; This represents the ideal state where the image information is most abundant; This represents the average image contrast. This represents the ideal mean maximum contrast. These represent the contributions of adjusting image entropy to visual information loss and adjusting contrast to visual information loss, respectively.

3. The bridge fall behavior perception method applicable to low-quality visual conditions according to claim 1, characterized in that, The mapping feature map is executed jointly by iterator and amplitude controller. A multi-level iterative enhancement strategy is used to obtain enhanced images of the bridge-side scene, including: The mapping feature map is executed jointly by iterator and amplitude controller. The iterative enhancement strategy involves an iterator generating enhancement parameters and an amplitude controller generating adjustment coefficients. Each iteration satisfies the following control logic: ； Based on the enhancement parameters and adjustment coefficients, the enhancement amplitude coefficient is generated. Level Iteration Enhancement Amplitude Coefficient ; An enhanced image of the bridge-side scene is generated based on the enhancement amplitude coefficient. Level Iteration Bridge Side Scene Enhancement Image ; in, Indicates the first Level iteration; This represents a nonlinear curve enhancement operator; This represents an image of the scene on the side of the bridge. Indicated in the red channel of the mapped feature map Green Channel Blue Channel The following is the prediction of the iterator. Level iteration enhancement parameters; All represent control coefficients; They represent the first Level iteration enhancement parameters and adjustment coefficients; Indicates the first Enhanced images of the bridge-side scene through iterative processing.

4. The bridge fall behavior perception method applicable to low-quality visual conditions according to claim 1, characterized in that, The process of inputting a blurred bridge-side scene image into a pre-built improved NAFNet model and outputting an enhanced bridge-side scene image includes: In the encoder, hierarchical information extraction is performed on the bridge-side scene image to obtain features at each level; In the non-local attention module, similarity calculation and weighted fusion are performed on the features at each level to obtain an enhanced feature map; In the decoder, the enhanced feature map is decoded and reconstructed, and the decoded and reconstructed features are fused with the features at each level to obtain an enhanced image of the bridge side scene.

5. The bridge fall behavior perception method applicable to low-quality visual conditions according to claim 4, characterized in that, The enhanced feature map is represented as follows: ； ； ； in, This indicates that the enhanced feature map is located at... eigenvectors; Indicates the location of features at each level. Feature vectors, at position eigenvectors; These represent the number of feature channels, feature height, and feature width, respectively. Indicates the location of features at each level. eigenvectors In position eigenvectors The similarity function; Indicates the location of features at each level. eigenvectors The characteristic transformation function; Represents the learnable weight matrix; This indicates the matrix transpose.

6. The bridge fall behavior perception method applicable to low-quality visual conditions according to claim 1, characterized in that, The step of inputting the enhanced image of the bridge-side scene into a pre-constructed improved CMFF model and outputting a fused feature map includes: The enhanced image of the bridge-side scene is input into the visible light mode branch and the infrared mode branch respectively, and the corresponding backbone features are extracted to obtain the visible light mode intermediate representation and the infrared mode intermediate representation. The intermediate representation of visible light modes is divided into visible light residual branch features and visible light interactive branch features, and the intermediate representation of infrared modes is divided into infrared residual branch features and infrared interactive branch features; A bidirectional context transfer path is constructed between the visible light modal intermediate representation and the infrared modal intermediate representation, enabling the visible light modal intermediate representation and the infrared modal intermediate representation to mutually perceive semantic information in a global scope, thereby obtaining visible light attention and infrared attention; By connecting the intermediate representation of the visible light mode with the visible light attention residual, we obtain the visible light enhanced representation; by connecting the intermediate representation of the infrared mode with the infrared attention residual, we obtain the infrared enhanced representation. After feature fusion of infrared enhancement representation and visible light interactive branch features, feature splicing is performed with visible light residual branch features to obtain visible light compressed features. Channel projection, residual connection and normalization are then performed on the visible light compressed features in sequence to obtain visible light fused features. After fusing the visible light enhanced representation with the infrared interactive branch features, it is then spliced ​​with the infrared residual branch features to obtain the infrared compressed features. The infrared compressed features are then subjected to channel projection, residual connection and normalization in sequence to obtain the infrared fused features. Visible light fusion features and infrared fusion features are spliced ​​together to obtain spliced ​​features. The spliced ​​features are then subjected to feature fusion, local context enhancement, residual connection and normalization in sequence to obtain hybrid features. Calculate the intermediate representation of visible light modes Intermediate representation of infrared modes residuals The residual term Residual enhancement weights are obtained through the SE attention module. Weight the residuals Incorporating hybrid features , to obtain the fused feature map .

7. The bridge fall behavior perception method applicable to low-quality visual conditions according to claim 6, characterized in that, The visible light attention is represented as: ； The infrared attention is represented as: ； in, These represent visible light attention and infrared attention, respectively. These represent the visible light query vector, key vector, and value vector, respectively. These represent the infrared query vector, key vector, and value vector, respectively. The scaling dimension representing attention; This indicates the matrix transpose.

8. The bridge fall behavior perception method applicable to low-quality visual conditions according to claim 1, characterized in that, The step of inputting the fused feature map into the pre-constructed Bridge-SIDE Net model and outputting the bridge-side fall behavior perception result includes: The fused feature map is input into the backbone network. After convolution processing, the fused feature map is sequentially processed through the first residual block, the second residual block, the third residual block, and the fourth residual block for feature extraction, resulting in the final stage feature map, the second stage feature map output by the second residual block, and the third stage feature map output by the third residual block. In each residual block, the feature map is divided into non-overlapping local windows by the SwinConv module. Multi-head self-attention is calculated in each local window to capture local and contextual relationships, while extracting local texture and boundary information. The feature map of the last stage is converted into a one-dimensional sequence through flattening and linear projection operations. Spatial location encoding is used to retain the spatial domain location information of the one-dimensional sequence to obtain the encoded features. After the encoded features are globally modeled using a multi-head self-attention mechanism, they are residually connected and normalized with a one-dimensional sequence to obtain processed features. The processed features are then refined using a feedforward network and residually connected and normalized with the processed features to obtain normalized features. The normalized features are input into the FPN module along with the second-stage and third-stage feature maps to perform cross-scale information interaction and obtain interactive features. The interaction features are used to focus on the target region and predict the target category to obtain the bridge side fall behavior perception results.

9. A bridge fall behavior perception system suitable for low-quality visual conditions, characterized in that, The system for performing the method according to any one of claims 1-8, the system comprising: The image acquisition module is used to: acquire images of the bridge side scene; The quality judgment module is used to: calculate the TBV index of the bridge side scene image, and determine whether the bridge side scene image has weak light and / or blur based on the TBV index of the bridge side scene image; An image enhancement module is used to: input a bridge-side scene image with low light into a pre-built D2A-DCE model and output an enhanced bridge-side scene image; input a bridge-side scene image with blurriness into a pre-built improved NAFNet model and output an enhanced bridge-side scene image; input a bridge-side scene image with both low light and blurriness sequentially into the pre-built D2A-DCE model and the pre-built improved NAFNet model and output an enhanced bridge-side scene image; the D2A-DCE model is constructed by replacing the single-branch structure of the Zero-DCE++ framework with a dual-branch structure in which each branch is embedded in a CBAM module, and adding an amplitude controller to the parameter prediction unit of the Zero-DCE++ framework; the improved NAFNet model is constructed by adding a non-local attention module between the encoder and decoder of the NAFNet model; The image fusion module is used to: input the enhanced image of the bridge-side scene into a pre-constructed improved CMFF model and output a fused feature map; the improved CMFF model is constructed by adding a bidirectional context transfer path in the cross-modal interaction stage of the CMFF model and adding a modal residual signal compensation mechanism in the fusion embedding stage; The behavior perception module is used to: input the fused feature map into the pre-built Bridge-SIDE Net model and output the bridge-side fall behavior perception result; the Bridge-SIDE Net model is constructed by embedding a SwinConv module in the last layer of each residual block of the backbone network in the RT-DETR architecture and adding an FPN module after the backbone network.

Images