Multi-channel feature fusion method and system based on environmental adaptation

Abstract

The invention relates to the technical field of image processing, in particular to a multi-channel feature fusion method and system based on environmental adaptation, and the method comprises the steps: obtaining the multi-channel features of input data; environment degradation factors including illumination brightness, illumination uniformity, image definition, view angle quality and scene complexity are extracted from the input data; inputting the environment degradation factor into a learnable mapping network to generate an environment adaptive weight of each channel; applying a hard threshold gating function to the weight to generate a binary gating mask; multiplying the weight by the mask to obtain a final weight; and scaling and splicing each channel feature vector based on the final weight to obtain a fusion feature. According to the method, dynamic weight distribution and effective filtering of low-quality channels are realized, the problem of insufficient robustness of a traditional method in an environment degradation scene is solved, and the accuracy and reliability of multi-channel feature fusion in a complex environment are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a multi-channel feature fusion method and system based on environment adaptation. Background Technology

[0002] With the rapid development of computer vision technology, video analysis methods based on multi-channel feature fusion have become a research hotspot. Traditional fusion methods employ simple feature concatenation or fixed-weight strategies, assuming that the feature quality of each channel remains constant. In recent years, attention mechanisms have been introduced into the field of multimodal fusion. Methods such as SE-Net and CBAM adaptively weight the features by learning the relative importance of channels. However, these methods are mainly based on the statistical properties of the features themselves (such as mean and variance), without explicitly modeling the influence of external environmental factors on feature quality.

[0003] In practical applications, video data faces complex environmental conditions such as changes in lighting, image blurring, occlusion, and changing viewpoints, leading to significant differences in feature quality across different channels. Existing technologies lack robustness in degraded environmental scenarios; fixed-weight or feature-statistic-based adaptive methods cannot effectively address the feature quality degradation caused by environmental changes, and noise introduced by low-quality channels severely impacts the fusion results. Therefore, how to dynamically evaluate the feature quality of each channel and adaptively allocate weights based on environmental conditions has become a critical problem that urgently needs to be solved in the field of multi-channel feature fusion. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a multi-channel feature fusion method and system based on environment adaptation. By explicitly modeling the impact of environmental factors on feature quality, it achieves dynamic weight allocation and effective filtering of low-quality channels, significantly improving the fusion robustness in complex environments.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] On one hand, embodiments of the present invention provide a multi-channel feature fusion method based on environment adaptation, the method comprising the following steps:

[0007] Acquire video frame data, extract multi-channel features from the video frame data, with each channel corresponding to a feature type and output a feature vector;

[0008] Environmental degradation factors are extracted from the video frame data. These environmental degradation factors include illumination intensity factor, illumination uniformity factor, image sharpness factor, viewpoint quality factor, and scene complexity factor.

[0009] The environmental degradation factor is input into a learnable mapping network to generate environmental adaptive weights for each channel;

[0010] A hard threshold gating function is applied to the environmental adaptive weights to generate a binary gating mask;

[0011] The environment-adaptive weights are multiplied element-wise by the binary gate mask to obtain the final weights of each channel.

[0012] Based on the final weights, the feature vectors of each channel are weighted and scaled, and the scaled feature vectors are concatenated to obtain a fused feature vector; the fused feature vector is then input into the downstream task model for processing.

[0013] Optionally, the acquisition of multi-channel features of the input data includes:

[0014] Global appearance features were extracted using a DINOv3 pre-trained model;

[0015] The HRNet model is used to extract key point features, and the key point features are encoded using the ResNet-18 model.

[0016] Temporal dynamic features were extracted using the 3DResNet-18 model;

[0017] Local region features are extracted using ROI cropping and then encoded using a ResNet-18 model.

[0018] In this case, the feature vectors output by each channel have the same dimension.

[0019] Optionally, extracting environmental degradation factors from the video frame data includes:

[0020] The video frame data is converted from RGB image to YUV color space, and the luminance channel is extracted.

[0021] The global average brightness is calculated based on the brightness channel to obtain the luminance factor.

[0022] The standard deviation of the brightness channel is calculated to obtain the illumination uniformity factor;

[0023] The image sharpness factor is obtained by calculating the image sharpness using the Laplacian operator and obtaining the variance of the Laplacian response.

[0024] The average value of the keypoint confidence output by the keypoint detector is calculated to obtain the viewpoint quality factor.

[0025] The number of detected targets is counted to obtain the scene complexity factor.

[0026] Optionally, the step of inputting the environmental degradation factor into a learnable mapping network to generate environmentally adaptive weights for each channel includes:

[0027] The environmental factor vector composed of the environmental degradation factors is normalized.

[0028] The normalized environmental factor vector is input into the mapping network, which consists of at least one fully connected hidden layer, a ReLU activation function, and an output layer.

[0029] The output layer is normalized by the Softmax function to generate N-dimensional environment adaptive weights, where N is the number of channels and the sum of all weights is 1.

[0030] Optionally, applying a hard threshold gating function to the environment adaptive weights to generate a binary gating mask includes:

[0031] Calculate the difference between the current channel's environment adaptive weight value and the gating threshold;

[0032] When the difference is less than -3, the binary gate mask is 0; when the difference is greater than 3, the binary gate mask is 1.

[0033] When the difference is between -3 and 3, add 3 to the difference and then divide by 6 to get an intermediate value; take the maximum value of the intermediate value compared with 0, and then take the minimum value of the maximum value compared with 1 to get the binary gating mask.

[0034] Optionally, the step of weight scaling of the feature vectors of each channel based on the final weights, and concatenating the scaled feature vectors to obtain a fused feature vector, includes:

[0035] The feature vector of each channel is multiplied by its corresponding final weight to obtain the scaled feature vector;

[0036] The scaled feature vectors of each channel are concatenated according to the channel dimension to obtain a preliminary fused feature vector;

[0037] The initial fused feature vector is concatenated with the original feature vectors of each channel to obtain the original concatenated feature vector. The residuals are added according to the trainable coefficients to obtain the fused feature vector for downstream tasks. The trainable coefficients are variable weights that are learned and adjusted through backpropagation.

[0038] Optionally, the method further includes a training step, wherein the total training loss is obtained in the following manner:

[0039] Calculate the square of the Euclidean distance between the anchor feature mapping result and the positive sample feature mapping result, subtract the square of the Euclidean distance between the anchor feature mapping result and the negative sample feature mapping result, add a boundary parameter, and then take the maximum value between the result and 0 to obtain the triplet loss.

[0040] The L2 regularization term is obtained by multiplying the square of the Euclidean distance between the environment adaptive weight vector and a uniformly distributed weight vector by a regularization coefficient.

[0041] The sum of the triplet loss and the L2 regularization term is taken as the total loss.

[0042] On the other hand, embodiments of the present invention provide an environment-adaptive multi-channel feature fusion system, including:

[0043] At least one processor;

[0044] At least one memory for storing at least one program;

[0045] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0046] On the other hand, embodiments of the present invention provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the above-described method.

[0047] The beneficial effects of this invention are as follows: This invention discloses a multi-channel feature fusion method and system based on environment adaptation. By explicitly introducing environmental degradation factors such as illumination intensity, illumination uniformity, and image sharpness, a dynamic weight allocation strategy combining a learnable mapping network and a hard threshold gating mechanism is constructed. This method can adaptively adjust the weights of each channel feature according to the actual environmental quality of the video frame data, effectively filtering low-quality channels, while preserving the original feature information through residual connections, significantly improving the robustness and accuracy of multi-channel feature fusion in complex environments. Compared with traditional fusion methods based on fixed weights or only feature statistics, this invention achieves a direct correlation between environmental factors and feature quality assessment, making the fusion process more consistent with the dynamically changing environmental scenarios in practical applications. This provides higher-quality input features for downstream video analysis tasks and has significant application value in fields that rely on video understanding. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is an overall framework diagram of a multi-channel feature fusion method based on environment adaptation provided in an embodiment of the present invention;

[0050] Figure 2 A flowchart for environmental perception and gating weight generation provided in an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0052] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”

[0053] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0055] It should be noted that in various specific embodiments of the present invention, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of the present invention require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of the present invention acquired.

[0056] The relevant technologies mainly have the following technical problems:

[0057] Impact of lighting changes: In low-light or backlight scenes, the quality of global appearance features (DINOv3) and local region features is severely degraded, but existing algorithms still fuse them with fixed weights, which leads to the introduction of noise and increases the false detection rate by 15-30%.

[0058] Image blur interference: When the camera is out of focus or people move quickly, causing motion blur, the confidence of the key point-based feature (HRNet) detection decreases, but the algorithm fails to identify and reduce the weight of this channel, resulting in feature vector distortion.

[0059] Occlusion and viewpoint issues: When people are partially occluded or the viewpoint is not frontal, keypoint detection and local ROI features are unreliable. Traditional methods cannot dynamically mask low-quality channels, resulting in fused features containing a large amount of erroneous information.

[0060] Scene complexity changes: In densely populated scenes with many people, the discrimination capabilities of different feature channels vary significantly, but fixed weights cannot adapt to the dynamic adjustment of scene complexity.

[0061] To address the shortcomings of existing technologies, this invention proposes a multi-channel feature fusion method and system based on environment adaptation, the core improvement of which lies in:

[0062] 1. Five-dimensional environmental degradation factor modeling: Establish environmental factor vectors in five dimensions: illumination intensity, illumination uniformity, image sharpness, view integrity, and scene complexity. Through a learnable mapping function, the abstract "environmental severity" is transformed into an interpretable "channel reliability score", realizing an explicit correlation between environmental factors and feature quality.

[0063] 2. Dual weighting mechanism: It innovatively combines environmental perception weights and hard-gated masks. It uses continuous weights generated by Softmax for fine-tuning, and then uses the HardSigmoid gating function to achieve hard screening of low-quality channels, thus avoiding the problem of noise still being retained by soft weighting.

[0064] 3. Environment-driven dynamic fusion framework: It realizes a complete closed loop of "environmental perception → quality assessment → dynamic screening → weighted fusion", organically combining environmental factors, characteristic quality and fusion strategy, and breaking through the limitations of traditional methods of blind and passive environmental fusion.

[0065] 4. End-to-end learnable architecture: The entire process of environment perception, weight generation, gating selection and feature fusion can be embedded into existing deep learning frameworks for end-to-end training. Through joint optimization of triple loss and L2 regularization, the rationality of the weight allocation of each channel is ensured.

[0066] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with the embodiments and accompanying drawings, so as to fully understand the purpose, solution, and effects of the present invention.

[0067] The core idea of ​​this invention is to achieve quality-aware adaptive fusion by explicitly modeling environmental degradation factors and dynamically evaluating the reliability of features in each channel, rather than relying on fixed weights or adaptive methods based solely on feature statistics. Specifically, this invention extracts a five-dimensional environmental factor vector from video frames, generates channel weights through a learnable mapping network, introduces a hard gating mechanism to mask low-quality channels, and finally obtains robust fused features through residual weighted fusion.

[0068] refer to Figure 1 ,like Figure 1 The figure shown illustrates a multi-channel feature fusion method based on environment adaptation provided by an embodiment of the present invention. The method includes the following steps:

[0069] S100, acquire video frame data, extract multi-channel features of the video frame data, with each channel corresponding to a feature type and outputting a feature vector;

[0070] S200, extract environmental degradation factors from the video frame data, the environmental degradation factors including illumination intensity factor, illumination uniformity factor, image sharpness factor, viewpoint quality factor and scene complexity factor;

[0071] S300, the environmental degradation factor is input into a learnable mapping network to generate environmental adaptive weights for each channel;

[0072] S400, apply a hard threshold gating function to the environmental adaptive weights to generate a binary gating mask;

[0073] S500, the environment adaptive weights are multiplied element-wise by the binary gate mask to obtain the final weights of each channel;

[0074] S600, the feature vectors of each channel are weighted and scaled based on the final weights, and the scaled feature vectors are concatenated to obtain a fused feature vector; the fused feature vector is then input into the downstream task model for processing.

[0075] In steps S100 to S600 of the present invention, by using explicit modeling of environmental factors and a hard gating mechanism for dual weighting, quality-aware scaling of each channel is performed before feature splicing, which solves the problems of existing technologies being unable to adapt to environmental changes and low-quality channels introducing noise.

[0076] In some embodiments, in S100, acquiring the multi-channel features of the input data includes:

[0077] Global appearance features were extracted using a DINOv3 pre-trained model;

[0078] The HRNet model is used to extract key point features, and the key point features are encoded using the ResNet-18 model.

[0079] Temporal dynamic features were extracted using the 3DResNet-18 model;

[0080] Local region features are extracted using ROI cropping and then encoded using a ResNet-18 model.

[0081] In this case, the feature vectors output by each channel have the same dimension.

[0082] This embodiment achieves comprehensive feature extraction through a multi-channel parallel architecture: the DINOv3 model, as a global feature extractor, can capture the overall contour, texture, and spatial distribution information of targets in video frames. Its pre-trained weights learn general visual knowledge on large-scale image datasets, providing strong generalization ability for global features; the HRNet model, combined with the ResNet-18 encoder, focuses on key point features. Through the maintenance and fusion of high-resolution feature maps, it accurately encodes the spatial relationships of local key positions such as human joints and object corners. These features are crucial for tasks such as action recognition and pose estimation; the 3DResNet-18 model effectively captures temporal dynamic features such as the motion trend and dynamic change trajectory of targets in video sequences by adding temporal convolution kernels on top of spatial convolution, making up for the lack of temporal information in static frame features; the combination of ROI cropping and ResNet-18 encoding performs refined feature extraction on local areas of interest in video frames, such as cropping specific human areas in multi-person scenes or focusing on local details of objects, enabling this channel to provide targeted local discrimination information. By unifying the dimension of the feature vectors output from each channel, the foundation is laid for subsequent weight scaling and feature concatenation operations, ensuring dimensional compatibility of features from different sources during fusion.

[0083] In some embodiments, S200, extracting environmental degradation factors from the video frame data includes:

[0084] The video frame data is converted from RGB image to YUV color space, and the luminance channel is extracted.

[0085] The global average brightness is calculated based on the brightness channel to obtain the luminance factor.

[0086] The standard deviation of the brightness channel is calculated to obtain the illumination uniformity factor;

[0087] The image sharpness factor is obtained by calculating the image sharpness using the Laplacian operator and obtaining the variance of the Laplacian response.

[0088] The average value of the keypoint confidence output by the keypoint detector is calculated to obtain the viewpoint quality factor.

[0089] The number of detected targets is counted to obtain the scene complexity factor.

[0090] This embodiment constructs a comprehensive evaluation system for video frame quality by extracting multi-dimensional environmental degradation factors. Specifically, the luminance channel is separated through YUV color space conversion, avoiding interference from color information in the RGB channel on illumination evaluation. The global average luminance directly reflects the overall intensity of ambient light; for example, this value decreases significantly in low-light scenes. The standard deviation of the luminance channel quantifies the uniformity of light distribution; in backlit scenes, this value increases due to strong contrast between bright and dark areas. The image sharpness factor is measured by the Laplacian operator's response to image edges and details. The larger the variance of the Laplacian response, the richer the high-frequency details in the image, i.e., the sharper the image. When the camera is out of focus or has motion blur, this variance value decreases significantly. The viewpoint quality factor indirectly evaluates the integrity of the target viewpoint by using the average confidence score output by the keypoint detector. For example, when a person is turned to the side or partially obscured, the average confidence score of keypoint detection decreases, thus reflecting a reduction in viewpoint quality. The scene complexity factor is defined by the number of targets detected in the video frames. The more targets there are, the greater the possibility of mutual occlusion and interference in the scene, and the higher the scene complexity becomes. These five environmental factors characterize the degradation of video frames from multiple aspects such as lighting, sharpness, viewpoint, and scene crowding, providing comprehensive and quantifiable input for subsequent channel reliability assessment.

[0091] In some embodiments, in S300, inputting the environmental degradation factor into a learnable mapping network to generate environmental adaptive weights for each channel includes:

[0092] The environmental factor vector composed of the environmental degradation factors is normalized.

[0093] The normalized environmental factor vector is input into the mapping network, which consists of at least one fully connected hidden layer, a ReLU activation function, and an output layer.

[0094] The output layer is normalized by the Softmax function to generate N-dimensional environment adaptive weights, where N is the number of channels and the sum of all weights is 1.

[0095] This embodiment solves the modeling problem of the complex relationship between environmental factors and feature quality by implementing a mapping network to achieve a nonlinear mapping from environmental factor vectors to channel weights. Through a data-driven mapping approach, the relative reliability of each channel feature under different environmental conditions can be learned more accurately compared to manually designed rules.

[0096] In some embodiments, S400, applying a hard threshold gating function to the environment adaptive weights to generate a binary gating mask includes:

[0097] Calculate the difference between the current channel's environment adaptive weight value and the gating threshold;

[0098] When the difference is less than -3, the binary gate mask is 0; when the difference is greater than 3, the binary gate mask is 1.

[0099] When the difference is between -3 and 3, add 3 to the difference and then divide by 6 to get an intermediate value; take the maximum value of the intermediate value compared with 0, and then take the minimum value of the maximum value compared with 1 to get the binary gating mask.

[0100] This embodiment implements a hard gating mechanism by introducing the HardSigmoid function. When the difference is less than -3, it outputs 0, completely masking low-weight channels; when the difference is greater than 3, it outputs 1, preserving high-quality channels. Within the transition range of -3 to 3, a continuous value between 0 and 1 is generated through linear scaling, avoiding the gradient vanishing problem caused by the step function and ensuring the stability of the training process. The gating threshold is determined through a grid search on the validation set. Different thresholds can be set for different feature channels. For example, a lower threshold (e.g., 0.2) is set for global appearance feature channels that are sensitive to illumination, making them easier to mask when illumination deteriorates, while a higher threshold (e.g., 0.4) is set for local region feature channels that are not sensitive to dynamic changes, maintaining their contribution in complex scenes. In this way, the hard-gating mask can adapt to the environment and dynamically determine the "on / off" state of each channel, effectively filtering low-quality channels with weights below the threshold and preventing their feature noise from contaminating the fusion result.

[0101] In some embodiments, S600, the step of weight scaling of the feature vectors of each channel based on the final weights, and concatenating the scaled feature vectors to obtain a fused feature vector, includes:

[0102] The feature vector of each channel is multiplied by its corresponding final weight to obtain the scaled feature vector;

[0103] The scaled feature vectors of each channel are concatenated according to the channel dimension to obtain a preliminary fused feature vector;

[0104] The initial fused feature vector is concatenated with the original feature vectors of each channel to obtain the original concatenated feature vector. The residuals are added according to the trainable coefficients to obtain the fused feature vector for downstream tasks. The trainable coefficients are variable weights that are learned and adjusted through backpropagation.

[0105] This embodiment further enhances the robustness of the fused features through a residual summation mechanism. Specifically, the original stitched features serve as a baseline, preserving the original information of all channels, while the preliminary fused features are optimized features after environmental adaptive filtering and weighting. By introducing trainable coefficients, the model can dynamically adjust the contribution ratio of the two during training: when the environment is stable and the quality of each channel is high, the trainable coefficients can be tilted towards the preliminary fused features, fully leveraging the advantages of dynamic weighting; while when extreme environments cause most channels to fail, the trainable coefficients can increase the weight of the original stitched features, avoiding over-reliance on a single low-quality channel. This residual design allows the fused features to utilize the environment-aware dynamic optimization capability while preserving basic information in complex environments, preventing the loss of feature information due to over-filtering. For example, when strong direct light causes severe overexposure of the global appearance feature (DINOv3), the preliminary fused features may significantly reduce the weight of that channel, while the original stitched features still retain some contour information of that channel. After residual summation, while suppressing overexposure noise, it can retain basic contour cues useful for downstream tasks, thereby improving the overall discriminative ability of the fused features.

[0106] In some embodiments, the method further includes a training step, wherein the total loss during training is obtained in the following manner:

[0107] Calculate the square of the Euclidean distance between the anchor feature mapping result and the positive sample feature mapping result, subtract the square of the Euclidean distance between the anchor feature mapping result and the negative sample feature mapping result, add a boundary parameter, and then take the maximum value between the result and 0 to obtain the triplet loss.

[0108] The L2 regularization term is obtained by multiplying the square of the Euclidean distance between the environment adaptive weight vector and a uniformly distributed weight vector by a regularization coefficient.

[0109] The sum of the triplet loss and the L2 regularization term is taken as the total loss.

[0110] This embodiment achieves dual constraints on feature discriminativeness and the rationality of weight distribution through joint optimization of triplet loss and L2 regularization. This allows the model to simultaneously optimize the balance of feature discriminativeness and weight distribution during training, ultimately achieving reasonable allocation of channel weights and robustness of fused features. Specifically, triplet loss effectively improves the discriminative ability of fused features in downstream classification or retrieval tasks by narrowing the distance between the feature mapping results of similar samples (positive samples) and the anchor feature mapping results, widening the distance between the feature mapping results of dissimilar samples (negative samples) and the anchor feature mapping results, and by introducing boundary parameters to ensure inter-class separation.

[0111] refer to Figure 2 The following is a specific embodiment provided by the present invention:

[0112] I. Model Building:

[0113] Step 1: Multi-channel feature extraction;

[0114] Taking the multi-channel abnormal behavior detection algorithm for secure rooms as an example, the algorithm obtains multi-channel embedding vectors, including but not limited to the following channels:

[0115]

[0116] The final fused feature dimension is: ;

[0117] Among them, global appearance features Extracted using the DINOv3 pre-trained model, it can capture the overall appearance information of the target; key point features HRNet was used to detect 17 human body keypoints, which were then encoded into high-level semantic features using ResNet-18; temporal dynamic features. 3DResNet-18 was used to process 16 consecutive frames of images to model motion patterns and local anomaly features. ROI clipping is performed on key areas such as the hands, shoulders, and waist, and fine-grained anomaly cues are extracted using ResNet18.

[0118] Step 2: Environmental factor perception and adaptive weight generation;

[0119] Extract environmental description information from video frames and automatically generate channel weights adapted to the current environment:

[0120] Convert the RGB image to the YUV color space and extract the luminance channel Y:

[0121] ;

[0122] in, For pixel coordinates, , , These are the red, green, and blue channel values ​​of the pixel, respectively. This represents the pixel value of the luminance channel.

[0123] Illuminance Factor : Calculate the global average brightness;

[0124] ;

[0125] in, The illuminance factor, , The lower the value, the dimmer the light. , These represent the image height and width, respectively.

[0126] Illumination uniformity factor : Calculate the standard deviation of the brightness channel to measure the uniformity of light distribution;

[0127] ;

[0128] in, This is the illumination uniformity factor. , A larger value indicates a more uneven distribution of light (such as strong backlighting or localized shadows).

[0129] Image sharpness factor Use the Laplacian operator to calculate image sharpness and assess whether the image is blurry.

[0130] ;

[0131] in, , , , , These represent the pixel values ​​of the image at position (x, y) and its four neighboring regions (up, down, left, and right), respectively. This is the Laplace response, i.e., the second derivative response value calculated by the Laplace operator.

[0132] Image sharpness factor Sharpness is defined as the variance of the Laplace response.

[0133] ;

[0134] in, Image sharpness factor The higher the value, the clearer the image; the lower the value, the blurrier the image.

[0135] Viewpoint quality factor : Evaluation of viewpoint quality based on confidence level of keypoint detection. Assume the keypoint detector outputs K keypoints, and the confidence level of each keypoint is... ;

[0136] ;

[0137] in, For the perspective quality factor, A higher value indicates that the detected target is more complete, has less occlusion, and that the key points are reliable. Let be the detection confidence level of the k-th keypoint.

[0138] Scene complexity factor Count the number of targets detected:

[0139] ;

[0140] in This reflects the number and complexity of people in the scene;

[0141] Environmental factor vector normalization:

[0142] ;

[0143] ;

[0144] in, and This is a vector of the mean and standard deviation of the training set statistics; is the original environmental factor vector, and c is the standardized five-dimensional environmental degradation factor vector, with the values ​​of each dimension distributed in the interval [-1,1], which facilitates subsequent mapping function processing.

[0145] Environment-adaptive weight generation: through a learnable mapping network Environmental factors are mapped to channel weights to learn the degree of influence of different environmental factors on each channel.

[0146] ;

[0147] Among them, mapping network It consists of at least one fully connected hidden layer, a ReLU activation function, and an output layer. The output layer is normalized using Softmax to generate N-dimensional environment-adaptive weights. ,and , ;

[0148] Through multi-layer nonlinear transformations, the mapping network can learn the complex nonlinear relationship between environmental factors and the reliability of each channel. For example, under low light conditions ( (Low value), the mapping network can automatically reduce the dependence on appearance features. The weights; when the image is blurred ( (Low value), reducing the impact on key point features Dependence; in high occlusion scenarios ( (Low values) suppress the influence of keypoint channels. This learning mechanism enables the model to make optimal weight allocation decisions for different environmental degradation modes.

[0149] Step 4: Gating selection mechanism;

[0150] Apply a hard threshold gating function to the environmental weights, and use the HardSigmoid function to convert the continuous log-odds ratios. Convert to binary gated mask .

[0151] ;

[0152] in, Let θ be the environment adaptive weight value for the i-th channel, and θ be the preset gating threshold. Let be the gating mask for the i-th channel.

[0153] Threshold determination logic:

[0154] when hour: ;

[0155] when hour: ;

[0156] Middle area: smooth transition;

[0157] in, Determined by the validation set. This example is set up... ,Right now At that time, it was considered that the channel was of poor quality and was completely shielded. At the same time, the problem of soft weighting still retaining low-quality channels was avoided, and noise introduction was reduced.

[0158] Will and The final weights are obtained by multiplying each element:

[0159] ;

[0160] in, , , These are the final weight vector, the environment-adaptive weight vector, and the hard-gated mask vector, respectively. ⊙ represents the element-wise multiplication operation.

[0161] This hard gating mechanism allows for the complete blocking of a channel when its weight falls below a threshold. Instead of simply reducing its influence with a smaller weight, the channel is completely blocked, effectively "hard-deleting" it. This mechanism is particularly important in extreme degradation scenarios, such as when a target is completely occluded, causing keypoint detection to fail (c_4≈0). Even a small weight can introduce significant noise, and hard gating can completely block the channel.

[0162] Step 5: Scaling and concatenating the weights;

[0163] Weight scaling is applied to the feature vectors of each channel:

[0164] ;

[0165] in, , , , These are the final weights for channels 1, 2, 3, and 4, respectively. , , , These are the feature vectors for each channel. To fuse feature vectors.

[0166] To preserve the original feature information, a residual path is added. With original splicing features Based on trainable coefficients The residuals are added together to obtain the final feature. This is used for downstream tasks. Here, λ is a variable weight that can be adjusted through backpropagation. Residual connections ensure that the model can still maintain basic performance using the original spliced ​​features even when environmental factor extraction fails or the mapping network is not sufficiently trained, thus enhancing training stability.

[0167] II. Loss Function;

[0168] The main loss function is consistent with the original multi-channel feature fusion model. In this example, triplet loss is used as the main body of the loss function.

[0169] ;

[0170] Where f(·) represents the feature mapping function, which is the output of the multi-channel feature fusion described above; , , These are the mapping results for anchor features, positive sample features, and negative sample features, respectively; margin is the boundary parameter, which is set to 0.5 in this example; This represents the square of the Euclidean distance (simplified to Euclidean distance).

[0171] The total loss is optimized using a combination of triplet loss and L2 regularization to avoid excessive weight concentration on a single channel.

[0172] ;

[0173] in, For the total loss, , .

[0174] L2 regularization forces the environment-adaptive weight distribution to be close to a uniform distribution, preventing the model from over-relying on a single channel in some simple scenarios and losing its ability to learn other channels, thus ensuring that each channel is fully optimized during training.

[0175] III. Training methods;

[0176] The training is divided into three phases:

[0177] Phase 1 (Epoch 1~20) freezes the gating mechanism (g=[1,1,1,1], all channels are open), and only trains the FC network of the environment-aware module; this phase enables the mapping network to learn the relationship between environmental factors and channel quality in the initial stage, without being affected by gating.

[0178] Phase 2 (Epoch 21~50): Unfreeze the gating mechanism and gradually reduce the gating threshold θ. In this example, θ decreases linearly from 0.30 to 0.15. Gradually reducing the threshold allows the model to smoothly transition to the hard gating mechanism and avoids training oscillations.

[0179] Phase 3 (Epoch 51~80): A fixed threshold θ=0.15 is used, allowing for full adjustment of environmental weights and gating parameters. This phase involves overall fine-tuning to optimize final performance.

[0180] Compared with related technologies, the present invention has the following advantages:

[0181] 1. Solved the core problem of multi-channel fusion failure in environmental degradation scenarios: Through explicit modeling of five-dimensional environmental factors, it can accurately identify degradation conditions such as insufficient lighting, image blurring, and target occlusion, and dynamically adjust the weight of each channel, reducing the false detection rate by 15-30% compared with traditional methods.

[0182] 2. Improved robustness and stability of the system in real-world scenarios: The hard gating mechanism can completely shield failed channels and avoid noise introduced by low-quality features. In practical applications such as abnormal behavior detection in secure rooms, the false negative rate is reduced to below 1%.

[0183] 3. The explicit environmental factor modeling and weight allocation process provides a clear basis for decision-making: compared with black-box methods, it is easier for users to understand and accept, meets the transparency requirements of sensitive fields such as security and examinations, and facilitates system debugging and troubleshooting.

[0184] 4. End-to-end trainable with low computational overhead: The number of parameters in the mapping network and gating function accounts for only 0.5% of the overall model, and the time increase for a single forward propagation during inference is less than 2ms, making it suitable for real-time video analysis scenarios.

[0185] This invention also provides an environment-adaptive multi-channel feature fusion system, comprising:

[0186] At least one processor;

[0187] At least one memory for storing at least one program;

[0188] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0189] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0190] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above. This electronic device can be any intelligent terminal, including servers, edge computing devices, smart cameras, etc.

[0191] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0192] Those skilled in the art will understand that all or some of the steps and systems disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and can include any information delivery medium.

[0193] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

Claims

1. A multi-channel feature fusion method based on environment adaptation, characterized in that, The method includes the following steps: Acquire video frame data, extract multi-channel features from the video frame data, with each channel corresponding to a feature type and output a feature vector; Environmental degradation factors are extracted from the video frame data. These environmental degradation factors include illumination intensity factor, illumination uniformity factor, image sharpness factor, viewpoint quality factor, and scene complexity factor. The environmental degradation factor is input into a learnable mapping network to generate environmental adaptive weights for each channel; A hard threshold gating function is applied to the environmental adaptive weights to generate a binary gating mask; The environment-adaptive weights are multiplied element-wise by the binary gate mask to obtain the final weights of each channel; Based on the final weights, the feature vectors of each channel are weighted and scaled, and the scaled feature vectors are concatenated to obtain a fused feature vector; the fused feature vector is then input into the downstream task model for processing.

2. The method according to claim 1, characterized in that, The multi-channel features of the input data to be acquired include: Global appearance features were extracted using a DINOv3 pre-trained model; The HRNet model is used to extract key point features, and the key point features are encoded using the ResNet-18 model. Temporal dynamic features were extracted using the 3DResNet-18 model; Local region features are extracted using ROI cropping and then encoded using a ResNet-18 model. In this case, the feature vectors output by each channel have the same dimension.

3. The method according to claim 1, characterized in that, The step of extracting environmental degradation factors from the video frame data includes: The video frame data is converted from RGB image to YUV color space, and the luminance channel is extracted. The global average brightness is calculated based on the brightness channel to obtain the luminance factor. The standard deviation of the brightness channel is calculated to obtain the illumination uniformity factor; The image sharpness factor is obtained by calculating the image sharpness using the Laplacian operator and obtaining the variance of the Laplacian response. The average value of the keypoint confidence output by the keypoint detector is calculated to obtain the viewpoint quality factor. The number of detected targets is counted to obtain the scene complexity factor.

4. The method according to claim 1, characterized in that, The step of inputting the environmental degradation factor into a learnable mapping network to generate environmentally adaptive weights for each channel includes: The environmental factor vector composed of the environmental degradation factors is normalized. The normalized environmental factor vector is input into the mapping network, which consists of at least one fully connected hidden layer, a ReLU activation function, and an output layer. The output layer is normalized by the Softmax function to generate N-dimensional environment adaptive weights, where N is the number of channels and the sum of all weights is 1.

5. The method according to claim 1, characterized in that, The step of applying a hard threshold gating function to the environment adaptive weights to generate a binary gating mask includes: Calculate the difference between the current channel's environment adaptive weight value and the gating threshold; When the difference is less than -3, the binary gate mask is 0; when the difference is greater than 3, the binary gate mask is 1. When the difference is between -3 and 3, add 3 to the difference and then divide by 6 to get an intermediate value; take the maximum value of the intermediate value compared with 0, and then take the minimum value of the maximum value compared with 1 to get the binary gating mask.

6. The method according to claim 1, characterized in that, The step of scaling the feature vectors of each channel based on the final weights and concatenating the scaled feature vectors to obtain a fused feature vector includes: The feature vector of each channel is multiplied by its corresponding final weight to obtain the scaled feature vector; The scaled feature vectors of each channel are concatenated according to the channel dimension to obtain a preliminary fused feature vector; The initial fused feature vector is concatenated with the original feature vectors of each channel to obtain the original concatenated feature vector. The residuals are added according to the trainable coefficients to obtain the fused feature vector for downstream tasks. The trainable coefficients are variable weights that are learned and adjusted through backpropagation.

7. The method according to claim 1, characterized in that, The method also includes a training step, wherein the total training loss is obtained in the following manner: Calculate the square of the Euclidean distance between the anchor feature mapping result and the positive sample feature mapping result, subtract the square of the Euclidean distance between the anchor feature mapping result and the negative sample feature mapping result, add a boundary parameter, and then take the maximum value between the result and 0 to obtain the triplet loss. The L2 regularization term is obtained by multiplying the square of the Euclidean distance between the environment adaptive weight vector and a uniformly distributed weight vector by a regularization coefficient. The sum of the triplet loss and the L2 regularization term is taken as the total loss.

8. A multi-channel feature fusion system based on environment adaptation, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images