A camera blocking state detection method

Abstract

The camera occlusion detection method disclosed in this invention expands the 3×3 convolution into a multi-branch structure during the training phase, increasing the model's nonlinearity and feature representation ability. During the detection phase, reparameterization is used to fuse the multi-branch structure into an equivalent convolutional layer during inference, achieving model lightweighting while maintaining detection accuracy, thereby improving the accuracy of camera occlusion detection. The method divides the feature maps of edge color texture, inter-channel information exchange, and original image features into two parts according to the number of feature channels using a trained C2f-Star module. One part undergoes multiple feature transformations through multiple Faster-Star modules to obtain a feature map reflecting the essence of the occlusion. The other part obtains the initial feature maps of edge color texture, inter-channel information exchange, and original image features through a residual path. These are used to calculate the occlusion score for each frame, thereby determining the camera's occlusion status.

Description

Technical Field

[0001] This invention relates to the field of camera occlusion detection, and in particular to a method for detecting camera occlusion status. Background Technology

[0002] With the continuous development of computer vision technology, image acquisition devices have begun to penetrate all aspects of daily life and production. In systems such as security monitoring, Advanced Driving Assistance Systems (ADAS), Driver Monitor Systems (DMS), and dashcams, cameras, as key visual sensors, are responsible for providing real-time environmental images to support the normal operation and decision-making of the system. However, in practical use, cameras are often obstructed by external objects (e.g., dust, hands, leaves, clothing), preventing the device from capturing the true scene, thus affecting its normal functioning and creating safety hazards. Therefore, camera occlusion detection has significant practical importance.

[0003] Currently, methods for camera occlusion detection mainly rely on color histograms or texture analysis to determine the similarity between consecutive frames. For example, in patent application CN201911114424X, entitled "A Method for Occlusion Detection in Dashcam Video," the video frame is divided into multiple sub-blocks. The Marr-Hildreth edge operator is used to calculate the number of texture edges in each sub-block. When the number of edges is lower than a preset threshold, the sub-block is determined to be occluded. Furthermore, SSIM (Simultaneous Similarity Detection) is used to determine the similarity between consecutive frames to ascertain whether the video is continuously occluded. These methods often perform poorly in handling dynamic scenes with varying lighting, noise interference, and complex backgrounds, exhibiting low robustness. Additionally, edge detection and SSIM calculation can be time-consuming for each frame, especially in high-resolution and high-frame-rate videos, potentially impacting real-time performance. In addition, some camera occlusion detection methods mainly employ deep learning. To improve detection performance, they employ relatively complex models and strategies. For example, in patent application CN2021107937477, entitled "A Deep Learning-Based Method for Occlusion Detection of Surveillance Video Images," the proportion of obstacle colors in the color histogram is used for initial judgment. If the proportion exceeds a certain threshold, YOLOv5 is used for object detection to detect the occluded portion. Patent application CN2022117187838, entitled "A Camera Occlusion Detection Method, Device, Storage Medium, and Electronic Device," transforms occlusion detection into a classification and semantic segmentation task. These algorithms offer significantly improved accuracy compared to traditional methods, but they are generally more complex and lack real-time performance and lightweight design.

[0004] Furthermore, many current works adopt an end-to-end training approach, which only supervises the output of the last layer of the model. In this approach, the model relies on the feedback information of the final layer to adjust the previous feature extraction layers one by one, which does not make full use of the features at different levels in the model and ignores the contribution of intermediate layers to specific targets (such as local details and edge information), resulting in false detections.

[0005] In practical applications, there are many types of occlusion, and the occlusion scenarios are diverse. Some non-occluded scenarios can easily be confused with the characteristics of occluded scenarios (e.g., sky, open road surface, etc.), which poses a challenge for data collection and processing. Summary of the Invention

[0006] The purpose of this invention is to provide a camera occlusion detection method that can fully utilize features at different levels in the model and has real-time performance, thereby improving the accuracy of detection.

[0007] The camera occlusion detection method of the present invention includes the following steps:

[0008] S1. Training phase: Image features with and without occlusions are divided into three paths in the feature extraction Rep-Conv module. One path is passed through a 3×3 convolution and batch normalization layer, and then through weights and bias parameters to obtain the convolution output features corresponding to this branch. Another path is passed through a 1×1 convolution and batch normalization layer, and then through weights and bias parameters to obtain the convolution output features corresponding to this branch. The third path is passed through an identity transformation to obtain the output features corresponding to this branch. The above three paths are fused to obtain an equivalent convolution kernel that can recognize the target edge color texture and original image features.

[0009] S2. After training the Rep-Conv module, pass its output features to the C2f-Star module to train the C2f-Star module, so as to obtain a C2f-Star module with the ability to recognize occluded and unoccluded objects and combine information from different dimensions.

[0010] S3. After completing the training of the C2f-Star module, pass its output features to the attention module and train the attention module to obtain an attention module with the ability to identify key feature information of occluded objects.

[0011] S4. After training the attention module, pass its output features to the final classifier and train the final classifier to obtain a final classifier that can identify the categories of occluded and unoccluded objects.

[0012] S5. Detection stage: Each frame of the acquired image is input into the trained equivalent convolution kernel for convolution calculation to obtain a feature map of the original image that combines the edge color texture of occluded and non-occluded objects, information exchange between channels;

[0013] S6. Divide the feature map of edge color texture, information exchange between channels, and original image features into two parts according to the number of feature channels in the trained C2f-Star module. One part undergoes multiple feature transformations through multiple Faster-Star modules to obtain a feature map that reflects the essence of the occluded object. The other part obtains the initial feature map of edge color texture, information exchange between channels, and original image features of occluded and non-occluded objects through the residual path.

[0014] S7. The feature map reflecting the essence of the occluded object and the initial edge color texture of the occluded and non-occluded objects are combined and the information is exchanged between channels. The feature map of the original image is integrated and stitched through channels, and then the number of channels is adjusted to output the feature map of the occluded and non-occluded objects that combines information from different dimensions.

[0015] S8. Input the feature map that integrates information from occluded and unoccluded objects in different dimensions into the attention module. By weighting and adjusting each channel, the key features of the occluded object are highlighted, and a feature map with key feature information of the occluded object is obtained.

[0016] S9. The final classifier calculates the occlusion score of each frame image using the feature map of key feature information of the occluder, and determines whether the occlusion score is greater than the occlusion threshold. If yes, the camera in that frame image is in an occluded state; otherwise, the camera in that frame image is in an unoccluded state.

[0017] The camera occlusion detection method described in this invention expands the 3×3 convolution into a multi-branch structure during the training phase, increasing the model's nonlinearity and feature representation capability. During the detection phase, reparameterization is used to fuse the multi-branch structure into an equivalent convolutional layer during inference. This achieves lightweight and real-time performance while maintaining detection accuracy, thereby improving the accuracy of camera occlusion detection. Furthermore, the method combines edge color texture, inter-channel information exchange, and the feature map of the original image, dividing it into two parts based on the number of feature channels within the trained C2f-Star module. One part undergoes multiple feature processing sessions using multiple Faster-Star modules. The transformation yields a feature map reflecting the essence of the occlusion. Another part is processed through a residual path to obtain the initial edge color texture. Information is exchanged between channels, and the feature map of the original image is used to calculate the occlusion score of each frame, thereby determining the occlusion state of the camera. This allows for further full utilization of features at different levels in the model, improving the accuracy of camera occlusion state detection. In addition, by weighting the feature maps that combine information from different dimensions through the attention module to obtain feature maps of key features of the occlusion, the model can focus on key features and suppress irrelevant information, thereby improving the discriminative and generalization abilities of camera occlusion state detection.

[0018] As a preferred embodiment of the present invention, in step S1, the convolutional output features corresponding to this branch are obtained through a 3×3 convolution and batch normalization layer, using weights and bias parameters. The specific design is as follows: y 3×3 =W 3×3 *x, where W 3×3 These are the weights of a 3x3 convolution, where x is the input image feature with and without occlusions, and y is the weights. 3×3 This is the feature map obtained after convolution; subsequently, the features obtained from convolution are adjusted by a batch normalization layer, and the output is: in, It is the feature map of the output edge color texture, μ 3×3 , b 3×3 It represents the mean, variance, and bias parameters of the current feature map, γ. 3×3 and β 3×3 These are the learning parameters for the batch normalized layer.

[0019] As a preferred embodiment of the present invention, in step S1, the convolution output features corresponding to the branch are obtained through a 1×1 convolution and a batch normalization layer with weights and bias parameters. The specific design is as follows: the 1x1 convolution is padded and expanded into a 3x3 feature form. These are the weights of a 1×1 convolution; the calculated features after padding are as follows: Where x is the input image feature with and without occlusions, and y1×1 The output feature map is obtained through a 1x1 convolution; subsequently, the features obtained from the convolution are adjusted by a batch normalization layer, and the output is: in, It is y 1×1 The feature map, μ, after adjustment by the normalization layer, enables information exchange between feature channels. 1×1 , b 1×1 , where are the mean, variance, and bias parameters of the current feature map, and γ is the bias parameter. 1×1 and β 1×1 These are the learning parameters for the batch normalized layer.

[0020] As a preferred embodiment of the present invention, the output features corresponding to the branch are obtained by identity transformation in step S1. The specific design is as follows: the convolution kernel of one identity matrix is ​​expanded into a 3x3 feature form. These are the weights of the identity transformation; the calculated features after filling are as follows: Where x is the input image feature with and without occlusions, and y identity This is the feature map output after identity transformation; subsequently, after adjustment by the batch normalization layer, the output is: in, It is the feature map of the original image features output, μ identity , b identity , where are the mean, variance, and bias parameters of the current feature map, and γ is the bias parameter. identity and β identity These are the learning parameters for the batch normalized layer.

[0021] As a preferred embodiment of the present invention, multiple Rep-Conv modules and C2f-Star modules are provided. Two Rep-Conv modules are set near the input end of the image features, and the remaining Rep-Conv modules and C2f-Star modules are set at intervals and connected in series. The C2f-Star module is used as the final output of the image features. The Rep-Conv modules and C2f-Star modules form feature extraction modules at different levels.

[0022] When training the Rep-Conv module and the C2f-Star module, the first C2f-Star module, the second C2f-Star module, and the third C2f-Star module following the Rep-Conv module are connected to the first auxiliary classifier, the second auxiliary classifier, and the third auxiliary classifier, respectively.

[0023] The occlusion score and non-occlusion probability score calculated by the final classifier are compared with the occlusion label and non-occlusion label corresponding to the original image to calculate the first difference value using the loss function;

[0024] The first auxiliary classifier calculates the occlusion score and non-occlusion score of each frame image using the feature map of key feature information of occluders in this stage;

[0025] The second auxiliary classifier calculates the occlusion score and non-occlusion score of each frame image using the feature map of key feature information of occluders in this stage.

[0026] The third auxiliary classifier calculates the occlusion score and non-occlusion score of each frame image using the feature map of key feature information of occluders in this stage;

[0027] The occlusion scores and non-occlusion probability scores calculated by the first, second, and third auxiliary classifiers are compared with the occlusion and non-occlusion labels of the original image using a loss function to calculate the corresponding difference values.

[0028] The occlusion scores and non-occlusion scores calculated by the first, second, and third auxiliary classifiers are compared with the occlusion scores and non-occlusion scores calculated by the final classifier using the loss function to calculate the corresponding difference values.

[0029] The final difference value is calculated by weighting and combining all the difference values.

[0030] Based on the final difference value, gradient backpropagation is performed to update the weights and bias parameters of the Rep-Conv module, C2f-Star module, attention module, and each classifier. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the structure during the training phase of the camera occlusion detection method of the present invention;

[0032] Figure 2 A diagram illustrating the three-way training of the Rep-Conv module;

[0033] Figure 3 This is a schematic diagram of the detection stage of the camera occlusion detection method of the present invention;

[0034] Figure 4 This is a flowchart illustrating the camera occlusion detection method of the present invention;

[0035] Figure 5 A flowchart illustrating multiple feature transformations for the C2f-Star module;

[0036] Figure 6 This is a schematic diagram showing the use of various types of obstructions. Detailed Implementation

[0037] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0038] A schematic diagram of the training phase structure for camera occlusion detection is shown below. Figure 1 As shown, the image features with and without occlusions are mainly processed by the Rep-Conv (reparameterized) module, which divides the processing into three paths. The feature maps of edge color texture, channel exchange, and original image features are mainly processed by the C2f-Star module, which divides them into two parts based on the number of feature channels. Multiple Rep-Conv and C2f-Star modules are used. Two Rep-Conv modules are placed near the input of the image features, while the remaining Rep-Conv modules are spaced apart from the C2f-Star modules and connected in series. The C2f-Star module serves as the final output of the image features. The Rep-Conv and C2f-Star modules form different levels of feature extraction modules. The output of the feature extraction module is connected to the attention module, and the output of the attention module is connected to the final classifier (fully connected layer). Furthermore, during training of the Rep-Conv and C2f-Star modules, the first, second, and third C2f-Star modules following the Rep-Conv module are connected to the first, second, and third auxiliary classifiers, respectively.

[0039] A method for detecting camera occlusion, such as Figure 3 As shown, in the training phase (S1), the image features with and without occlusions are divided into three paths in the feature extraction Rep-Conv module. One path passes through a 3×3 convolution and batch normalization layer with weights and bias parameters to obtain the convolution output features corresponding to that branch. Another path passes through a 1×1 convolution and batch normalization layer with weights and bias parameters to obtain the convolution output features corresponding to that branch. The third path undergoes an identity transformation to obtain the output features corresponding to that branch. The above three paths are fused to obtain an equivalent convolution kernel that can recognize the target edge color texture and original image features.

[0040] like Figure 2 As shown, the convolutional output features corresponding to this branch are obtained through a 3×3 convolution and batch normalization layer, using weights and bias parameters. The specific design is as follows: y 3×3 =W 3×3 *x, where W 3×3These are the weights of a 3x3 convolution, where x is the input image feature with and without occlusions, and y is the weights. 3×3 This is the feature map obtained after convolution; subsequently, the features obtained from convolution are adjusted by a batch normalization layer, and the output is: in, It is the feature map of the output edge color texture, μ 3×3 , b 3×3 It represents the mean, variance, and bias parameters of the current feature map, γ. 3×3 and β 3×3 These are the learning parameters for the batch normalized layer.

[0041] The convolution output features corresponding to this branch are obtained through 1×1 convolution and batch normalization layers with weights and bias parameters. The specific design is as follows: the 1x1 convolution is padded and expanded into a 3x3 feature form. These are the weights of a 1×1 convolution; the calculated features after padding are as follows: Where x is the input image feature with and without occlusions, and y 1×1 The output feature map is obtained through a 1x1 convolution; subsequently, the features obtained from the convolution are adjusted by a batch normalization layer, and the output is:

[0042] in, It is y 1×1 The feature map, μ, after adjustment by the normalization layer, enables information exchange between feature channels. 1×1 , b 1×1 , where are the mean, variance, and bias parameters of the current feature map, and γ is the bias parameter. 1×1 and β 1×1 These are the learning parameters for the batch normalized layer.

[0043] The output features corresponding to this branch are obtained through identity transformation. The specific design is as follows: the convolution kernel of a single identity matrix is ​​padded and expanded into a 3x3 feature form. These are the weights of the identity transformation; the calculated features after filling are as follows: Where x is the input image feature with and without occlusions, and y identity This is the feature map output after identity transformation; subsequently, after adjustment by the batch normalization layer, the output is: in, It is the feature map of the original image features output, μ identity , b identity , where are the mean, variance, and bias parameters of the current feature map, and γ is the bias parameter. identity and β identityThese are the learning parameters for the batch normalized layer.

[0044] The weights and bias parameters of the three branches are added together to obtain the fused convolutional kernel and bias parameters. This achieves the fusion of the 3x3 convolution, 1x1 convolution, and identity transformation branches from the training phase into an equivalent convolutional kernel with weights and bias parameters that can identify the edge color and texture features of occluded and unoccluded objects as well as the original image features. This maintains the good learning ability of the multi-branch structure for complex feature distributions while keeping the model structure lightweight. The formula for fusing the three branches is as follows: The weights and bias parameters of the equivalent convolution kernel can be expressed by the following formula:

[0045]

[0046] The output of the fused equivalent convolution kernel is: y fused =W fused *x+b fused Among them, W fused b represents the weights of the equivalent convolution kernel. fused The bias parameters are those of the equivalent convolution kernel.

[0047] S2. After training the Rep-Conv module, pass its output features to the C2f-Star module to train the C2f-Star module, so as to obtain a C2f-Star module with the ability to recognize occluded and unoccluded objects and combine information from different dimensions.

[0048] S3. After completing the training of the C2f-Star module, pass its output features to the attention module and train the attention module to obtain an attention module with the ability to recognize key feature information of occluded objects.

[0049] S4. After training the attention module, pass its output features to the final classifier and train the final classifier to obtain a final classifier that can identify the categories of occluded and unoccluded objects.

[0050] During training of the Rep-Conv and C2f-Star modules, the first, second, and third C2f-Star modules following the Rep-Conv module are connected to the first, second, and third auxiliary classifiers, respectively. The occlusion score and non-occlusion probability score calculated by the final classifier are compared with the occlusion and non-occlusion labels corresponding to the original image using a loss function to calculate the first difference value. The first auxiliary classifier calculates the occlusion score and non-occlusion score for each frame using the key feature map of the occluded objects at this stage. The second auxiliary classifier calculates the occlusion score and non-occlusion score for each frame using the key feature map of the occluded objects at this stage. The third auxiliary classifier calculates the occlusion score and non-occlusion score for each frame using the key feature map of the occluded objects at this stage. The key feature information of the occluded objects in each stage is used to calculate the occlusion score and non-occlusion score of each frame of the image. The occlusion scores and non-occlusion scores calculated by the first, second, and third auxiliary classifiers are compared with the occlusion and non-occlusion labels of the original image using a loss function to calculate the corresponding difference values. The occlusion scores and non-occlusion scores calculated by the first, second, and third auxiliary classifiers are compared with the occlusion scores and non-occlusion scores calculated by the final classifier using a loss function to calculate the corresponding difference values. All difference values ​​are weighted and combined to calculate the final difference value. Based on the final difference value, gradient backpropagation is performed to update the weights and bias parameters of the Rep-Conv module, C2f-Star module, attention module, and each classifier.

[0051] The loss function consists of two parts: the supervised loss of the final classifier and the supervised loss of the auxiliary classifier. The loss function is defined as follows: Among them, L final It is the supervision loss of the final classifier and the ground truth labels (occluded and unoccluded labels corresponding to the original image); Let be the supervisory loss of the i auxiliary classifiers, used to guide feature learning in the intermediate layers of the feature extraction module; α is an adjustment parameter that controls the proportion of the supervisory loss in the total loss; N is the number of auxiliary classification heads. The final classifier's loss function is expressed as follows:

[0052]

[0053] Among them, L ce It is cross-entropy, p final This is the model's final classification output (occlusion score and non-occlusion probability score), and y is the true label. By minimizing this loss, the model learns the correct classification decision.

[0054] The loss function of the auxiliary classifier consists of two parts: one part is the supervision from the true labels, and the other part is the prediction distribution of the classifier in S4 as the supervision from the soft labels. The formula is as follows: Where, p i y is the predicted probability distribution output by the i-th auxiliary classifier (there are three in total); y is the true label, which is a one-hot distribution. β is the soft label output by the final classifier, represented as a probability distribution; β is an adjustment parameter that balances the weights of real label supervision and soft label supervision; L ce (p i (y) is the cross-entropy loss, used to constrain the consistency between the auxiliary classifier's predicted distribution and the true label. This is the KL divergence, which measures the difference between the predicted distribution and the soft label. The formula for KL divergence is as follows: In the actual prediction phase of the model, no auxiliary classifier is used. Instead, the end-to-end model structure is used directly to output the prediction results, thereby ensuring the model's lightweight nature and real-time performance.

[0055] By adding an auxiliary classifier after the outputs of the three C2f-Star modules in the model backbone (corresponding to 8x, 16x, and 32x downsampling), respectively, to output the prediction results of features at different downsampling ratios, and using the ground truth labels as supervision, the intermediate layers of the model can extract more detailed and hierarchical features during training. In particular, the 8x and 16x downsampling layers can typically capture the edges and occlusion features of occluded objects, while the 32x downsampling layer often contains richer global information, making it more suitable for capturing overall contours and scene features. This hierarchical supervision design improves the model's adaptability to different degrees of occlusion and features at different scales, effectively enhancing the accuracy of occlusion detection. Furthermore, this training method can enhance the gradient flow in intermediate layers, effectively alleviating the gradient vanishing problem when the network has deep layers.

[0056] Furthermore, by introducing the final classifier's prediction distribution as the output of the teacher model to guide the auxiliary classifier's prediction distribution, the auxiliary classifier can better learn features that align with the final task objective. This approach not only balances the optimization directions of the auxiliary and final classifiers but also enhances the coordination of global features, further improving the model's generalization ability. Moreover, the model's final classifier still uses the real labels for optimization, ensuring overall performance for the final task.

[0057] For the collection of positive samples (obstacles), this invention uses various types of obstacles, including towels and rags, leaves, paper, plastic bags, water bottles, limbs, small items hanging in vehicles, etc. Figure 6As shown. The acquisition process is as follows: 1. Use various obstructions (including direct hand covering) to obstruct the lens from all directions (up, down, left, and right). Perform 3 obstructions in each direction, with each obstruction lasting approximately 3 seconds. 80%–90% of the image needs to be obstructed to be considered obstructed, but be careful not to obscure the image to the point of complete darkness, as this will prevent the acquisition of any learnable features. 2. After acquiring the fixed obstructions, use moving obstructions (simulating the relative movement between the obstruction and the lens). Specifically, slowly move the obstruction left and right and up and down in front of the lens. A single translation is considered as the image moving from unobstructed state → completely obstructed → unobstructed. Perform two translations in total, one from left to right and one from top to bottom, with each translation lasting approximately 2–3 seconds.

[0058] For the collection of negative samples (non-occluded objects), this invention divides them into two types: regular negative samples and difficult negative samples. Regular negative samples refer to images captured by the lens while the vehicle is stationary, showing normal, unobstructed or only partially occluded scenes. However, in practical applications, the textures of some non-occluded scenes are very similar to those of occluded objects. For example, there may be large areas of sky or open road in the image; scenes with very low light levels or even near darkness; scenes with significant motion blur during driving; scenes where the lens is covered by mud or water droplets, etc. This invention specifically samples for these scenarios, thereby significantly improving the stability of the invention in various application scenarios.

[0059] S5, such as Figure 4 As shown, in the detection stage: each frame of the acquired image is input into the trained equivalent convolution kernel for convolution calculation, resulting in a feature map of the original image that combines the edge colors and textures of occluded and unoccluded objects, information exchange between channels, and the original image.

[0060] S6. The feature map of the original image, which combines edge color texture, information exchange between channels, and feature map of the original image, is divided into two parts according to the number of feature channels in the trained C2f-Star module. One part undergoes multiple feature transformations through multiple Faster-Star modules to obtain a feature map that reflects the essence of the occluded object. The other part obtains the initial feature map of the original image by combining edge color texture, information exchange between channels, and feature map of the original image through the residual path.

[0061] like Figure 5 As shown, the Faster-Star module is optimized within the C2f-Star module for feature extraction efficiency and nonlinear enhancement. The structure and function of the Faster-Star module are as follows:

[0062] 1. Partial Channel Convolution (PwConv)

[0063] The Faster-Star module first uses a PwConv (partial channel convolution) to downsample the input features. PwConv is a variant of DwConv (separate convolution). Compared to DwConv, PwConv only convolves a portion of the channels and then reassembles the convolved features with the unconvolved channel features to maintain the integrity of the information.

[0064] 2. Channel integration and splicing

[0065] The concatenated features are channel-integrated using two parallel 1×1 convolutions. One branch is activated by a ReLU function after the 1×1 convolution to introduce non-linear expressiveness. The other branch is only convolved by a 1×1 convolution without activation to preserve linear information.

[0066] 3. Dot product operation

[0067] The outputs of the two branches will undergo a dot product operation, as shown in the formula below:

[0068]

[0069] Where W1 is the weight of the convolution kernel of the first branch, W2 is the weight of the convolution kernel of the second branch, x is the input feature, and d represents the dimension of the feature.

[0070] By performing dot product operations, the model can map low-dimensional features to high-dimensional nonlinear features, similar to a kernel function, learn the correlation between feature channels, improve the model's understanding of the overall information of the feature map, and enhance the model's generalization ability.

[0071] 4. Dimensional Adjustment

[0072] The number of channels in the dot product result is adjusted by a 1×1 convolution to adapt to the subsequent network structure requirements.

[0073] S7. The feature map reflecting the essence of the occluded object and the initial edge color and texture of the occluded and non-occluded objects are combined and information is exchanged between channels. The feature map of the original image is integrated and stitched through channels, and then the number of channels is adjusted to output a feature map combining the occluded and non-occluded objects with different dimensions of information.

[0074] S8. Input the feature map that integrates information about occluded and unoccluded objects in different dimensions into the attention module. By weighting and adjusting each channel, the key features of the occluded object are highlighted, resulting in a feature map with key feature information of the occluded object.

[0075] S9. The final classifier calculates the occlusion score of each frame image using the feature map of key feature information of the occluder, and determines whether the occlusion score is greater than the occlusion threshold. If yes, the camera in that frame image is in an occluded state; otherwise, the camera in that frame image is in an unoccluded state.

[0076] The definition of whether a scene is occluded is as follows: if the occluded area (occlusion score) occupies more than 80% of the screen (occlusion threshold), it is defined as occlusion; otherwise, it is not considered occlusion.

[0077] The above embodiments are only used to illustrate the detailed solutions of the present invention. The present invention is not limited to the above detailed solutions, that is, it does not mean that the present invention must rely on the above detailed solutions to be implemented. Those skilled in the art should understand that any improvements to the present invention, equivalent substitutions of the raw materials of the product of the present invention, addition of auxiliary components, selection of specific methods, etc., are all within the protection scope and disclosure scope of the present invention.

Claims

1. A method for detecting camera occlusion, comprising the following steps: S1, Training Phase: Image features with and without occlusions are divided into three paths in the feature extraction Rep-Conv module. One path is passed through a 3×3 convolution and batch normalization layer, and then through weights and bias parameters to obtain the convolution output features corresponding to this branch. Another path is passed through a 1×1 convolution and batch normalization layer, and then through weights and bias parameters to obtain the convolution output features corresponding to this branch. The third path is passed through an identity transformation to obtain the output features corresponding to this branch. The above three paths are fused to obtain an equivalent convolution kernel that can recognize the target edge color texture and original image features. S2. After training the Rep-Conv module, pass its output features to the C2f-Star module to train the C2f-Star module, so as to obtain a C2f-Star module with the ability to recognize occluded and unoccluded objects and combine information from different dimensions. S3. After completing the training of the C2f-Star module, pass its output features to the attention module and train the attention module to obtain an attention module with the ability to identify key feature information of occluded objects. S4. After training the attention module, pass its output features to the final classifier and train the final classifier to obtain a final classifier that can identify the categories of occluded and unoccluded objects. S5. Detection stage: Each frame of the acquired image is input into the trained equivalent convolution kernel for convolution calculation to obtain a feature map of the original image that combines the edge color texture of occluded and non-occluded objects, information exchange between channels; S6. Combining edge color texture, information exchange between channels, and the feature map of the original image, the trained C2f-Star module divides the feature map into two parts according to the number of feature channels. One part undergoes multiple feature transformations through multiple Faster-Star modules to obtain a feature map reflecting the essence of the occluded object. The other part obtains the initial edge color texture of the occluded and non-occluded objects, information exchange between channels, and the feature map of the original image through the residual path. The Faster-Star module is a PwConv (partial channel convolution) that performs the following on the input features: Sampling, PwConv is a variant of DwConv (separate convolution). Compared with DwConv, PwConv only performs convolution on a portion of the channels and re-concatenates the convolved features with the unconvolved channel features. The concatenated features are then channel-integrated through two parallel 1×1 convolutions. One branch is activated by a ReLU function after the 1×1 convolution, while the other branch is only activated by a 1×1 convolution. The outputs of the two branches are then multiplied, and the result of the multiplication is adjusted by a 1×1 convolution module to adjust the number of channels. S7. The feature map reflecting the essence of the occluded object and the initial edge color texture of the occluded and non-occluded objects are combined and the information is exchanged between channels. The feature map of the original image is integrated and stitched through channels, and then the number of channels is adjusted to output a feature map that integrates the information of the occluded and non-occluded objects in different dimensions. S8. Input the feature map that integrates information from occluded and unoccluded objects in different dimensions into the attention module. By weighting and adjusting each channel, the key features of the occluded object are highlighted, and a feature map with key feature information of the occluded object is obtained. S9. The final classifier calculates the occlusion score of each frame image using the feature map of key feature information of the occluder, and determines whether the occlusion score is greater than the occlusion threshold. If yes, the camera in that frame image is in an occluded state; otherwise, the camera in that frame image is in an unoccluded state.

2. The camera occlusion detection method according to claim 1, characterized in that, In step S1, the convolutional output features corresponding to this branch are obtained through a 3×3 convolution and batch normalization layer, using weights and bias parameters. The specific design is as follows: y 3×3 =W 3×3 *x, where W 3×3 These are the weights of a 3x3 convolution, where x is the input image feature with and without occlusions, and y is the weights. 3×3 This is the feature map obtained after convolution; subsequently, the features obtained from convolution are adjusted by a batch normalization layer, and the output is: in, It is the feature map of the output edge color texture, μ 3×3 , b 3×3 It represents the mean, variance, and bias parameters of the current feature map, γ. 3×3 and β 3×3 These are the learning parameters for the batch normalized layer.

3. The camera occlusion detection method according to claim 1, characterized in that, In step S1, the convolution output features corresponding to the branch are obtained through 1×1 convolution and batch normalization layers with weights and bias parameters. The specific design is as follows: the 1x1 convolution is padded and expanded into a 3x3 feature form. These are the weights of a 1×1 convolution; the calculated features after padding are as follows: Where x is the input image feature with and without occlusions, and y 1×1 The output feature map is obtained through a 1x1 convolution; subsequently, the features obtained from the convolution are adjusted by a batch normalization layer, and the output is: in, It is y 1×1 The feature map, μ, after adjustment by the normalization layer, enables information exchange between feature channels. 1×1 , b 1×1 , where are the mean, variance, and bias parameters of the current feature map, and γ is the bias parameter. 1×1 and β 1×1 These are the learning parameters for the batch normalized layer.

4. The camera occlusion detection method according to claim 1, characterized in that, In step S1, the output features corresponding to this branch are obtained through identity transformation. The specific design is as follows: the convolution kernel of one identity matrix is ​​padded and expanded into a 3x3 feature form; These are the weights of the identity transformation; the calculated features after filling are as follows: Where x is the input image feature with and without occlusions, and y identity This is the feature map output after identity transformation; subsequently, after adjustment by the batch normalization layer, the output is: in, It is the feature map of the original image features output, μ identity , b identity , where are the mean, variance, and bias parameters of the current feature map, and γ is the bias parameter. identity and β identity These are the learning parameters for the batch normalized layer.

5. The camera occlusion detection method according to claim 1, characterized in that, There are multiple Rep-Conv and C2f-Star modules. Two Rep-Conv modules are placed near the input end of the image features. The remaining Rep-Conv modules are placed at intervals with the C2f-Star modules and connected in series. The C2f-Star module is used as the final output of the image features. The Rep-Conv and C2f-Star modules form feature extraction modules at different levels. When training the Rep-Conv module and the C2f-Star module, the first C2f-Star module, the second C2f-Star module, and the third C2f-Star module following the Rep-Conv module are connected to the first auxiliary classifier, the second auxiliary classifier, and the third auxiliary classifier, respectively. The occlusion score and non-occlusion score calculated by the final classifier are compared with the occlusion label and non-occlusion label corresponding to the original image to calculate the first difference value using the loss function; The first auxiliary classifier calculates the occlusion score and non-occlusion score of each frame image using the feature map of key feature information of occluders in this stage; The second auxiliary classifier calculates the occlusion score and non-occlusion score of each frame image using the feature map of key feature information of occluders in this stage. The third auxiliary classifier calculates the occlusion score and non-occlusion score of each frame image using the feature map of key feature information of occluders in this stage; The occlusion scores and non-occlusion scores calculated by the first, second, and third auxiliary classifiers are compared with the occlusion and non-occlusion labels of the original image using a loss function to calculate the corresponding difference values. The occlusion scores and non-occlusion scores calculated by the first, second, and third auxiliary classifiers are compared with the occlusion scores and non-occlusion scores calculated by the final classifier using a loss function to calculate the corresponding difference values. The final difference value is calculated by weighting and combining all the difference values. Based on the final difference value, gradient backpropagation is performed to update the weights and bias parameters of the Rep-Conv module, C2f-Star module, attention module, and each classifier.

Images