Camera shielding detection method and system and electronic equipment

Abstract

The invention provides a camera shielding detection method and system and electronic equipment, and the method comprises the steps: obtaining continuous multiple frames of to-be-detected original images, and splicing the continuous multiple frames of to-be-detected original images into a multi-channel image; and inputting the multi-channel image into the constructed shielding detection model, extracting spatial-temporal characteristics of the multi-channel image through the shielding detection model, and obtaining a shielding detection result according to the spatial-temporal characteristics. According to the method, joint input of spatial and temporal characteristics is realized, low-parameter and high-robustness camera shielding detection is realized, the problems of false detection and missing detection of an existing method in a complex scene are solved, and meanwhile, the real-time requirement of edge equipment is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a camera occlusion detection method and system and electronic equipment. BACKGROUND

[0002] As a core device for image acquisition, the camera lens will cause image information loss when it is blocked, which will further cause security vulnerabilities, device misjudgment and other risks. Traditional camera lens occlusion detection methods mostly rely on threshold segmentation techniques or are based on manually extracted features such as edge density and brightness mutation for judgment. However, such methods have a big discount in detection effect when facing complex lighting environments, as well as diversified occlusion modes such as dust and fingerprint coverage, and color changes of the occlusion object. Although deep learning technology has been widely applied in image classification, there are the following bottlenecks:

[0003] 1. Single-frame detection defects: susceptible to light interference, unable to distinguish between occlusion and natural shadows (such as tree leaf projections).

[0004] 2. Low efficiency of spatio-temporal modeling: traditional methods such as optical flow are complex to calculate and difficult to embed into lightweight models.

[0005] 3. Insufficient model lightweight: mainstream models have large parameter quantities (such as YOLOv5n reaching 1.8M), and the deployment cost of edge devices is high. SUMMARY

[0006] The present application aims to solve at least one of the technical problems existing in the prior art, and proposes a camera occlusion detection method, system and electronic equipment.

[0007] In a first aspect, the present application discloses a camera occlusion detection method, comprising the following steps:

[0008] Obtaining continuous multiple frames of original detection images, and splicing the continuous multiple frames of original detection images into a multi-channel image;

[0009] Inputting the multi-channel image into a constructed occlusion detection model, extracting spatio-temporal features of the multi-channel image through the occlusion detection model, and obtaining an occlusion detection result according to the spatio-temporal features.

[0010] In some embodiments, obtaining continuous multiple frames of original detection images and splicing the continuous multiple frames of original detection images into a multi-channel image comprises: obtaining continuous two frames of original detection images and splicing the continuous two frames of original detection images into a six-channel image. The original detection image is an RGB image.

[0011] In some embodiments, the occlusion detection model comprises a backbone network and a classification module, the backbone network is used to extract spatio-temporal features of the multi-channel image, and the classification module is used to convert the spatio-temporal features extracted by the backbone network into an occlusion probability output.

[0012] In some embodiments, the backbone network includes a flow base module, a base module, and a cascade module. The cascade module is composed of multiple base modules cascaded sequentially. The flow base module serves as the entry module of the backbone network, and the base modules and cascade modules are stacked alternately.

[0013] In some embodiments, the classification module includes an adaptive global average pooling layer and a binary classification decision layer. The adaptive global average pooling layer is used to receive the feature map output by the backbone network, take the average value of all spatial locations of each feature channel, and output it to the binary classification decision layer. The binary classification decision layer is used to output binary classification based on the output of the adaptive global average pooling layer using a linear classification head, corresponding to the two states of occlusion and non-occlusion respectively.

[0014] In some embodiments, the adaptive global average pooling layer receives the [B,128,H,W] feature map from the backbone network and outputs the [B,128,1,1] global feature, providing a fixed-dimensional input to the binary classification decision layer.

[0015] In some embodiments, the occlusion detection model uses label smooth cross-entropy as the loss function.

[0016] In some embodiments, the formula for the loss function is:

[0017]

[0018] Where L is the output value of the loss function, representing the overall difference between the model's prediction and the true label (the smaller the value, the better the model fits), N is the number of samples, that is, the total number of samples contained in the current training / validation batch; i=1 to N means traversing all samples, and K is the number of categories, that is, the total number of categories that need to be distinguished in the classification task. =1 to K indicates traversing all categories. The sample index ranges from 1 to N, representing the nth index. One sample, This is a category index, with values ​​ranging from 1 to K, representing the th . Categories For the sample The actual label corresponding to category k, For the model to the first The predicted value for the nth sample is... The probability values ​​of each category, These are hyperparameters (weighting coefficients), typically ranging from [0,1], used to balance the contributions of the two parts of the loss function. This is the smoothing coefficient.

[0019] In some embodiments, the occlusion detection results of the continuous frames are obtained, and the occlusion detection results of the continuous frames are smoothed to obtain a final occlusion detection result.

[0020] In some embodiments, the smoothing of the occlusion detection results of the continuous frames specifically comprises: obtaining the occlusion detection results of the continuous frames in a preset sliding window, counting the number of occlusion frames, calculating the proportion of the occlusion frames, and if the proportion of the occlusion frames exceeds a preset occlusion proportion threshold, marking the final occlusion detection result as occlusion; otherwise, marking the final occlusion detection result as non-occlusion.

[0021] In some embodiments, the camera occlusion detection method further comprises the steps of: presetting a sliding window and an occlusion proportion threshold.

[0022] In some embodiments, the camera occlusion detection method further comprises the steps of: obtaining a vehicle speed, and dynamically adjusting the size of the sliding window and the preset occlusion proportion threshold according to the vehicle speed.

[0023] In a second aspect, the present application further discloses a camera occlusion detection system, comprising:

[0024] An image stitching module is configured to obtain a plurality of continuous frames of original images to be detected, and stitch the plurality of continuous frames of original images to be detected into a multi-channel image.

[0025] An occlusion detection module is configured to input the multi-channel image into a constructed occlusion detection model, extract spatio-temporal features of the multi-channel image through the occlusion detection model, and obtain an occlusion detection result according to the spatio-temporal features.

[0026] In some embodiments, the camera occlusion detection system further comprises a smoothing module configured to obtain the occlusion detection results of the continuous frames, and smooth the occlusion detection results of the continuous frames to obtain a final occlusion detection result.

[0027] In a third aspect, the present application further discloses an electronic device, comprising:

[0028] One or more processors;

[0029] A memory configured to store one or more programs;

[0030] When the one or more programs are executed by the one or more processors, the one or more processors implement the method of the first aspect.

[0031] In a fourth aspect, the present application further discloses a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in the method of the first aspect.

[0032] The application has at least the following beneficial effects: the application discloses a lightweight camera occlusion detection method based on space-time feature fusion, directly models space-time differences of continuous frames through multiple (such as 6) channel images, replaces traditional optical flow calculation, realizes joint input of space-time features, then inputs the multiple channel images into the constructed occlusion detection model, extracts space-time features of the multiple channel images through the occlusion detection model, and obtains an occlusion detection result according to the space-time features, the above scheme of the application realizes low-parameter and high-robustness camera occlusion detection, solves the false detection and missed detection problems of existing methods in complex scenes, and meets real-time requirements of edge devices. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 A flowchart of a camera occlusion detection method provided by the embodiment of the application is shown in the figure.

[0034] Figure 2 A network structure diagram of an occlusion detection model provided by the embodiment of the application is shown in the figure.

[0035] Figure 3 A flowchart of a smoothing step provided by the embodiment of the application is shown in the figure.

[0036] Figure 4 A schematic diagram of a camera occlusion detection system provided by the embodiment of the application is shown in the figure.

[0037] Figure 5 A structural block diagram of an electronic device provided by the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0038] In order for those skilled in the art to better understand the technical solutions of the present application, the following describes exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to help understanding, which should be considered only as exemplary. Therefore, those skilled in the art should realize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, in order to be clear and concise, the description in the following description omits the description of well-known functions and structures.

[0039] In the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

[0040] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0041] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. "Coupled" or "connected" or similar terms are not restricted to physical or mechanical connections or associations, but can also include electrical connections, whether direct or indirect.

[0042] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an overly literal or overly formal sense unless expressly so defined herein.

[0043] In the technical solutions of the present application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information comply with relevant laws and regulations and do not violate public order and good customs. The use of user data in the technical solutions complies with relevant national laws and regulations (for example, "Information Security Technology Personal Information Security Specification" and the like). For example, appropriate measures are taken for personal information access control; the display of personal information is limited as specified; the use purpose of personal information does not exceed the direct or reasonably related range; and the use of personal information eliminates explicit identity pointing and avoids precise positioning to a specific individual.

[0044] As the core device of image acquisition, the camera lens will cause image information loss if it is blocked, which will further cause security loopholes, device misjudgment and other risks. Traditional camera lens blocking detection methods mostly rely on threshold segmentation technology, or are based on manually extracted features such as edge density and brightness mutation for judgment. However, such methods have a big discount in detection effect when facing complex lighting environments, as well as diversified blocking modes such as dust and fingerprint coverage, and color changes of the blocking object. Although deep learning technology has been widely applied in image classification, there are the following bottlenecks:

[0045] 1. Single-frame detection defects: susceptible to light interference, unable to distinguish between blocking and natural shadows (such as tree leaf projections).

[0046] 2. Low efficiency of space-time modeling: traditional methods such as optical flow are complex to calculate and difficult to embed into lightweight models.

[0047] 3. Insufficient model lightweight: mainstream models have large parameter quantities (e.g., YOLOv5n reaches 1.8M), and the deployment cost of edge devices is high.

[0048] Related technologies disclose a threshold detection scheme based on a single frame image, which segments the image brightness and texture features by a fixed threshold, and marks the abnormal area as an occlusion. The defect is that it cannot handle dynamic light changes, and the false detection rate is high (e.g., glass reflection triggers false positives).

[0049] Related technologies also disclose a spatiotemporal detection scheme based on optical flow, which calculates the optical flow field between consecutive frames, and judges the occlusion by motion consistency. The defect is that the optical flow calculation requires O (n²) time complexity, and the real-time performance of edge devices is poor.

[0050] Related technologies also disclose a lightweight classification model detection scheme, which uses a lightweight network such as MobileNetV3 to directly classify whether an image is occluded. The defect is that the parameter quantity still reaches 1.6M, and the detection accuracy of small target occlusions (e.g., local stains on the lens) is insufficient (mAP<70%).

[0051] To effectively improve, or even completely solve the above problems existing in related technologies, the present disclosure provides a corresponding solution. The present disclosure provides a low-parameter and high-robustness camera occlusion detection scheme to solve the false detection and missed detection problems of existing methods in complex scenes, while meeting the real-time requirements of edge devices.

[0052] Figure 1 The flowchart of the camera occlusion detection method provided by an embodiment of the present application is shown in Figure 1 A camera occlusion detection method includes the following steps:

[0053] S1, acquire a plurality of consecutive frames of original images to be detected, and splice the plurality of consecutive frames of original images to be detected into a multi-channel image;

[0054] S2, input the multi-channel image into a constructed occlusion detection model, extract the spatiotemporal features of the multi-channel image through the occlusion detection model, and obtain an occlusion detection result according to the spatiotemporal features.

[0055] The present application directly models the spatiotemporal differences of multiple (e.g., 6) channel images, replaces the traditional optical flow calculation, realizes the joint input of spatiotemporal features, then inputs the multi-channel image into the constructed occlusion detection model, extracts the spatiotemporal features of the multi-channel image through the occlusion detection model, and obtains an occlusion detection result according to the spatiotemporal features. The above scheme of the present application realizes low-parameter and high-robustness camera occlusion detection, solves the false detection and missed detection problems of existing methods in complex scenes, and meets the real-time requirements of edge devices.

[0056] In some embodiments, acquiring multiple consecutive frames of original images to be detected and stitching them together into a multi-channel image includes: acquiring N consecutive frames of original images to be detected and stitching them together into a 3*N channel image. N is a positive integer. Preferably, N is greater than or equal to 2.

[0057] In a preferred embodiment, acquiring multiple consecutive frames of the original image to be detected and stitching them together into a multi-channel image includes: acquiring two consecutive frames of the original image to be detected and stitching them together into a six-channel image. The original image to be detected is an RGB (three-primary-color mode, used to represent color images) image.

[0058] In some embodiments, the occlusion detection model is a lightweight occlusion detection model.

[0059] In some embodiments, the occlusion detection model includes a backbone network and a classification module (classification head). The backbone network is used to extract spatiotemporal features from multi-channel images, and the classification module is used to convert the spatiotemporal features extracted by the backbone network into occlusion probability outputs. The backbone network is a lightweight backbone network (V11).

[0060] In some embodiments, the backbone network includes a flow basic module (BasicBlock_Flow), a basic module (BasicBlock), and a cascade module (ConcatBlock). The cascade module is composed of multiple basic modules cascaded in sequence. The flow basic module serves as the entry module of the backbone network, and the basic modules and cascade modules are stacked alternately.

[0061] The lightweight backbone architecture uses alternating stacks of BasicBlock and ConcatBlock to achieve an efficient pipeline for "downsampling-multi-scale feature fusion". The residual-free connection design reduces parameters by more than 30%.

[0062] The BasicBlock_Flow module serves as the network's entry point, receiving multi-channel (e.g., six-channel) RGB images. Its internal structure includes Conv2d (a two-dimensional convolutional layer used to extract spatial features), BatchNorm2d (a two-dimensional batch normalization layer used to accelerate model convergence), and activation functions (such as ReLU, a linear rectified activation function that introduces non-linear characteristics). This enables rapid downsampling of the input image and extraction of low-level image features.

[0063] The BasicBlock module also includes Conv2d (a 2D convolutional layer for extracting spatial features), BatchNorm2d (a 2D batch normalization layer to accelerate model convergence), and activation functions (such as ReLU, a linear rectified activation function that introduces non-linear characteristics). By setting the stride=2 in convolution operations, it gradually reduces the image resolution while increasing the number of channels. This module adopts a ResNet-like hierarchical structure, which can gradually expand the network's receptive field and extract higher-dimensional image features.

[0064] The ConcatBlock module consists of multiple cascaded BasicBlocks, with each ConcatBlock containing multiple (e.g., 4) BasicBlocks. These multiple (e.g., 4) BasicBlocks are cascaded in a "front → back" order, occupying different network layer positions (the first one is a "shallow sub-block," and the last one is a "deep sub-block"). This module, through sub-blocks of different depths, can capture image features of different granularities, effectively enhancing feature diversity.

[0065] The feature fusion method of the present invention can replace ConcatBlock with DepthwiseSeparableConvolution, further reducing the number of parameters (by about 20%), but slightly reducing the feature diversity.

[0066] In some embodiments, the present invention may also replace the ReLU activation function with H-Swish to improve nonlinear expression.

[0067] All blocks have no residual connections, belonging to simple convolutional blocks, which are suitable for lightweight models.

[0068] In some embodiments, the classification module includes an adaptive global average pooling layer (AdaptiveAvgPool2d module) and a binary classification decision layer (Linear binary classification decision layer). The adaptive global average pooling layer is used to receive the feature map output by the backbone network, take the average value of all spatial locations of each feature channel, and output it to the binary classification decision layer. The binary classification decision layer is used to output binary classification based on the output of the adaptive global average pooling layer using a linear classification head, corresponding to the two states of occlusion and non-occlusion respectively.

[0069] In some embodiments, the AdaptiveAvgPool2d module receives the feature map (with shape [B,128,H,W]) output by the backbone network, averages all spatial locations for each channel, and outputs a feature with shape [B,128,1,1]. This operation can effectively preserve global information, such as the overall response intensity of the occluded area.

[0070] In some embodiments, the Linear binary classification decision layer outputs binary logits based on the output of the AdaptiveAvgPool2d module using a linear classification head, corresponding to the two states of "occlusion" and "unocclusion".

[0071] In some embodiments, see Figure 2 The backbone network consists of the following modules arranged in sequence: BasicBlock_Flow, BasicBlock, and at least one ConcatBlock (at least one concatenated module is cascaded in sequence).

[0072] The basic module serves as the entry point for the backbone network. The basic module (BasicBlock) is used for feature extraction, and the concatenated module (ConcatBlock) is used for feature concatenation / fusion. After the basic module, the basic module (BasicBlock) and the concatenated module (ConcatBlock) are stacked alternately to achieve an efficient pipeline of "downsampling-multi-scale feature fusion".

[0073] This invention involves inputting multi-channel images into a trained occlusion detection model.

[0074] In some embodiments, the occlusion detection model uses label smooth cross-entropy as the loss function.

[0075] In some embodiments, the formula for the loss function is:

[0076]

[0077] Where L is the output value of the loss function, representing the overall difference between the model's prediction and the true label (the smaller the value, the better the model fits), N is the number of samples, that is, the total number of samples contained in the current training / validation batch; i=1 to N means traversing all samples, and K is the number of categories (in this embodiment, K=2), that is, the total number of categories that need to be distinguished in the classification task. =1 to K indicates traversing all categories. The sample index ranges from 1 to N, representing the nth index. One sample, This is a category index, with values ​​ranging from 1 to K, representing the th . Categories For the sample The actual label corresponding to category k, For the model to the first The predicted value for the nth sample is... The probability values ​​of each category, These are hyperparameters (weighting coefficients), typically ranging from [0,1], used to balance the contributions of the two parts of the loss function. This is the smoothing coefficient (with a value of, for example, 0.1).

[0078] This invention effectively prevents overfitting during model training by introducing label-smoothed cross-entropy.

[0079] In some embodiments, occlusion detection results of consecutive frames are obtained, and the occlusion detection results of consecutive frames are smoothed to obtain the final occlusion detection result. This invention employs dynamic sliding window filtering to achieve state smoothing.

[0080] In some embodiments, see Figure 3 The method smooths the occlusion detection results of consecutive frames (i.e., sliding window filtering), specifically including: pre-setting the window size `smooth_length` and the occlusion ratio threshold `smooth_block_ratio`; and performing a voting decision on the detection results of multiple consecutive frames, including: obtaining the occlusion detection results of consecutive frames within the preset sliding window (`smooth_length`), counting the number of occluded frames, calculating the proportion of occluded frames (the proportion of occluded frames within the preset sliding window); if the proportion of occluded frames (`num_blocked`) exceeds the preset occlusion ratio threshold (`smooth_block_ratio`), the final occlusion detection result is marked as occluded; otherwise, it is marked as not occluded. This method can effectively suppress false judgments of single-frame detection results and improve detection stability, for example, it can avoid false alarms triggered by brief occlusion by leaves.

[0081] In some embodiments, the camera occlusion detection method of the present invention further includes the following steps: acquiring vehicle speed, and dynamically adjusting the sliding window size and a preset occlusion ratio threshold according to the vehicle speed.

[0082] This invention dynamically adjusts the sliding window size (smooth_length) and the occlusion ratio threshold (smooth_block_ratio) based on vehicle speed. When the vehicle is traveling at high speed, the window size is shortened to achieve a rapid response to occlusion; when the vehicle is traveling at low speed or stationary, the window size is increased to reduce false positives. This invention also adaptively adjusts the smoothing strategy based on vehicle speed, balancing detection speed and stability.

[0083] This invention receives multiple (e.g., two) consecutive RGB images from a camera, stitches them together to form a multi-channel image (e.g., six-channel image with dimensions [batch, 6, H, W]), and directly inputs it into a constructed occlusion detection model. The occlusion detection model outputs the image as a confidence score for either "occluded" or "unoccluded" categories. A post-processing algorithm is then used to calibrate the detection results. The post-processing algorithm employs a sliding window filter to achieve state smoothing. This invention achieves robust detection of camera occlusion through key steps including image input, model prediction, and result processing. In particular, the sliding window mechanism significantly improves the stability of the detection results, making it suitable for scenarios with high reliability requirements, such as autonomous driving and security monitoring. Furthermore, the model has approximately 220,000-270,000 parameters (0.22M-0.27M), significantly less than mainstream detection models, making it suitable for deployment on edge devices with limited computing resources (such as Jetson Nano).

[0084] Based on the same inventive concept, the present invention provides a camera occlusion detection system. Figure 4 This is a schematic diagram of a camera occlusion detection system provided in an embodiment of the present invention. See also: Figure 4 A camera occlusion detection system includes: an image stitching module for acquiring multiple consecutive frames of original images to be detected and stitching the multiple consecutive frames of original images to be detected into a multi-channel image;

[0085] The occlusion detection module is used to input multi-channel images into the constructed occlusion detection model, extract the spatiotemporal features of the multi-channel images through the occlusion detection model, and obtain the occlusion detection result based on the spatiotemporal features.

[0086] In some embodiments, the camera occlusion detection system of the present invention further includes a smoothing module, which is used to obtain occlusion detection results of consecutive frames, smooth the occlusion detection results of consecutive frames, and obtain the final occlusion detection result.

[0087] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 5 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the camera occlusion detection methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0088] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0089] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0090] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0091] This invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the camera occlusion detection methods described in the above embodiments. The computer-readable storage medium may be volatile or non-volatile.

[0092] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described camera occlusion detection method.

[0093] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0094] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0095] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0096] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0097] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0098] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0099] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0100] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0102] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A method for detecting camera occlusion, characterized in that, Includes the following steps: Acquire multiple consecutive frames of the original image to be detected, and stitch the multiple consecutive frames of the original image to be detected into a multi-channel image; The multi-channel image is input into the constructed occlusion detection model. The spatiotemporal features of the multi-channel image are extracted by the occlusion detection model, and the occlusion detection result is obtained based on the spatiotemporal features.

2. The method according to claim 1, characterized in that: The occlusion detection model includes a backbone network and a classification module. The backbone network is used to extract spatiotemporal features of multi-channel images, and the classification module is used to convert the spatiotemporal features extracted by the backbone network into occlusion probability output.

3. The method according to claim 2, characterized in that: The backbone network includes a flow base module, a base module, and a cascade module. The cascade module is composed of multiple base modules cascaded in sequence. The flow base module serves as the entry module of the backbone network, and the base modules and cascade modules are stacked alternately.

4. The method according to claim 2, characterized in that: The classification module includes an adaptive global average pooling layer and a binary classification decision layer. The adaptive global average pooling layer receives the feature map output by the backbone network, averages all spatial locations of each feature channel, and outputs it to the binary classification decision layer. The binary classification decision layer outputs a binary classification based on the output of the adaptive global average pooling layer using a linear classification head, corresponding to the two states of occlusion and non-occlusion.

5. The method according to claim 1, characterized in that: The occlusion detection model uses label smooth cross-entropy as the loss function.

6. The method according to claim 1, characterized in that: Obtain the occlusion detection results of consecutive frames, smooth the occlusion detection results of consecutive frames, and obtain the final occlusion detection result.

7. The method according to claim 6, characterized in that: The smoothing process for the occlusion detection results of consecutive frames specifically includes: obtaining the occlusion detection results of consecutive frames within a preset sliding window, counting the number of occluded frames, calculating the proportion of occluded frames, and if the proportion of occluded frames exceeds a preset occlusion ratio threshold, the final occlusion detection result is marked as occluded; otherwise, it is marked as unoccluded.

8. The method according to claim 7, characterized in that: It also includes the following steps: obtaining the vehicle speed, and dynamically adjusting the size of the sliding window and the preset occlusion ratio threshold according to the vehicle speed.

9. A camera occlusion detection system, characterized in that, include: The image stitching module is used to acquire multiple consecutive frames of the original image to be detected and stitch them together into a multi-channel image. The occlusion detection module is used to input multi-channel images into the constructed occlusion detection model, extract the spatiotemporal features of the multi-channel images through the occlusion detection model, and obtain the occlusion detection result based on the spatiotemporal features.

10. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 8.

Images