A method and device for identifying foreign objects in a camera lens

Abstract

This application relates to the field of image processing technology, and in particular to a method and apparatus for identifying foreign objects on camera lenses, to solve the problem in the prior art that the type of foreign object on a camera lens cannot be determined. The method provided in this application includes: acquiring a first video frame and a second video frame; dividing the first and second video frames into N sub-image blocks, and identifying sub-image blocks in the N sub-image blocks corresponding to the first and second video frames that contain suspected foreign objects; performing contour searching based on the sub-image blocks in the first and second video frames to determine a first contour region in the first video frame and a second contour region in the second video frame that contain suspected foreign objects, and determining a circumscribed rectangular region based on the second contour region; and determining whether the suspected foreign object is a water droplet or a stain based on the first contour region, the second contour region, and the circumscribed rectangular region.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and apparatus for identifying foreign objects in a camera lens. Background Technology

[0002] With the development of video surveillance technology, surveillance cameras have gradually become a widely used security device. Combining deep learning and traditional image processing technologies, automated monitoring has penetrated various fields, such as high-altitude object throwing detection, traffic violation detection, and safety passage obstruction detection. Once non-compliant behavior is detected, an alarm can be automatically triggered, eliminating the need for manual inspection and saving significant human and material resources. Clear monitoring images and good image quality are crucial for effective monitoring. When the lens is dirty, the image captured by the camera will be blurry, requiring personnel to take appropriate measures. Current technology can identify whether the lens is dirty and the degree of dirtiness, but it cannot determine whether the foreign object on the lens is a stain or a water droplet, thus failing to accurately alert relevant personnel to take appropriate action. Summary of the Invention

[0003] This application provides a method and apparatus for identifying foreign objects on camera lenses, which solves the problem in the prior art that the type of foreign object on a camera lens cannot be determined.

[0004] In a first aspect, embodiments of this application provide a method for identifying foreign objects in a camera lens, including:

[0005] The process involves acquiring a first video frame and a second video frame from a video stream captured by a camera. The second video frame is the video frame to be detected, and a set time interval is set between the first and second video frames. The first and second video frames are each divided into N sub-image blocks. From the N sub-image blocks in the first video frame, M1 sub-image blocks with suspected lens foreign objects are identified, and from the N sub-image blocks in the second video frame, M2 sub-image blocks with suspected lens foreign objects are identified. Contour lookup is performed on the M1 sub-image blocks to determine a first contour region of the suspected lens foreign object. Contour lookup is performed on the M2 sub-image blocks to determine a second contour region of the suspected lens foreign object, and the circumscribed rectangular region of the second contour region is determined. The first and second contour regions overlap. An optical flow map of the suspected lens foreign object is determined based on the first and second contour regions. Finally, the image of the circumscribed rectangular region of the second contour region and the optical flow map of the suspected lens foreign object are used to determine whether the suspected lens foreign object is a water droplet.

[0006] Based on the above scheme, the area where foreign objects are located can be determined, and then the type of foreign object can be identified to alert relevant personnel to take appropriate measures. Furthermore, the above scheme is robust to changes in lighting and environment, and has high identification accuracy.

[0007] In one possible implementation, the camera is an action camera, and the process of determining M1 sub-image blocks with suspected lens foreign objects from N sub-image blocks in the first video frame, and determining M2 sub-image blocks with suspected lens foreign objects from N sub-image blocks in the second video frame, includes:

[0008] The N sub-image blocks in the first video frame are matched with the sub-image blocks in the second video frame to identify M1 sub-image blocks with suspected lens foreign objects from the N sub-image blocks in the first video frame, and M2 sub-image blocks with suspected lens foreign objects from the N sub-image blocks in the second video frame; wherein the M1 sub-image blocks and the M2 sub-image blocks have a matching relationship.

[0009] In one possible implementation, determining the match between the M1 sub-image blocks and the M2 sub-image blocks includes: determining the similarity between the feature vector of the first sub-image block in the first video frame and the feature vectors of K sub-image blocks in the second video frame; the K sub-image blocks include sub-image blocks in the second video frame that are at the same position as the first sub-image block in the first video frame, as well as sub-image blocks within their neighborhood range.

[0010] When it is determined that the similarity between the feature vector of the first sub-image block and the feature vector of the second sub-image block among the K sub-image blocks is greater than a first set threshold, the first sub-image block is a sub-image block in the first video frame that has a suspected lens foreign object, and the second sub-image block is a sub-image block in the second video frame that has a suspected lens foreign object.

[0011] In one possible implementation, the camera is a still camera, and the process of determining M1 sub-image blocks with suspected lens foreign objects from N sub-image blocks in the first video frame, and determining M2 sub-image blocks with suspected lens foreign objects from N sub-image blocks in the second video frame, includes:

[0012] The N sub-image blocks included in the first video frame and the second video frame are matched with the feature information of the images in the first sample set to determine the category corresponding to the N sub-image blocks included in the first video frame and the second video frame, wherein the category includes background, water droplets or stains; wherein, the first sample set includes multiple lens water droplet images, lens stain images and background images;

[0013] The M1 sub-image blocks in the first video frame that are classified as water droplets or stains are considered as sub-image blocks with suspected foreign objects in the lens, and the M2 sub-image blocks in the second video frame that are classified as water droplets or stains are considered as sub-image blocks with suspected foreign objects in the lens.

[0014] Based on the above scheme, different methods were used to obtain the polygonal region of the foreign object in the lens for both moving and stationary cameras. Different methods can be used for cameras in different states, making it applicable to different application scenarios.

[0015] In one possible implementation, the method further includes: when it is determined that the suspected lens foreign object is not a water droplet, determining the area change rate of the second contour region relative to the first contour region, and determining the category of the second contour region through a first classification model; when the area change rate is less than a second set threshold and the category of the second contour region determined by the first classification model is a stain, determining that the suspected lens foreign object in the second contour region in the second video frame is a stain.

[0016] In one possible implementation, determining whether the suspected lens foreign object is a water droplet based on the image of the circumscribed rectangular region of the second contour region and the optical flow map of the suspected lens foreign object includes:

[0017] The image of the outer rectangular region of the second contour region is used as the input of the spatial flow network in the first classification model, and the optical flow map of the suspected lens foreign object is used as the input of the temporal flow network in the first classification model, so as to determine whether the suspected lens foreign object in the second contour region is a water droplet through the first classification model.

[0018] Based on the above scheme, determining whether a lens foreign object is a water droplet is based on image information and motion information over time. Determining whether a lens foreign object is a lens stain is based on the rate of change of the polygonal region's area and the detection result of the polygon. Therefore, this method is robust to changes in lighting and environment, reducing the probability of misidentifying background areas as lens foreign objects and improving the algorithm's accuracy.

[0019] In one possible implementation, the first classification model includes a spatial flow network and a temporal flow network. The first classification model is trained based on a first training sample set, which includes multiple samples. Each sample includes multiple temporally consecutive images containing lens foreign objects and the type of lens foreign object corresponding to each sample. The method further includes obtaining a first neural network model in the following manner:

[0020] Determine the optical flow map of the first image of the first sample among the plurality of samples;

[0021] The first image in the first sample is input into the spatial flow network to output a first probability distribution of the lens foreign object type corresponding to the first image. The optical flow map is input into the temporal flow network to output a second probability distribution of the lens foreign object type corresponding to the first image. The first sample includes multiple temporally consecutive images including the first image and the lens foreign object type corresponding to the first sample.

[0022] The predicted type of the first image is determined based on the first probability distribution and the second probability distribution; a first loss value is determined based on the predicted type of the first image and the category of the first sample; the network parameters of the spatial flow network and the temporal flow network are adjusted based on the first loss value to obtain the first classification model.

[0023] In one possible implementation, the method further includes: outputting the detection result of a suspected lens foreign object in the second contour region of the second video frame, the detection result including the type of lens foreign object and the coordinates of the smallest bounding rectangle region of the lens foreign object.

[0024] Secondly, embodiments of this application provide a foreign object identification device for a camera lens, comprising:

[0025] The acquisition module is used to acquire the first video frame and the second video frame in the video stream captured by the camera. The second video frame is the video frame to be detected. The interval between the first video frame and the second video frame is set to a certain duration.

[0026] The segmentation module is used to divide the first video frame and the second video frame into N sub-image blocks respectively;

[0027] The determination module is configured to: determine M1 sub-image blocks containing suspected lens foreign objects from N sub-image blocks in the first video frame; and determine M2 sub-image blocks containing suspected lens foreign objects from N sub-image blocks in the second video frame; perform contour lookup on the M1 sub-image blocks to determine a first contour region of the suspected lens foreign object; perform contour lookup on the M2 sub-image blocks to determine a second contour region of the suspected lens foreign object, and determine the circumscribed rectangular region of the second contour region; the first contour region and the second contour region overlap; determine the optical flow map of the suspected lens foreign object based on the first contour region and the second contour region; and determine whether the suspected lens foreign object is a water droplet based on the image of the circumscribed rectangular region of the second contour region and the optical flow map of the suspected lens foreign object.

[0028] In one possible implementation, the camera is an action camera, and the determining module, when determining M1 sub-image blocks with suspected lens foreign objects from N sub-image blocks in the first video frame, and M2 sub-image blocks with suspected lens foreign objects from N sub-image blocks in the second video frame, is specifically used for:

[0029] The feature information of N sub-image blocks in the first video frame is matched with that of the sub-image blocks in the second video frame, so as to determine M1 sub-image blocks with suspected lens foreign objects from the N sub-image blocks in the first video frame, and M2 sub-image blocks with suspected lens foreign objects from the N sub-image blocks in the second video frame.

[0030] Among them, the M1 sub-image blocks and the M2 sub-image blocks have a matching relationship.

[0031] In one possible implementation, the determining module, when determining that the M1 sub-image blocks match the M2 sub-image blocks, is specifically configured to: determine the similarity between the feature vector of the first sub-image block in the first video frame and the feature vectors of K sub-image blocks in the second video frame; the K sub-image blocks include sub-image blocks in the second video frame that are at the same position as the first sub-image block in the first video frame and sub-image blocks within their neighborhood; when the similarity between the feature vector of the first sub-image block and the feature vector of the second sub-image block among the K sub-image blocks is determined to be greater than a first preset threshold, the first sub-image block is a sub-image block in the first video frame that has a suspected lens foreign object, and the second sub-image block is a sub-image block in the second video frame that has a suspected lens foreign object.

[0032] In one possible implementation, the camera is a still camera. The determining module, when determining M1 sub-image blocks with suspected lens foreign objects from N sub-image blocks in the first video frame, and M2 sub-image blocks with suspected lens foreign objects from N sub-image blocks in the second video frame, specifically performs the following: matching the feature information of the N sub-image blocks included in the first and second video frames with images in a first sample set to determine the category corresponding to the N sub-image blocks included in the first and second video frames, where the category includes background, water droplets, or stains; wherein, the first sample set includes multiple lens water droplet images, lens stain images, and background images; and designating the M1 sub-image blocks in the first video frame with the category of water droplets or stains as sub-image blocks with suspected lens foreign objects, and the M2 sub-image blocks in the second video frame with the category of water droplets or stains as sub-image blocks with suspected lens foreign objects.

[0033] In one possible implementation, the determining module is further configured to: when it is determined that the suspected lens foreign object is not a water droplet, determine the area change rate of the second contour region relative to the first contour region, and determine the category of the second contour region through a first classification model;

[0034] When the area change rate is less than the second set threshold and the category of the second contour region determined by the first classification model is a stain, the suspected lens foreign object in the second contour region of the second video frame is determined to be a stain.

[0035] In one possible implementation, the determining module, when determining whether the suspected lens foreign object is a water droplet based on the image of the circumscribed rectangular region of the second contour region and the optical flow map of the suspected lens foreign object, is specifically configured to: use the image of the circumscribed rectangular region of the second contour region as input to the spatial flow network in the first classification model, and use the optical flow map of the suspected lens foreign object as input to the temporal flow network in the first classification model, so as to determine whether the suspected lens foreign object in the second contour region is a water droplet through the second classification model.

[0036] In one possible implementation, the first classification model is trained based on a first training sample set, which includes multiple samples. Each sample includes multiple temporally consecutive images containing lens foreign objects and the type of lens foreign object corresponding to each sample. The determining module is further configured to obtain the first neural network model in the following manner:

[0037] Determine the optical flow map of the first image of the first sample among the plurality of samples;

[0038] The first image from the first sample is input into the spatial flow network to output a first probability distribution of the lens foreign object type corresponding to the first image. The optical flow map is input into the temporal flow network to output a second probability distribution of the lens foreign object type corresponding to the first image. The first sample includes a first image and the type of lens foreign object corresponding to the first image. The predicted type of the first image is determined based on the first and second probability distributions. A first loss value is determined based on the predicted type of the first image and the category of the first sample. The network parameters of the spatial flow network and the temporal flow network are adjusted based on the first loss value to obtain the first classification model.

[0039] In one possible implementation, the device further includes an output module: the output module is further configured to output the detection result of a suspected lens foreign object in the second contour region of the second video frame, the detection result including the type of lens foreign object and the coordinates of the smallest bounding rectangle region of the lens foreign object.

[0040] Thirdly, embodiments of this application provide a device for identifying foreign objects in a camera lens, including:

[0041] Memory, used to store program instructions;

[0042] A processor is configured to invoke program instructions stored in the memory and execute the method described in the first aspect and different implementations of the first aspect according to the obtained program instructions.

[0043] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect and different implementations of the first aspect.

[0044] Furthermore, the technical effects of any of the implementation methods in the second to fourth aspects can be found in the first aspect and the technical effects of different implementation methods of the first aspect, which will not be repeated here. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1A This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0047] Figure 1B A schematic diagram of a server structure provided in an embodiment of this application;

[0048] Figure 2 A flowchart illustrating a method for identifying foreign objects in a camera lens, as provided in this application embodiment;

[0049] Figure 3 This is a schematic diagram of two consecutive video frames divided into blocks, as provided in an embodiment of this application.

[0050] Figure 4 A schematic diagram of a sub-image block containing a suspected foreign object in a lens, provided as an embodiment of this application;

[0051] Figure 5 A schematic diagram of the outline region of a suspected lens foreign object provided in an embodiment of this application;

[0052] Figure 6 A schematic diagram of the training process of the first classification model provided in the embodiments of this application;

[0053] Figure 7 A schematic diagram of the network structure of the first classification model provided in the embodiments of this application;

[0054] Figure 8 This is a schematic diagram of the training process of the second classification model provided in the embodiments of this application;

[0055] Figure 9 This is a schematic diagram of a foreign object identification device for a camera lens provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0057] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0058] The terms "first" and "second" in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The term "multiple" in this application can mean at least two, for example, two, three, or more, and is not limited by the embodiments of this application.

[0059] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0060] In existing technologies, automated monitoring has been widely applied in various fields, such as scene recognition and traffic incident recognition. When non-compliant behavior is detected, the camera can issue an alarm, eliminating the need for manual inspection and saving human and material resources. However, in some scenarios, camera lenses may accumulate dirt, dust, etc. Or, in special weather conditions such as rain, water droplets may adhere, causing the monitoring image to be blurry and affecting the normal use of the camera. For example, in a forest scene, water droplets on the lens appear as a grayish-white patch with a dynamically changing outline, which is similar to smoke, causing the area to be misidentified as smoke. In addition, existing technologies can only determine whether the foreign objects on the camera lens are dirt, but cannot determine whether the foreign objects on the lens are dirt or water droplets, so they cannot accurately alert relevant personnel to take appropriate measures.

[0061] To address the aforementioned issues, this application proposes a method and apparatus for identifying foreign objects in camera lenses. The method and apparatus provided by this application can detect the presence of foreign objects in a camera lens and identify whether the foreign object is a water droplet or a stain, enabling relevant personnel to take appropriate measures. Furthermore, the method and apparatus provided by this application employ different methods to determine the presence of foreign objects in both moving and non-moving scenarios. Based on the slow downward movement of water droplets, and combining the image information of the water droplets with their temporal motion information, a spatial and temporal dual-branch classification model is used to determine the presence of water droplets. Based on the characteristic that lens stains do not change significantly in a short period, the presence of lens stains is determined by combining the area change of polygonal regions and the classification results of a binary classification model. The classification model used in this application is robust to changes in lighting and environment, effectively filtering out interfering information in areas suspected to be water droplets or stains, reducing the probability of misidentifying background areas as foreign objects in the lens, and improving the accuracy of identification.

[0062] The method for identifying foreign objects in camera lenses provided in this application can be implemented by an execution device. In some embodiments, the execution device can be an electronic device, which can be implemented by one or more servers. Figure 1A Let's take a server with 100 servers as an example. (Reference) Figure 1AThe diagram illustrates a possible application scenario provided by an embodiment of this application, including an electronic device 100 and a data acquisition device 200. The server 100 can be implemented as a physical server or a virtual server. The server can be implemented as a single server or as a server cluster consisting of multiple servers; the event recognition method provided in this application can be implemented using either a single server or a server cluster. The data acquisition device 200 is a device with image acquisition capabilities, including electronic police equipment, electronic monitoring equipment, surveillance cameras, video recorders, and terminal devices with video acquisition capabilities (such as laptops, computers, mobile phones, and televisions). The data acquisition device 200 can send the acquired video to be detected to the server 100 via a network. Optionally, the server 100 can be connected to the terminal device 300, receiving a foreign object identification task for the camera lens sent by the terminal device 300, and performing foreign object identification for the camera lens based on the received video to be detected sent by the data acquisition device 200. In some scenarios, the server 100 can send the foreign object identification result for the camera lens to the terminal device 300. The terminal device 300 can be a television, mobile phone, tablet computer, personal computer, etc.

[0063] As an example, see Figure 1B As shown, server 100 may include processor 110, communication interface 120, and memory 130. Of course, server 100 may also include other components. Figure 1B Not shown in the image.

[0064] The communication interface 120 is used to communicate with the acquisition device 200 and the terminal device 300, to receive the video to be detected sent by the acquisition device 200, or to receive the camera lens foreign object identification task sent by the terminal device 300, or to send the camera lens foreign object identification result to the terminal device 300.

[0065] In the embodiments of this application, the processor 110 may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0066] Processor 110 is the control center of server 100, connecting various parts of server 100 through various interfaces and routes. It executes various functions and processes data by running or executing software programs and / or modules stored in memory 130, and by calling data stored in memory 130. Optionally, processor 110 may include one or more processing units. Processor 110 may be, for example, a processor, microprocessor, controller, or other control component. It may be a general-purpose central processing unit (CPU), a general-purpose processor, a digital signal processing unit (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0067] The memory 130 can be used to store software programs and modules. The processor 110 executes various functional applications and data processing by running the software programs and modules stored in the memory 130. The memory 130 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function, etc.; the data storage area may store data created according to business processing, etc. As a non-volatile computer-readable storage medium, the memory 130 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The memory 130 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. The memory 130 is any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory 130 in this embodiment can also be a circuit or any other means capable of implementing storage functions for storing program instructions and / or data.

[0068] In other embodiments, the execution device may be a terminal device. In some scenarios, the terminal device can receive the video to be detected sent by the acquisition device and identify foreign objects in the camera lens based on the video. The terminal device may include a display device, which may be a liquid crystal display, an organic light-emitting diode (OLED) display, a projection display device, etc., and this application does not specifically limit it.

[0069] It should be noted that the above Figure 1A and Figure 1B The structure shown is merely an example, and the embodiments of this application are not limited thereto.

[0070] This application provides a method for identifying foreign objects in a camera lens. Figure 2 An exemplary flowchart illustrates the process of a method for identifying foreign objects in a camera lens, which can be executed by a foreign object identification device for the camera lens, the device being located in, for example... Figure 1B The server 100 shown could be, for example, a processor 110 or the server 100 itself. The foreign object detection device for the camera lens can also be located in the terminal device. The specific process is as follows:

[0071] 201. Obtain the first and second video frames from the video stream captured by the camera.

[0072] In this system, the second video frame is the video frame to be detected, and the first video frame is the video frame preceding the second video frame. The first and second video frames are consecutive video frames captured by the acquisition device.

[0073] The data acquisition equipment can be electronic police equipment, electronic monitoring equipment, surveillance cameras, video recorders, and terminal devices with video acquisition capabilities (such as laptops, computers, mobile phones, and televisions).

[0074] For example, after the acquisition device acquires the video to be detected, the server obtains the first video frame and the second video frame from the acquisition device.

[0075] In some embodiments, the server receives a video file sent by the acquisition device, which includes the video to be detected. This video file can be an encoded video file. The server can then decode the received video file to obtain the video to be detected. Encoding the video effectively reduces its file size, facilitating transmission. This improves the video transmission speed, thereby increasing the efficiency of subsequent confirmation of video events. The encoded bitstream data can be obtained using any applicable method, including but not limited to: Real-Time Streaming Protocol (RTSP), Open Network Video Interface Forum (ONVIF) standards, or proprietary protocols.

[0076] 202, divide the first video frame and the second video frame into N sub-image blocks respectively.

[0077] 203, determine M1 sub-image blocks with suspected lens foreign objects from N sub-image blocks in the first video frame, and determine M2 sub-image blocks with suspected lens foreign objects from N sub-image blocks in the second video frame.

[0078] In some embodiments, when the camera is an action camera, determining M1 sub-image blocks with suspected lens foreign objects from N sub-image blocks in the first video frame, and M2 sub-image blocks with suspected lens foreign objects from N sub-image blocks in the second video frame, can be achieved as follows: Feature information matching is performed between the N sub-image blocks in the first video frame and the sub-image blocks in the second video frame to determine M1 and M2 sub-image blocks with suspected lens foreign objects from the N sub-image blocks in the first video frame, respectively. The M1 and M2 sub-image blocks have a matching relationship. Specifically, the similarity between the feature vector of the first sub-image block in the first video frame and the feature vectors of K sub-image blocks in the second video frame is determined. The K sub-image blocks include sub-image blocks in the second video frame that are in the same position as the first sub-image block in the first video frame, as well as sub-image blocks within their neighborhood. When the similarity between the feature vector of the first sub-image block and the feature vector of the second sub-image block among the K sub-image blocks is greater than a first set threshold, the first sub-image block is a sub-image block in the first video frame that is suspected to contain a lens foreign object, and the second sub-image block is a sub-image block in the second video frame that is suspected to contain a lens foreign object.

[0079] As an example, the first sub-image block in the first video frame can be the i-th sub-image block. A third neural network model can be used to extract features from the i-th sub-image block in the first video frame, the i-th sub-image block in the second video frame, and the sub-image blocks in the lower neighborhood of the i-th sub-image block, respectively, to obtain the feature vector corresponding to the i-th sub-image block in the first video frame, as well as the feature vectors corresponding to the i-th sub-image block in the second video frame and the sub-image blocks in the lower neighborhood of the i-th sub-image block. Further, the similarity between the feature vector of the i-th sub-image block in the first video frame and the feature vector of the i-th sub-image block in the second video frame, as well as the similarity between the feature vector of the i-th sub-image block in the first video frame and the feature vectors corresponding to the sub-image blocks in the lower neighborhood of the i-th sub-image block in the second video frame, can be determined. When the similarity is greater than a first preset threshold, the two sub-image blocks corresponding to the similarity are determined to be the sub-image blocks in the first video frame and the second video frame respectively containing suspected lens foreign objects. For example, if the similarity between the feature vector of the i-th sub-image block in the first video frame and the feature vector of the i-th sub-image block in the second video frame is greater than a first set threshold, then the i-th sub-image block in the first video frame is determined to be a sub-image block in the first video frame that has a suspected lens foreign object, and the i-th sub-image block in the second video frame is determined to be a sub-image block in the second video frame that has a suspected lens foreign object.

[0080] In one possible example, the structure of the third neural network model can be multiple layers of convolutional layers, downsampling layers, and pooling layers. After inputting the i-th sub-image block from the first video frame into the third neural network model, the third neural network model extracts features from the i-th sub-image block in the first video frame to obtain the feature vector corresponding to the i-th sub-image block in the first video frame. The i-th sub-image patch in the second video frame is input into the third neural network model for feature extraction, resulting in the feature vector corresponding to the i-th sub-image patch in the second video frame. The similarity between the feature vector of the i-th sub-image block in the first video frame and the feature vector of the i-th sub-image block in the second video frame satisfies the condition described in the following formula:

[0081]

[0082] Here, cosθ represents the similarity between the feature vector of the i-th sub-image block in the first video frame and the feature vector of the i-th sub-image block in the second video frame. The value of cosθ ranges from [-1, 1]. The larger the value of cosθ, the higher the similarity.

[0083] In some embodiments, a first preset threshold can be pre-set. When the similarity is greater than the first preset threshold, the two sub-image blocks corresponding to the similarity are determined to be sub-image blocks in the first video frame and the second video frame respectively that contain suspected lens foreign objects. For example, the first preset threshold can be expressed as cosθ. th The similarity between the feature vector of the i-th sub-image block in the first video frame and the feature vector of the i-th sub-image block in the second video frame is cosθ. When cosθ ≥ cosθ th If the i-th sub-image block in the first video frame is determined to be the sub-image block with a suspected lens foreign object in the first video frame, and the i-th sub-image block in the second video frame is determined to be the sub-image block with a suspected lens foreign object in the second video frame. Further, the similarity between the feature vector of the i-th sub-image block in the first video frame and the feature vectors corresponding to the sub-image blocks in the lower neighborhood of the i-th sub-image block in the second video frame can be used to determine the sub-image block with a suspected lens foreign object in the second video frame. For example, as... Figure 3 As shown, Figure 3 This diagram illustrates the segmentation of two consecutive video frames. During image matching, as an example, the sub-image block labeled 0 in the left image can be matched with the sub-image block at the same position (labeled 0) in the right image, as well as the sub-image blocks in its lower neighborhood (labeled 0, 1, 2, 3), to determine whether the sub-image block contains a suspected lens foreign object. In some scenarios, when the lens foreign object has not moved or has only moved slightly, the sub-image block labeled 0 in the left image only has a similarity greater than a set threshold with the sub-image block labeled 0 in the right image; therefore, the sub-image block labeled 0 is considered to contain a suspected lens foreign object. In other embodiments, when the lens foreign object has moved significantly in the vertical direction, the sub-image block labeled 0 in the left image has a similarity greater than a first set threshold with the sub-image blocks labeled 0 and 2 in the right image; therefore, the sub-image blocks labeled 0 and 2 are both considered to contain a lens foreign object.

[0084] In some embodiments, when the camera is a still camera, determining M1 sub-image blocks with suspected lens foreign objects from N sub-image blocks in the first video frame, and determining M2 sub-image blocks with suspected lens foreign objects from N sub-image blocks in the second video frame, can be achieved as follows: matching the feature information of the N sub-image blocks included in the first and second video frames respectively with images in a first sample set to determine the category corresponding to the N sub-image blocks included in the first and second video frames respectively. The categories include background, water droplets, or stains, and the first sample set includes multiple images of lens water droplets, lens stains, and background images. Further, the M1 sub-image blocks in the first video frame categorized as water droplets or stains can be considered as sub-image blocks with suspected lens foreign objects, and the M2 sub-image blocks in the second video frame categorized as water droplets or stains can be considered as sub-image blocks with suspected lens foreign objects.

[0085] As an example, feature information matching can be performed on sub-image blocks in the first and second video frames and images in the first sample set, and the category of the image with the highest similarity to the sub-image block in the first sample set can be taken as the category of the sub-image block. Further, based on sub-image blocks corresponding to the categories of lens water droplets or lens stains, M1 sub-image blocks in the first video frame and M2 sub-image blocks in the second video frame suspected of containing lens foreign objects can be determined.

[0086] 204. Perform contour search on M1 sub-image blocks to determine the first contour region of the suspected lens foreign object, perform contour search on M2 sub-image blocks to determine the second contour region of the suspected lens foreign object, and determine the bounding rectangle region of the second contour region.

[0087] In some embodiments, after identifying M1 sub-image blocks in the first video frame that contain suspected lens foreign objects and M2 sub-image blocks in the second video frame that contain suspected lens foreign objects, contour lookup is performed on the M1 sub-image blocks to determine a first contour region of the suspected lens foreign object, and contour lookup is performed on the M2 sub-image blocks to determine a second contour region of the suspected lens foreign object, and the bounding rectangle region of the second contour region is determined. As an example, such as... Figure 4 As shown, grayscale sub-image blocks represent sub-image blocks in the first video frame where suspected lens foreign objects exist. Adjacent sub-image blocks with suspected lens foreign objects can be merged, and contour lookup is performed within the region composed of multiple adjacent sub-image blocks with suspected lens foreign objects to determine the first contour region of the suspected lens foreign object in the first video frame. For example, contour lookup is performed within the region composed of multiple adjacent sub-image blocks with suspected lens foreign objects to determine the first contour region of the suspected lens foreign object in the first video frame, such as... Figure 5As shown. Similarly, the second contour region of the suspected lens foreign object in the second video frame can be determined in the same way. The first contour region and the second contour region exist. Furthermore, the circumscribed rectangular region of the suspected lens foreign object can be determined based on the second contour region.

[0088] 205. Determine the optical flow map of the suspected lens foreign object based on the first contour region and the second contour region.

[0089] In some embodiments, motion information can be represented by optical flow. In some scenarios, a pixel can be represented as P(x,y,t) (t represents time). When point P moves a distance of (dx,dy) at time t+dt, let μ and ν represent the velocity vectors along the horizontal and vertical directions, respectively. Then μ and ν satisfy the conditions shown in the following formula: u and v are the optical flow vectors of that pixel. The optical flow map of the suspected lens foreign object in the second contour region can be represented by the optical flow vector. Specifically, the optical flow vector of the suspected lens foreign object can be determined based on the pixels in the first contour region and the pixels in the second contour region to obtain the optical flow map of the suspected lens foreign object.

[0090] 206. Determine whether the suspected lens foreign object is a water droplet based on the image of the outer rectangular region of the second contour region and the optical flow diagram of the suspected lens foreign object.

[0091] In some embodiments, the optical flow map of the suspected lens foreign object and the bounding rectangle of the second contour region can be used as input to the first classification model to determine whether the suspected lens foreign object in the second video frame is a water droplet. In some scenarios, when it is determined that the type of lens foreign object is not a water droplet, the area change rate of the second contour region relative to the first contour region is determined, and the category of the second contour region is determined by the second classification model. When the area change rate is less than a second set threshold and the category of the second contour region determined by the second classification model is a stain, the suspected lens foreign object in the second contour region of the second video frame is determined to be a stain. Specifically, the area change rate satisfies the condition shown in the following formula:

[0092]

[0093] Where S represents the rate of change of area, S t This represents the area of ​​the polygonal region in the second video frame. This represents the area of ​​the polygonal region in the first video frame.

[0094] Furthermore, the second contour region is used as input to a second classification model to determine whether the suspected lens foreign object in the second video frame is a stain. In some scenarios, the second set threshold can be expressed as S. th When S≤S thFurthermore, when the category of the second contour region is determined to be a stain by the second classification model, the suspected lens foreign object in the second contour region is determined to be a stain.

[0095] In some embodiments, after determining the type of suspected lens foreign object, the identification result of the lens foreign object in the second video frame can be output. The identification result includes the type of lens foreign object and the coordinates of the bounding rectangle of the lens foreign object.

[0096] Based on the above scheme, different methods were used to initially determine the presence of foreign objects in the lens for cameras in different motion states. Further, using a first classification model, combined with image spatial features and the slow movement of lens droplets over time, it was determined whether the target was a lens droplet. The area change of the suspected lens region and the model classification results were then combined to determine whether the target was a lens stain. This method is unaffected by changes in lighting and environment, effectively filtering out interfering information in areas suspected to be lens droplets or lens stains, thus improving the accuracy of lens foreign object detection.

[0097] In some embodiments, the first classification model includes a spatial flow network and a temporal flow network, and the first classification model is trained based on a first training sample set. The first training sample set includes multiple samples, each of which includes multiple temporally consecutive images containing lens foreign objects and the type of lens foreign object corresponding to each sample. The training process of the first classification model is as follows: Figure 6 As shown, the details are as follows:

[0098] 601, Determine the optical flow map of the first image of the first sample.

[0099] In some embodiments, the first image is any image in the first sample. In some scenarios, the optical flow map of the first image can be the optical flow map of the first image and the image preceding it in time. In other scenarios, the optical flow map of the first image can also be the optical flow map determined by the first image and the next image preceding it in time. In some scenarios, the optical flow map of the first image can also be determined by the first image and any image in the first sample. In other scenarios, the optical flow map of the first image can also be determined by the first image and multiple images in the first sample. For example, the optical flow map of the first image can be determined by the first image and multiple images preceding it in the first sample.

[0100] In some embodiments, motion information can be represented by optical flow. In some scenarios, a pixel in the first image can be represented as P(x,y,t) (t represents time). When point P moves a distance of (dx,dy) at time t+dt, let μ and ν represent the velocity vectors along the horizontal and vertical directions, respectively. Then μ and ν satisfy the conditions shown in the following formula: u and v are the optical flow vectors of the pixel. The optical flow vectors are then superimposed to obtain the optical flow map of the first image.

[0101] 602, input the first image in the first sample into the spatial flow network and perform feature extraction, so as to output the first probability distribution of the lens foreign object type corresponding to the first image through the spatial flow network, and input the optical flow map into the temporal flow network, so as to output the second probability distribution of the lens foreign object type corresponding to the first image through the temporal flow network.

[0102] In some embodiments, spatial flow networks may include multiple convolutional layers, downsampling layers, pooling layers, two fully connected layers, and a softmax layer. Temporal flow networks may include multiple convolutional layers, downsampling layers, pooling layers, two fully connected layers, and a softmax layer. Figure 7 As shown. When the first image is input into the spatial flow network, it undergoes convolution and pooling operations. Finally, softmax is used to map the input to the probability value of the first image being predicted as a certain type of lens foreign object. The temporal flow network performs convolution and pooling operations on the optical flow information image, and finally, softmax is used to map the input to the probability value of the first image being predicted as a certain type of lens foreign object. The sum of the probability values ​​corresponding to each lens foreign object type is 1.

[0103] 603, determine the prediction type of the first image based on the first probability distribution and the second probability distribution.

[0104] In some embodiments, the first probability distribution and the second probability distribution can be fused, and the prediction type of the first image can be determined based on the fusion result. The fusion method can be to take the average value.

[0105] 604. Determine the first loss value based on the prediction type of the first image and the category of the first sample.

[0106] 605. Adjust the network parameters of the spatial flow network and the temporal flow network according to the first loss value to obtain the first classification model.

[0107] In some embodiments, when the first loss value is greater than a third preset threshold, the network parameters of the spatial flow network and the temporal flow network can be adjusted, and steps 601-604 above can be repeated until the first loss value is less than or equal to the third preset threshold. In some scenarios, when the first loss value is less than or equal to the third preset threshold, the network parameters are saved to obtain a first classification model.

[0108] In some embodiments, the second classification model is trained based on a second training sample set, which includes multiple lens smudge images, background images, and category labels for each image. The training process of the second classification model is as follows: Figure 8 As shown, the details are as follows:

[0109] 801, Input the second image from the second sample into the binary classification neural network, so that the binary classification neural network outputs the predicted type of the second image.

[0110] The second sample is any sample in the second training sample set, and the second sample includes the second image and the category corresponding to the second image.

[0111] 802, determine the second loss value based on the prediction type and the type of the second image in the second sample.

[0112] 803. The parameters of the binary classification neural network are adjusted based on the second loss value to obtain the second classification model.

[0113] In some embodiments, when the second loss value is greater than the fourth preset threshold, the network parameters of the binary classification network can be adjusted, and steps 801-802 above can be repeated until the second loss value is less than or equal to the fourth preset threshold. In some scenarios, when the second loss value is less than or equal to the fourth preset threshold, the network parameters are saved to obtain the second classification model.

[0114] Based on the same technical concept, this application provides a device for identifying foreign objects in a camera lens, see [link to relevant documentation]. Figure 9 As shown. This device can perform any step of the aforementioned method for identifying foreign objects in camera lenses; to avoid repetition, it will not be described again here. The device includes an acquisition module 901, a segmentation module 902, and a determination module 903.

[0115] The acquisition module 901 is used to acquire a first video frame and a second video frame in the video stream captured by the camera, wherein the second video frame is a video frame to be detected, and the interval between the first video frame and the second video frame is set to a certain duration.

[0116] The segmentation module 902 is used to divide the first video frame and the second video frame into N sub-image blocks respectively;

[0117] The determining module 903 is configured to: determine M1 sub-image blocks containing suspected lens foreign objects from N sub-image blocks in the first video frame; and determine M2 sub-image blocks containing suspected lens foreign objects from N sub-image blocks in the second video frame; perform contour lookup on the M1 sub-image blocks to determine a first contour region of the suspected lens foreign object; perform contour lookup on the M2 sub-image blocks to determine a second contour region of the suspected lens foreign object, and determine the circumscribed rectangular region of the second contour region; the first contour region and the second contour region overlap; determine the optical flow map of the suspected lens foreign object based on the first contour region and the second contour region; and determine whether the suspected lens foreign object is a water droplet based on the image of the circumscribed rectangular region of the second contour region and the optical flow map of the suspected lens foreign object.

[0118] In one possible implementation, the camera is an action camera, and the determining module 903, when determining M1 sub-image blocks with suspected lens foreign objects from N sub-image blocks in the first video frame, and M2 sub-image blocks with suspected lens foreign objects from N sub-image blocks in the second video frame, is specifically used for:

[0119] The N sub-image blocks in the first video frame are matched with the sub-image blocks in the second video frame to identify M1 sub-image blocks with suspected lens foreign objects from the N sub-image blocks in the first video frame, and M2 sub-image blocks with suspected lens foreign objects from the N sub-image blocks in the second video frame; wherein the M1 sub-image blocks and the M2 sub-image blocks have a matching relationship.

[0120] In one possible implementation, the determining module 903, when determining that the M1 sub-image blocks match the M2 sub-image blocks, is specifically configured to: determine the similarity between the feature vector of the first sub-image block in the first video frame and the feature vector of K sub-image blocks in the second video frame; the K sub-image blocks include sub-image blocks in the second video frame that are at the same position as the first sub-image block in the first video frame and sub-image blocks within their neighborhood; when the similarity between the feature vector of the first sub-image block and the feature vector of the second sub-image block in the K sub-image blocks is determined to be greater than a first preset threshold, the first sub-image block is a sub-image block in the first video frame that has a suspected lens foreign object, and the second sub-image block is a sub-image block in the second video frame that has a suspected lens foreign object.

[0121] In one possible implementation, the camera is a still camera. The determining module 903, when determining M1 sub-image blocks with suspected lens foreign objects from N sub-image blocks in the first video frame, and M2 sub-image blocks with suspected lens foreign objects from N sub-image blocks in the second video frame, specifically performs the following: matching the feature information of the N sub-image blocks included in the first and second video frames with images in a first sample set to determine the category corresponding to the N sub-image blocks included in the first and second video frames, where the category includes background, water droplets, or stains; wherein, the first sample set includes multiple lens water droplet images, lens stain images, and background images; and designating the M1 sub-image blocks in the first video frame with the category of water droplets or stains as sub-image blocks with suspected lens foreign objects, and the M2 sub-image blocks in the second video frame with the category of water droplets or stains as sub-image blocks with suspected lens foreign objects.

[0122] In one possible implementation, the determining module 903 is further configured to: when it is determined that the suspected lens foreign object is not a water droplet, determine the area change rate of the second contour region relative to the first contour region, and determine the category of the second contour region through a first classification model;

[0123] When the area change rate is less than the second set threshold and the category of the second contour region determined by the first classification model is a stain, the suspected lens foreign object in the second contour region of the second video frame is determined to be a stain.

[0124] In one possible implementation, the determining module 903, when determining whether the suspected lens foreign object is a water droplet based on the image of the circumscribed rectangular region of the second contour region and the optical flow map of the suspected lens foreign object, is specifically used to: use the image of the circumscribed rectangular region of the second contour region as input to the spatial flow network in the first classification model, and use the optical flow map of the suspected lens foreign object as input to the temporal flow network in the first classification model, so as to determine whether the suspected lens foreign object in the second contour region is a water droplet through the first classification model.

[0125] In one possible implementation, the first classification model includes a spatial flow network and a temporal flow network. The first classification model is trained based on a first training sample set, which includes multiple samples. Each of the multiple samples includes multiple temporally consecutive images containing lens foreign objects and the type of lens foreign object corresponding to each sample. The determining module 903 is further configured to obtain the first neural network model in the following manner:

[0126] Determine the optical flow map of the first image of the first sample among the plurality of samples;

[0127] The first image from the first sample is input into the spatial flow network to output a first probability distribution of the lens foreign object type corresponding to the first image. The optical flow map is input into the temporal flow network to output a second probability distribution of the lens foreign object type corresponding to the first image. The first sample includes a first image and the type of lens foreign object corresponding to the first image. The predicted type of the first image is determined based on the first and second probability distributions. A first loss value is determined based on the predicted type of the first image and the category of the first sample. The network parameters of the spatial flow network and the temporal flow network are adjusted based on the first loss value to obtain the first classification model.

[0128] In one possible implementation, the device further includes an output module 904, which is used to output the detection result of a suspected lens foreign object in the second contour region of the second video frame. The detection result includes the type of lens foreign object and the coordinates of the smallest bounding rectangle region of the lens foreign object.

[0129] Based on the same technical concept, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform any step in the above-described method for identifying foreign objects in a camera lens.

[0130] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0132] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0133] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0134] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for identifying foreign objects in a camera lens, characterized in that, The method comprises the following steps: acquiring a first video frame and a second video frame in a video stream collected by a camera, the second video frame being a video frame to be detected, and the first video frame and the second video frame being separated by a set time interval; dividing the first video frame and the second video frame into N sub-image blocks respectively, N being a positive integer; determining M1 sub-image blocks in which there is a suspected lens foreign matter from the N sub-image blocks in the first video frame, and determining M2 sub-image blocks in which there is a suspected lens foreign matter from the N sub-image blocks in the second video frame; performing contour searching on the M1 sub-image blocks to determine a first contour region of the suspected lens foreign matter, performing contour searching on the M2 sub-image blocks to determine a second contour region of the suspected lens foreign matter, and determining a circumscribed rectangular region of the second contour region; the first contour region and the second contour region overlap; determining an optical flow map of the suspected lens foreign matter according to the first contour region and the second contour region; determining whether the suspected lens foreign matter is a water droplet according to an image of the circumscribed rectangular region of the second contour region and the optical flow map of the suspected lens foreign matter, specifically comprising: taking the image of the circumscribed rectangular region of the second contour region as an input of a spatial flow network in a first classification model to output a first predicted probability distribution of a lens foreign matter type corresponding to the second contour region through the spatial flow network, and taking the optical flow map of the suspected lens foreign matter as an input of a temporal flow network in the first classification model to output a second predicted probability distribution of the lens foreign matter type corresponding to the second contour region through the temporal flow network; determining whether the suspected lens foreign matter is a water droplet based on the first predicted probability distribution and the second predicted probability distribution.

2. The method of claim 1, wherein, The camera is a motion camera, and the determination of the M1 sub-image blocks in which there is a suspected lens foreign matter from the N sub-image blocks in the first video frame and the determination of the M2 sub-image blocks in which there is a suspected lens foreign matter from the N sub-image blocks in the second video frame comprise: respectively matching feature information of the N sub-image blocks in the first video frame with feature information of the sub-image blocks in the second video frame to determine the M1 sub-image blocks in which there is a suspected lens foreign matter from the N sub-image blocks in the first video frame and determine the M2 sub-image blocks in which there is a suspected lens foreign matter from the N sub-image blocks in the second video frame; wherein the M1 sub-image blocks and the M2 sub-image blocks have a matching relationship.

3. The method of claim 2, wherein, The determination of the matching of the M1 sub-image blocks and the M2 sub-image blocks comprises: determining a similarity of a feature vector of a first sub-image block in the first video frame with feature vectors of K sub-image blocks in the second video frame; the K sub-image blocks include a sub-image block in the second video frame which has the same position as the first sub-image block in the first video frame and sub-image blocks within a neighborhood range; When it is determined that the similarity between the feature vector of the first sub-image block and the feature vector of a second sub-image block in the K sub-image blocks is greater than a first set threshold, the first sub-image block is a sub-image block in the first video frame that has a suspected lens foreign matter, and the second sub-image block is a sub-image block in the second video frame that has a suspected lens foreign matter.

4. The method of claim 1, wherein, The camera is a still camera, and determining M1 sub-image blocks in the N sub-image blocks in the first video frame that have a suspected lens foreign matter and determining M2 sub-image blocks in the N sub-image blocks in the second video frame that have a suspected lens foreign matter includes: Matching the N sub-image blocks included in the first video frame and the second video frame with feature information of a first sample set to determine categories corresponding to the N sub-image blocks included in the first video frame and the second video frame, the categories including background, water droplet, or stain; wherein the first sample set includes a plurality of lens water droplet images, lens stain images, and background images; Taking M1 sub-image blocks in the first video frame that have the category of water droplet or stain as sub-image blocks that have a suspected lens foreign matter, and taking M2 sub-image blocks in the second video frame that have the category of water droplet or stain as sub-image blocks that have a suspected lens foreign matter.

5. The method according to any one of claims 1 to 4, wherein The method further includes: When it is determined that the suspected lens foreign matter is not a water droplet, determining an area change rate of the second contour region relative to the first contour region, and determining a category of the second contour region by a first classification model; When the area change rate is less than a second set threshold and the category of the second contour region determined by the first classification model is a stain, determining that the suspected lens foreign matter in the second contour region in the second video frame is a stain.

6. The method according to any one of claims 1 to 4, wherein The first classification model is trained based on a first training sample set, the first training sample set includes a plurality of samples, each sample in the plurality of samples includes a plurality of time-continuous images including a lens foreign matter and a type of the lens foreign matter corresponding to the sample, and the method further includes: The first neural network model is obtained by: Determining an optical flow map of a first image of a first sample in the plurality of samples; Inputting the first image in the first sample into the spatial flow network to output a first probability distribution of the type of the lens foreign matter corresponding to the first image by the spatial flow network, and inputting the optical flow map into the temporal flow network to output a second probability distribution of the type of the lens foreign matter corresponding to the first image by the temporal flow network; wherein the first sample includes a plurality of time-continuous images including a lens foreign matter and a type of the lens foreign matter corresponding to the first sample; Determining a predicted type of the first image according to the first probability distribution and the second probability distribution; Determining a first loss value according to the predicted type of the first image and the category of the first sample; Adjusting network parameters of the spatial flow network and the temporal flow network according to the first loss value to obtain the first classification model.

7. The method according to any one of claims 1 to 4, wherein The method further includes: Output the detection result of the suspected lens foreign matter in the second contour region in the second video frame, the detection result including the type of the lens foreign matter and the coordinates of the minimum circumscribed rectangle region of the lens foreign matter.

8. A device for identifying foreign objects in a camera lens, characterized in that, The method comprises the steps of: obtaining a first video frame and a second video frame in a video stream collected by a camera, the second video frame being a video frame to be detected, and the first video frame being separated from the second video frame by a set time interval; segmenting the first video frame and the second video frame into N sub-image blocks respectively; determining M1 sub-image blocks in which suspected lens foreign matter exists from the N sub-image blocks in the first video frame, and determining M2 sub-image blocks in which suspected lens foreign matter exists from the N sub-image blocks in the second video frame; performing contour search on the M1 sub-image blocks to determine a first contour region of suspected lens foreign matter, performing contour search on the M2 sub-image blocks to determine a second contour region of the suspected lens foreign matter, and determining a circumscribed rectangle region of the second contour region; the first contour region and the second contour region overlap; determining an optical flow map of the suspected lens foreign matter according to the first contour region and the second contour region; determining whether the suspected lens foreign matter is a water droplet according to the image of the circumscribed rectangle region of the second contour region and the optical flow map of the suspected lens foreign matter, specifically comprising: inputting the image of the circumscribed rectangle region of the second contour region as an input of a spatial flow network in a first classification model, so as to output a first predicted probability distribution of the lens foreign matter type corresponding to the second contour region through the spatial flow network; and inputting the optical flow map of the suspected lens foreign matter as an input of a temporal flow network in the first classification model, so as to output a second predicted probability distribution of the lens foreign matter type corresponding to the second contour region through the temporal flow network; determining whether the suspected lens foreign matter is a water droplet based on the first predicted probability distribution and the second predicted probability distribution.

9. A device for identifying foreign objects in a camera lens, characterized in that, The method comprises the steps of: a memory for storing program instructions; a processor for calling the program instructions stored in the memory and executing the method according to any one of claims 1-7 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions, which, when executed on a computer, cause the computer to perform the method according to any one of claims 1-7.

Images