Vehicle-mounted camera attachment detection method, system and terminal

Abstract

The invention provides a vehicle-mounted camera attachment detection method, system and terminal, and the method comprises the steps: obtaining continuous video frames collected by a vehicle-mounted camera, and storing the continuous video frames in a frame list; preprocessing each video frame in the frame list; performing difference detection on all the preprocessed adjacent video frames, and identifying a pixel invariant region between the adjacent video frames based on a difference detection result; and carrying out multi-frame similarity comparison on the pixel invariant region to detect the existence of the attachment. The vehicle-mounted camera attachment detection speed and accuracy can be effectively improved, meanwhile, the requirement for computing resources is reduced, and the vehicle-mounted camera attachment detection method has the advantages of being high in detection precision, high in real-time performance, high in robustness, low in implementation cost, high in reliability and the like.

Description

Technical Field

[0001] The present application belongs to the field of automobile autonomous driving technology, and specifically relates to a method, system and terminal for detecting attachments of a vehicle-mounted camera. Background Art

[0002] With the rapid development of autonomous driving technology and advanced driver assistance systems (ADAS), the ability of vehicles to perceive the surrounding environment has become crucial. As a key component of the perception system, the quality of the video data provided by the vehicle camera directly affects the decision-making accuracy and driving safety of the system. However, the surface of the vehicle camera lens may be blurred or blocked due to the adhesion of rain, snow, ice, dust, insect residue or other contaminants, thus affecting the clarity and accuracy of the data. Therefore, timely detection and processing of these attachments is an important prerequisite for ensuring the normal operation of the camera and thus ensuring the performance of the autonomous driving and ADAS systems.

[0003] Traditional methods for detecting attachments using vehicle-mounted cameras usually rely on image processing techniques, such as edge detection and blur detection, to identify anomalies in images. These methods may be feasible in static or controlled environments, but they face multiple challenges in practical applications. First, these methods often require complex calculations and model analysis for each frame of the image, which not only consumes a lot of computing resources, but also makes it difficult to achieve real-time performance, and is not suitable for embedded systems that require fast response. Second, traditional methods have difficulty distinguishing between normal scene changes (such as weather changes, lighting changes) and abnormal changes caused by attachments, and are easily affected by environmental factors, resulting in false positives or missed positives. In addition, for dynamically changing driving environments, such as high-speed moving vehicles and constantly changing backgrounds, the stability and accuracy of traditional detection methods will also be greatly reduced. Summary of the invention

[0004] The purpose of the present application is to provide a method, system and terminal for detecting attachments by a vehicle-mounted camera, so as to improve the speed and accuracy of detecting attachments by a vehicle-mounted camera, while reducing the demand for computing resources.

[0005] In a first aspect, the present application provides a method for detecting attachments using a vehicle-mounted camera, comprising: obtaining continuous video frames captured by a vehicle-mounted camera and saving them to a frame list; preprocessing each video frame in the frame list; performing difference detection on all adjacent video frames that have been preprocessed, and identifying pixel-invariant regions between the adjacent video frames based on the difference detection results; and performing multi-frame similarity comparison on the pixel-invariant regions to detect the presence of attachments.

[0006] In an implementation of the first aspect, obtaining continuous video frames captured by a vehicle-mounted camera and saving them to a frame list includes:

[0007] When the vehicle speed exceeds a first preset threshold, controlling the vehicle-mounted camera to collect images at preset time intervals to obtain the continuous video frames;

[0008] The continuous video frames are sequentially saved into the frame list.

[0009] In an implementation of the first aspect, preprocessing each video frame in the frame list includes:

[0010] Removing a static unchanged area in the video frame that is captured due to a shooting angle limitation of the vehicle-mounted camera;

[0011] Performing grayscale processing on the effective area retained in the video frame to convert the original color image into a grayscale image;

[0012] The grayscale image is binarized to obtain a black-and-white image.

[0013] In an implementation of the first aspect, performing difference detection on all adjacent video frames that have been preprocessed, and identifying pixel-invariant regions between the adjacent video frames based on the difference detection results includes:

[0014] Calculating the absolute difference between pixels at corresponding positions of each group of adjacent video frames;

[0015] Determining whether the absolute difference is greater than a second preset threshold;

[0016] If yes, the location of the corresponding pixel is marked as a pixel change area;

[0017] Otherwise, the location of the corresponding pixel is marked as the pixel unchanged area.

[0018] In an implementation of the first aspect, performing a multi-frame similarity comparison on the pixel unchanged region to detect the presence of the attachment includes:

[0019] Constructing a set of contours of pixel-invariant regions between all adjacent video frames, and sequentially adding the set of contours to a list of pixel-invariant regions;

[0020] The similarity between two adjacent contour sets in the pixel unchanged region area list is calculated, and the existence of the attachment is detected based on the similarity calculation result.

[0021] In an implementation of the first aspect, constructing a contour set of pixel unchanged regions between adjacent video frames, and sequentially adding the contour set to a pixel unchanged region area list includes:

[0022] Extracting the contour of the pixel-invariant region;

[0023] Obtaining coordinate information and area of ​​the contour;

[0024] Determining whether there is a contour whose area is greater than a third preset threshold;

[0025] If so, all contours whose areas are greater than the third preset threshold are combined to form a contour set L i [L i1 ,L i2 ,…,L in ], and the contour set L i [L i1 ,L i2 ,…,L in ] is added to the list of pixel unchanged areas, where L in Represents the contour of the nth pixel unchanged region between the i-th group of adjacent video frames whose area is greater than a third preset threshold, i∈[1,N-1], N is the length of the frame list, and both N and n are natural numbers;

[0026] Otherwise, the contour set L of the pixel unchanged area is constructed based on the next set of adjacent video frames. i [L i1 ,L i2 ,…,L in ].

[0027] In an implementation method of the first aspect, it also includes: determining whether the length of the pixel unchanged region area list exceeds a fourth preset threshold; if so, removing the contour set that was first added to the pixel unchanged region area list; otherwise, constructing the contour set of the pixel unchanged region based on the next group of adjacent video frames until the length of the pixel unchanged region area list reaches the fourth preset threshold.

[0028] In an implementation of the first aspect, calculating the similarity between two adjacent contour sets in the pixel unchanged region area list, and detecting the presence of the attachment based on the similarity calculation result includes:

[0029] Based on the coordinate information of the contours, performing consistency matching on the contours in the two adjacent contour sets to obtain a plurality of contour pairs that match each other;

[0030] Calculating the similarity between the contours in each of the contour pairs;

[0031] Determine whether the similarity is greater than a fifth preset threshold for Q consecutive times, where Q is a natural number;

[0032] If yes, it is determined that there is an attachment on the vehicle-mounted camera;

[0033] Otherwise, it is determined that there is no attachment on the vehicle-mounted camera.

[0034] In a second aspect, the present application provides a vehicle-mounted camera attachment detection system, comprising: a video frame acquisition module, used to acquire continuous video frames captured by the vehicle-mounted camera and save them to a frame list; a preprocessing module, used to preprocess each video frame in the frame list; a difference detection module, used to perform difference detection on all adjacent video frames that have been preprocessed, and identify the pixel-invariant areas between the adjacent video frames based on the difference detection results; a similarity comparison module, used to perform multi-frame similarity comparison on the pixel-invariant areas to detect the presence of attachments.

[0035] In a third aspect, the present application provides a terminal, comprising: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the terminal executes any of the above-mentioned vehicle camera attachment detection methods.

[0036] As described above, the vehicle-mounted camera attachment detection method, system and terminal described in the present application have the following beneficial effects:

[0037] (1) High-precision detection: This application uses the dynamic differences between consecutive video frames and combines the multi-frame similarity comparison of pixel-invariant areas to effectively distinguish normal environmental changes from abnormal changes caused by attachments. Even in complex dynamic environments, it can maintain a high accuracy rate, thereby significantly improving the accuracy of detection.

[0038] (2) Strong real-time performance: The detection method of the present application is simple and efficient, supports real-time detection while the vehicle is driving, and can quickly respond to the presence of attachments.

[0039] (3) High robustness: The application has good adaptability to various environmental changes. Whether it is in light changes, different weather conditions (such as rainy or snowy days), or complex traffic scenes, it can maintain a high performance level.

[0040] (4) Low-cost implementation: This application only relies on the video frame data from the vehicle-mounted camera and does not require additional sensor equipment. This not only significantly reduces the system hardware cost, but also simplifies system design and maintenance.

[0041] (5) High reliability: This application can effectively ensure the clarity of the vehicle camera's field of view, ensure the correct perception and decision-making of the autonomous driving system and advanced driver assistance system (ADAS), reduce the risk of accidents, and improve overall driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1Shown is a schematic diagram of a hardware application scenario of the vehicle-mounted camera attachment detection method, system and terminal in one embodiment of the present application.

[0043] Figure 2 Shown is a schematic diagram of the structure of a mobile terminal in an embodiment of the present application.

[0044] Figure 3 Shown is a flow chart of an embodiment of the vehicle-mounted camera attachment detection method described in this application.

[0045] Figure 4 Shown is a flow chart of another embodiment of the vehicle-mounted camera attachment detection method described in this application.

[0046] Figure 5 Shown is a pixel correspondence diagram of adjacent video frames in one embodiment of the present application.

[0047] Figure 6 Shown is a flow chart of another embodiment of the vehicle-mounted camera attachment detection method described in this application.

[0048] Figure 7 Shown is a flow chart of another embodiment of the vehicle-mounted camera attachment detection method described in this application.

[0049] Figure 8 Shown is a schematic diagram of contour matching in an embodiment of the vehicle-mounted camera attachment detection method described in this application.

[0050] Fig. 9 Shown is a fisheye image of an embodiment of the vehicle-mounted camera attachment detection method described in this application.

[0051] Fig.10 Shown is a schematic structural diagram of an embodiment of the vehicle-mounted camera attachment detection system described in this application.

[0052] Fig.11 Shown is a schematic diagram of the structure of a terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0054] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0055] In addition, in this application, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0056] The following embodiments of the present application provide a method, system and terminal for detecting attachments on a vehicle camera, which can effectively improve the speed and accuracy of detecting attachments on a vehicle camera, while reducing the demand for computing resources. The technical solution of the present application is suitable for real-time detection of attachments on a vehicle camera during vehicle driving.

[0057] See also Figure 1 , which is a schematic diagram of a hardware application scenario of the vehicle-mounted camera attachment detection method, system and terminal described in this application in one embodiment. Figure 1 As shown in Figure 1, the vehicle's on-board monitoring system is usually equipped with on-board cameras facing the front, rear, left and right. As the core component of the environmental perception system, on-board cameras are widely used to capture key visual information about the vehicle's surroundings, including road conditions, pedestrians and neighboring vehicles, etc., in order to provide drivers with real-time driving environment data to improve driving safety.

[0058] Car cameras are usually connected to the car computer system, which is responsible for converting the captured visual information into data that can be understood and applied by the vehicle. For example, this data can be used in the automatic driving assistance system to realize functions such as driving record, blind spot monitoring, lane departure warning, etc.

[0059] During vehicle driving, the continuous dynamic changes in the environment cause the attachments (such as raindrops, dust or dirt, etc.) on the vehicle camera to remain relatively still. This application can effectively identify foreign objects attached to the vehicle camera by detecting the dynamic differences between adjacent video frames and combining the characteristics of unchanged pixels in the attachment area, and improve the accuracy and stability of attachment detection, thereby ensuring that the autonomous driving system and advanced driver assistance system (ADAS) obtain high-quality environmental perception data.

[0060] The vehicle-mounted camera attachment detection method provided in the embodiment of the present application can be run in a mobile terminal, a computer terminal, or similar devices. Taking running on the mobile terminal as an example, Figure 2 The hardware structure block diagram of the mobile terminal of the vehicle-mounted camera attachment detection method is as follows: Figure 2 As shown, the mobile terminal may include: a processor and a memory, the processor may be a central processing unit, and the memory is used to store data. Figure 2 The mobile terminal in the figure is only used as an example and does not limit the specific structure of the mobile terminal.

[0061] Optionally, the mobile terminal may further include: a communication transmission device and an input and output device.

[0062] Optionally, the memory may be used to store computer programs, such as software programs and modules of application software, and the memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0063] Optionally, the communication transmission device can be used to receive or send data via a network, which may include a wireless network provided by a communication provider of the mobile terminal. The communication transmission device may include a NIC (Network Interface Controller) that can be connected to other network devices through a base station so as to communicate with the Internet.

[0064] The technical solutions in the embodiments of the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application.

[0065] See also Figure 3 , which is a flow chart of an embodiment of the vehicle-mounted camera attachment detection method described in this application.

[0066] like Figure 3As shown, this embodiment provides a method for detecting attachments using a vehicle-mounted camera, comprising the following steps S100 to S400.

[0067] In step S100, continuous video frames captured by the vehicle-mounted camera are obtained and saved in a frame list.

[0068] In one embodiment of the present application, obtaining continuous video frames captured by a vehicle-mounted camera and saving them to a frame list includes: when the vehicle speed exceeds a first preset threshold, controlling the vehicle-mounted camera to perform image capture at a preset time interval to obtain the continuous video frames; and saving the continuous video frames in the frame list in sequence.

[0069] See also Figure 4 , which is a flow chart of another embodiment of the vehicle-mounted camera attachment detection method described in this application.

[0070] In the embodiment of the present application, the first preset threshold is defined as A, and the frame list is [1, 2, ..., N]. The image acquisition process is started only when the vehicle speed exceeds the first preset threshold A, which can effectively avoid invalid image acquisition when the vehicle is stationary or driving at a low speed, thereby improving the working efficiency and accuracy of the system.

[0071] It should be noted that the length N of the frame list is a variable parameter and can be adaptively adjusted according to the change of vehicle speed. For example, when the vehicle speed is 60km / h, N can be set to 10 frames. When the vehicle speed is 80km / h, considering that the road conditions change more rapidly at high speeds, in order to capture key road conditions information in a timely manner, N can be set to 5 frames. This dynamic adjustment strategy can effectively reduce the amount of calculation and improve real-time performance.

[0072] The preset time interval can be dynamically calculated based on the vehicle speed and the frame list length N, aiming to balance the image quality and the efficiency of storage space usage.

[0073] In step S200, each video frame in the frame list is pre-processed.

[0074] In an embodiment of the present application, pre-processing each video frame in the frame list includes the following steps S201 to S203.

[0075] In step S201, a static unchanged area in the video frame that is captured due to the shooting angle limitation of the vehicle-mounted camera is removed.

[0076] Since the installation angle of the vehicle camera is fixed, its viewing angle is also fixed, which will result in a part of the camera imaging process being static. Take the forward-facing vehicle camera in the vehicle monitoring system as an example. The camera is responsible for capturing the real-time image in front of the vehicle, but the license plate installed at the front of the vehicle may appear in a fixed position in the image, forming a static unchanging area. Such static unchanging areas are irrelevant for analyzing the vehicle's driving conditions and forward road conditions, and may even distract the driver or the system. Therefore, in order to improve the effectiveness of the image data, it is necessary to remove the static unchanging areas in the video frame.

[0077] In one embodiment of the present application, by applying the mask technology in the field of computer vision, the static unchanged area in the video frame can be accurately removed. The mask is similar to a filter, which sets the pixel values ​​of the static unchanged area to transparent or fills it with the background color, retaining only those parts containing important information, such as road conditions, pedestrians and other vehicles and other dynamic elements, thereby visually "removing" these static unchanged areas.

[0078] It should be noted that the masking technology used in this application can be replaced by other methods for removing static unchanged areas. Any technical means that can achieve similar effects can be applied within the protection scope of this application.

[0079] In step S202, grayscale processing is performed on the valid area retained in the video frame to convert the original color image into a grayscale image.

[0080] Specifically, the original video frame is usually a color image, and the grayscale processing can convert the values ​​of the three color channels of red, green, and blue (RGB) for each pixel in the color image into a single grayscale value. The grayscale value is usually obtained by weighted averaging the three color channels, where the weight of each channel may be different to reflect the sensitivity of the human eye to different colors. Each pixel in the converted grayscale image has only one brightness value and no longer contains color information, which greatly reduces the dimension and complexity of the data, reduces the demand for storage resources, improves the image processing speed, and facilitates subsequent image analysis and processing tasks.

[0081] In step S203, the grayscale image is binarized to obtain a black-and-white image.

[0082] In the embodiments of the present application, the area where the attachment is located usually appears as a lower grayscale value in the grayscale image, which may be because the material, color or other characteristics of the attachment absorb more light. Therefore, after the binarization process, these areas will appear black. In contrast, the background of the dynamic area, such as moving objects or changing lighting conditions, often has a higher grayscale value and therefore appears white after binarization.

[0083] Specifically, the binarization process includes selecting an appropriate threshold value, which can be globally fixed or dynamically adjusted according to different areas of the image. Once the threshold value is determined, all pixels with grayscale values ​​above the threshold value are set to white, while pixels below the threshold value are set to black. This binary representation method makes the boundary between the attachment and the background clearer.

[0084] In step S300, difference detection is performed on all the adjacent video frames that have been preprocessed, and pixel-invariant regions between the adjacent video frames are identified based on the difference detection results.

[0085] Under normal circumstances, continuous video frames captured by a camera should show consistent dynamic changes that reflect the activities in the scene, such as pedestrians walking, vehicles driving, etc. However, when attachments such as dust, water marks or other contaminants appear on the lens of the on-board camera, these elements will interfere with the normal dynamic change pattern. In order to identify and handle this situation, this application needs to perform difference detection on all adjacent video frames that have been pre-processed.

[0086] See also Figure 5 , which is a pixel correspondence diagram of adjacent video frames in one embodiment described in the present application.

[0087] In one embodiment of the present application, difference detection is performed on all adjacent video frames that have been preprocessed, and the pixel unchanged area between the adjacent video frames is identified based on the difference detection results, including: calculating the absolute difference of pixels at corresponding positions between each group of adjacent video frames; determining whether the absolute difference is greater than a second preset threshold; if so, marking the position of the corresponding pixel as a pixel change area; otherwise, marking the position of the corresponding pixel as a pixel unchanged area.

[0088] In the embodiment of the present application, the absolute difference is used to quantify the degree of change in the pixels at the corresponding positions of two adjacent video frames. Since the pixel values ​​of the areas where the attachments are located are usually close to zero, this indicates that the changes in these areas between consecutive frames will be very small or almost no change.

[0089] In order to clearly present the recognition results, the pixel change areas and pixel unchanged areas can be marked in the next frame of the adjacent video frames. This marking method can help users quickly understand the dynamic changes in the scene, so as to better identify and interpret key information. In this way, not only can the visualization of information be improved, but also an intuitive tool is provided to users to support the immediate understanding of dynamic scenes.

[0090] In step S400, a multi-frame similarity comparison is performed on the pixel-invariant region to detect the presence of attachments.

[0091] In an embodiment of the present application, performing a multi-frame similarity comparison on the pixel unchanged region to detect the presence of attachments includes the following steps S410 and S420.

[0092] In step S410, a contour set of pixel-invariant regions between all adjacent video frames is constructed, and the contour set is sequentially added to a pixel-invariant region area list.

[0093] See also Figure 6 , which is a flow chart of another embodiment of the vehicle camera attachment detection method described in the present application. According to the aforementioned known frame list of the present application, a total of N video frames are included, and theoretically (N-1) adjacent video frame pairs will be formed. For each adjacent video frame pair, a contour set of pixel-invariant regions between the adjacent video frames is constructed, and the contour set is sequentially added to the pixel-invariant region area list, including the following steps S411 and S415.

[0094] In step S411, the contour of the pixel-invariant region is extracted.

[0095] Specifically, the outline of the pixel unchanged region refers to a closed curve formed by outlining the edges of all the pixel unchanged regions gathered together in the video frame. The closed curve connects the pixel points located on the boundary between the pixel unchanged region and the pixel changed region, thereby generating a contour line that clearly defines the scope of the region.

[0096] In step S412, the coordinate information and area of ​​the contour are obtained.

[0097] In step S413, it is determined whether there is a contour whose area is greater than a third preset threshold.

[0098] In one embodiment of the present application, the third preset threshold is defined as S, and the value of S may be 500 pixels.

[0099] In step S414, if there are contours with an area greater than the third preset threshold, all contours with an area greater than the third preset threshold are combined to form a contour set L i [Li1 ,L i2 ,…,L in ], and the contour set L i [L i1 ,L i2 ,…,L in ] is added to the list of pixel unchanged areas, where L in Represents the contour of the nth pixel unchanged region between the i-th group of adjacent video frames whose area is greater than a third preset threshold, i∈[1,N-1], N is the length of the frame list, and N and n are both natural numbers.

[0100] In step S415, if there is no contour whose area is larger than the third preset threshold, a contour set L of the pixel unchanged area is constructed based on the next group of adjacent video frames. i [L i1 ,L i2 ,…,L in ].

[0101] In the embodiment of the present application, a group of adjacent video frames may include contours of multiple pixel-invariant regions of different sizes. Considering that smaller foreign objects (such as small black dots) do not affect the driving of the vehicle, the present application chooses not to process the contours of pixel-invariant regions with an area smaller than S to improve the accuracy and speed of detection.

[0102] In one embodiment of the present application, the pixel unchanged region area list is defined as M, and the value of M can be 10 or 20, which can be set according to the vehicle computing power.

[0103] Assuming that video frame N and video frame N-1 are the first group of adjacent video frames, and the first group of adjacent frames has contours of three pixel-invariant regions whose areas are larger than S, the contour set corresponding to the first group of adjacent video frames can be expressed as L1[L 11 ,L 12 ,L 13 ], and add L1 to M. Similarly, video frame N-1 and video frame N-2 are the second group of adjacent video frames, and the second group of adjacent video frames has two contours with an area greater than S, then the contour set corresponding to the second group of adjacent video frames can be expressed as L2[L 21 ,L 23 ], and add L2 to M. Video frame N-2 and video frame N-3 are the third group of adjacent video frames, and the third group of adjacent video frames has two groups of contours that meet the requirements, then the contour set corresponding to the third group of adjacent video frames can be expressed as L3[L 32 ,L 33], and add L3 to M. Video frame N-3 and video frame N-4 are the fourth group of adjacent video frames, and the fourth group of adjacent video frames has two groups of contours that meet the requirements, then the contour set corresponding to the fourth group of adjacent video frames can be expressed as L4[L 42 ,L 43 ], add L4 to M.

[0104] In one embodiment of the present application, the vehicle-mounted camera attachment detection method provided by the present application also includes: determining whether the length of the pixel unchanged region area list exceeds a fourth preset threshold; if so, removing the contour set that was first added to the pixel unchanged region area list; otherwise, constructing the contour set of the pixel unchanged region based on the next group of adjacent video frames until the length of the pixel unchanged region area list reaches the fourth preset threshold.

[0105] Taking into account the limited computing power of the system, in order to ensure the efficiency and effectiveness of the calculation, the present application implements a real-time monitoring and management strategy for the pixel unchanged area list M to maintain the length of the pixel unchanged area list M within a preset range. When the pixel unchanged area list M is close to its upper capacity limit, the system will automatically clear the contour set that was first added to M based on the first-in-first-out principle, thereby freeing up the necessary storage space for the new contour set. This mechanism not only ensures that the latest contour set can be processed in a timely manner, but also ensures that it will not affect other functions of the vehicle. This implementation method optimizes the use of resources without sacrificing performance to ensure efficient operation of the system.

[0106] In step S420, the similarity between two adjacent contour sets in the pixel unchanged region area list is calculated, and the presence of the attachment is detected based on the similarity calculation result.

[0107] See also Figure 7 , which is a flow chart of another embodiment of the vehicle-mounted camera attachment detection method described in this application.

[0108] like Figure 7 As shown, in one embodiment of the present application, calculating the similarity between two adjacent contour sets in the pixel unchanged region area list, and detecting the presence of the attachment based on the similarity calculation result includes the following steps S421 to S424.

[0109] In step S421, based on the coordinate information of the contours, the contours in the two adjacent contour sets are matched for consistency to obtain a plurality of mutually matching contour pairs.

[0110] See also Figure 8 , which is a schematic diagram of contour matching of an embodiment of the vehicle-mounted camera attachment detection method described in this application.

[0111] like Figure 8 As shown, according to L1[L 11 ,L 12 ,L 13 ] and L2[L 21 ,L 23 ], we can determine the coordinate information of L 11 and L 21 Closest, L 13 and L 23 The closest. Therefore, it is determined that L 11 and L 21 Match each other and can form a contour pair (L 11 , L 21 );L 13 and L 23 Match each other to form a contour pair (L 13 , L 23 ); Since there is no 12 The contours with the closest coordinates, therefore cannot form a contour pair.

[0112] By analogy, according to L2[L 21 ,L 23 ] and L3[L 32 ,L 33 ], we can determine the coordinate information of L 23 and L 33 is closest, so we assume that L 23 and L 33 Match each other and can form a contour pair (L 23 , L 33 ). According to L3[L 32 ,L 33 ] and L4[L 42 ,L 43 ], we can determine the coordinate information of L 32 and L 42 Closest, L 33 and L 43 The closest. Therefore, it is determined that L 32 and L 42 Match each other to form a contour pair (L 32 , L 42 );L 33 and L 43 Match each other to form a contour pair (L 33 , L 43 ).

[0113] In step S422, the similarity between the contours in each contour pair is calculated.

[0114] In one embodiment of the present application, the similarity between the contours in each contour pair can be calculated by any similarity measurement method. The similarity measurement method includes but is not limited to structural similarity measurement (Structural Similarity Index, SSIM), cosine similarity algorithm, histogram-based algorithm and mutual information (Mutual Information, MI)-based algorithm.

[0115] In step S423, it is determined whether the similarity is greater than a fifth preset threshold for Q consecutive times, where Q is a natural number.

[0116] In an embodiment of the present application, the fifth preset threshold is defined as F, and the value of F may be 90%.

[0117] It should be noted that the fifth preset threshold F is a variable parameter, which can be dynamically adjusted according to environmental conditions. For example, on rainy days, F can be set to 80%.

[0118] This implementation can more flexibly adapt to different environmental changes, improve the sensitivity of system detection, ensure the effectiveness and reliability of the detection process, and ensure that a high level of recognition accuracy can be maintained even under adverse weather conditions.

[0119] In step S424, if the similarity is greater than the fifth preset threshold for Q consecutive times, it is determined that there is an attachment on the vehicle-mounted camera; otherwise, it is determined that there is no attachment on the vehicle-mounted camera.

[0120] In one embodiment of the present application, L 11 and L 21 The similarity between them is 70%, L 13 and L 23 The similarity between them is 90%, L 23 and L 33 The similarity between them is 91%, L 33 and L 43 The similarity between them is 92%, the value of Q is 3, and the value of the fifth preset threshold F is 90%.

[0121] Due to L 13 With L 23 , L 33 With L 43 If the similarity between consecutive frames exceeds 90% three times, then it can be judged that 13 , L 23 , L 33 and L 43 There are attachments in the corresponding pixel unchanged area.

[0122] In this implementation, by continuously observing and evaluating the changes in similarity, environmental interference can be effectively eliminated, and pixel loss caused by attachments can be accurately detected, thereby improving the accuracy and reliability of attachment detection in video frames.

[0123] In one embodiment of the present application, the vehicle-mounted camera attachment detection method provided by the present application further includes: mapping the coordinates of the detected attachments onto a fisheye image for visualization. Fig. 9 , showing a fisheye image of an embodiment of the vehicle-mounted camera attachment detection method described in this application.

[0124] As a special form of panoramic image, fisheye images can show a wider viewing angle than traditional cameras. Therefore, by marking the exact location of the attachment on the fisheye image, it can help maintenance personnel quickly locate and remove the attachment, ensuring that the camera can continue to maintain the best operating state, facilitating vehicle maintenance and research, and providing strong data support for the decision-making process of the autonomous driving system.

[0125] After identifying the attachment to the camera, this application further has the function of triggering an alarm or initiating removal measures. For example, once the presence of attachments is detected, the system can automatically activate the cleaning program to ensure the cleanliness of the camera lens, thereby maintaining the clarity and accuracy of the monitoring video. This function not only improves the autonomous maintenance capability of the camera system, but also ensures the continued high quality of monitoring data, which is crucial to ensuring vehicle driving safety and improving the decision-making efficiency of the autonomous driving system.

[0126] It should be noted that the protection scope of the vehicle-mounted camera attachment detection method described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0127] See also Fig.10 , which is a schematic diagram of the structure of an embodiment of the vehicle-mounted camera attachment detection system described in the present application.

[0128] like Fig.10 As shown, the present application provides a vehicle-mounted camera attachment detection system, including a video frame acquisition module, a preprocessing module, a difference detection module and a similarity comparison module.

[0129] The video frame acquisition module is used to acquire continuous video frames captured by the vehicle-mounted camera and save them to a frame list.

[0130] The preprocessing module is used to preprocess each video frame in the frame list.

[0131] The difference detection module is used to perform difference detection on all adjacent video frames that have been preprocessed, and identify pixel-invariant regions between the adjacent video frames based on the difference detection results.

[0132] The similarity comparison module is used to perform multi-frame similarity comparison on the pixel unchanged area to detect the existence of attachments.

[0133] It should be noted that the structures and principles of the video frame acquisition module, preprocessing module, difference detection module and similarity comparison module described in the embodiment of the present application correspond one-to-one to the steps in the above-mentioned vehicle camera attachment detection method, so they will not be repeated here.

[0134] The vehicle-mounted camera attachment detection system provided in the embodiment of the present application can implement the vehicle-mounted camera attachment detection method described in the present application, but the implementation device of the vehicle-mounted camera attachment detection method described in the present application includes but is not limited to the structure of the vehicle-mounted camera attachment detection system listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0135] See also Fig.11 , showing a schematic diagram of the structure of a terminal in one embodiment described in the present application.

[0136] like Fig.11 As shown, the present application provides a terminal, including a processor and a memory.

[0137] The memory is used to store computer programs.

[0138] The processor is used to execute the computer program stored in the memory so that the terminal executes any one of the above-mentioned methods for detecting attachments by a vehicle-mounted camera.

[0139] Preferably, the processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk.

[0140] This embodiment also includes one or more of a multimedia component, an input / output (I / O) interface, and a communication component.

[0141] The multimedia component may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in a memory or sent through a communication component. The audio component also includes at least one speaker for outputting audio signals. The I / O interface provides an interface between the processor and other interface modules, and the other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component is used for wired or wireless communication between the timer and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component may include: a Wi-Fi module, a Bluetooth module, an NFC module.

[0142] In one embodiment, the timer can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned drone network deployment and collaborative caching methods.

[0143] In the several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules or units, which can be electrical, mechanical or other forms.

[0144] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in the various embodiments of the present application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0145] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0146] The present application embodiment also provides a computer-readable storage medium. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by a program to instruct the processor, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state hard disk (SSD)), etc.

[0147] The present application embodiment may also provide a computer program product, the computer program product including one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer or data center.

[0148] When the computer program product is executed by a computer, the computer executes the method described in the above method embodiment. The computer program product may be a software installation package, and when the above method is required, the computer program product may be downloaded and executed on a computer.

[0149] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0150] In summary, the present application provides a method, system and terminal for detecting attachments of a vehicle-mounted camera, the method comprising obtaining continuous video frames collected by a vehicle-mounted camera and saving them to a frame list; preprocessing each video frame in the frame list; performing difference detection on all adjacent video frames that have been preprocessed, and identifying the pixel-invariant regions between the adjacent video frames based on the difference detection results; and performing multi-frame similarity comparison on the pixel-invariant regions to detect the presence of attachments. The present application can effectively distinguish normal environmental changes from abnormal changes caused by attachments by utilizing the dynamic differences between consecutive frames and combining the multi-frame similarity comparison of the pixel-invariant regions. Even in a complex dynamic environment, a high accuracy rate can be maintained, thereby significantly improving the accuracy of detection. The detection method of the present application is simple and efficient, supports real-time detection during vehicle driving, and can quickly respond to the presence of attachments, thereby ensuring driving safety. The present application shows good adaptability to various environmental changes. Whether in light changes, different weather conditions (such as rainy or snowy days), or complex traffic scenes, a high performance level can be maintained. The present application relies only on the video frame data of the vehicle-mounted camera, without the need for additional sensor equipment. This not only significantly reduces the system hardware cost, but also simplifies system design and maintenance. This application can effectively ensure the clarity of the vehicle camera's field of view, ensure the correct perception and decision-making of the autonomous driving system and advanced driver assistance system (ADAS), reduce accident risks, and improve overall driving safety.

[0151] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A method for detecting attachments of a vehicle-mounted camera, characterized in that: include: Get the continuous video frames captured by the vehicle camera and save them to the frame list; preprocessing each video frame in the frame list; Performing difference detection on all adjacent video frames that have been preprocessed, and identifying pixel-invariant regions between the adjacent video frames based on the difference detection results; A multi-frame similarity comparison is performed on the pixel-invariant region to detect the presence of attachments.

2. The method according to claim 1, characterized in that Get the continuous video frames captured by the vehicle camera and save them to the frame list including: When the vehicle speed exceeds a first preset threshold, controlling the vehicle-mounted camera to collect images at preset time intervals to obtain the continuous video frames; The continuous video frames are sequentially saved into the frame list.

3. The method according to claim 1, characterized in that Preprocessing each video frame in the frame list includes: Removing a static unchanged area in the video frame that is captured due to a shooting angle limitation of the vehicle-mounted camera; Performing grayscale processing on the effective area retained in the video frame to convert the original color image into a grayscale image; The grayscale image is binarized to obtain a black-and-white image.

4. The method according to claim 1, characterized in that: Performing difference detection on all adjacent video frames that have been preprocessed, and identifying the pixel-invariant regions between the adjacent video frames based on the difference detection results includes: Calculating the absolute difference between pixels at corresponding positions of each group of adjacent video frames; Determining whether the absolute difference is greater than a second preset threshold; If yes, the location of the corresponding pixel is marked as a pixel change area; Otherwise, the location of the corresponding pixel is marked as the pixel unchanged area.

5. The method according to claim 1, characterized in that Performing a multi-frame similarity comparison on the pixel unchanged region to detect the presence of the attachment includes: Constructing a set of contours of pixel-invariant regions between all adjacent video frames, and sequentially adding the set of contours to a list of pixel-invariant regions; The similarity between two adjacent contour sets in the pixel unchanged region area list is calculated, and the existence of the attachment is detected based on the similarity calculation result.

6. The method according to claim 5, characterized in that Constructing a contour set of pixel-invariant regions between adjacent video frames, and sequentially adding the contour set to a pixel-invariant region area list includes: Extracting the contour of the pixel-invariant region; Obtaining coordinate information and area of ​​the contour; Determining whether there is a contour whose area is greater than a third preset threshold; If so, all contours whose areas are greater than the third preset threshold are combined to form a contour set L i [L i1 ,L i2 ,…,L in ], and the contour set L i [L i1 ,L i2 ,…,L in ] is added to the list of pixel unchanged areas, where L in Represents the contour of the nth pixel unchanged region between the i-th group of adjacent video frames whose area is greater than a third preset threshold, i∈[1,N-1], N is the length of the frame list, and both N and n are natural numbers; Otherwise, the contour set L of the pixel unchanged area is constructed based on the next set of adjacent video frames. i [L i1 ,L i2 ,…,L in ].

7. The method according to claim 6, characterized in that Also includes: Determine whether the length of the pixel unchanged region area list exceeds a fourth preset threshold; if so, remove the contour set that was first added to the pixel unchanged region area list; Otherwise, a contour set of the pixel-unchanged region is constructed based on the next group of adjacent video frames until the length of the pixel-unchanged region area list reaches the fourth preset threshold.

8. The method according to claim 5, characterized in that Calculating the similarity between two adjacent contour sets in the pixel unchanged region area list, and detecting the presence of the attachment based on the similarity calculation result includes: Based on the coordinate information of the contours, the contours in the two adjacent contour sets are matched for consistency to obtain a plurality of contour pairs that match each other; Calculating the similarity between the contours in each of the contour pairs; Determine whether the similarity is greater than a fifth preset threshold for Q consecutive times, where Q is a natural number; If yes, it is determined that there is an attachment on the vehicle-mounted camera; Otherwise, it is determined that there is no attachment on the vehicle-mounted camera.

9. A vehicle-mounted camera attachment detection system, characterized in that: include: The video frame acquisition module is used to obtain continuous video frames collected by the vehicle-mounted camera and save them to the frame list; A preprocessing module, used for preprocessing each video frame in the frame list; A difference detection module, used to perform difference detection on all adjacent video frames that have been preprocessed, and identify pixel-invariant regions between the adjacent video frames based on the difference detection results; The similarity comparison module is used to perform multi-frame similarity comparison on the pixel unchanged area to detect the existence of attachments.

10. A terminal, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory so that the terminal executes the vehicle-mounted camera attachment detection method according to any one of claims 1 to 8.

Images