Camera device occlusion detection method, device, storage medium, product and vehicle

Abstract

The present application relates to a camera device occlusion detection method, device, computer-readable storage medium, computer program product, and vehicle, which obtains video information including at least two image frames collected by a shooting device; and determines the occlusion detection result of the camera device according to the time sequence characteristics of the image frames in the video information. The present application can improve the accuracy of camera device occlusion detection.

Description

Technical Field

[0001] The present application relates to the field of image detection technology, and in particular to a camera device occlusion detection method, device, computer-readable storage medium, computer program product, and vehicle. Background Art

[0002] On-board cameras exposed outside the car (such as surround-view fisheye cameras) are very easily contaminated by elements in extreme environments, such as raindrops, mud, weeds, etc. attached to the lens surface, causing a serious degradation of image quality and greatly affecting the safety of the intelligent driving system.

[0003] Currently, it is limited to processing and identifying a single frame of image captured by the camera, or identifying multiple frames of image separately and then fusing the results to detect whether the camera is contaminated.

[0004] However, the above method will identify lens glare, snow, rain, etc. as dirty cameras, and the detection is prone to false detection. Summary of the invention

[0005] Embodiments of the present application provide a camera device occlusion detection method, device, computer-readable storage medium, computer program product, and vehicle, which improve the camera device occlusion detection accuracy.

[0006] In order to achieve the above-mentioned object, according to a first aspect of the present application, a camera device occlusion detection method is provided, the method comprising:

[0007] Acquire video information including at least two image frames collected by a shooting device;

[0008] The occlusion detection result of the camera device is determined according to the time sequence characteristics of the image frame in the video information.

[0009] Optionally, before the step of determining the time sequence characteristics of the image frames in the video information, the method further comprises:

[0010] Acquire sequence information of the image frames in the video information;

[0011] The image frames are sequentially encoded according to the sequence information of the image frames to obtain the time sequence characteristics of the image frames.

[0012] Optionally, determining the occlusion detection result of the camera device according to the time sequence characteristics of the image frame in the video information includes:

[0013] Extracting features from the image frames in the video information to obtain the image features; wherein the image features include: at least one of content features, shape features, structural features, pixel features, texture features, brightness features, and chromaticity features;

[0014] Performing correlation analysis on the image features according to the time sequence features of the image frames to obtain correlation feature information;

[0015] Based on the associated feature information, an occlusion detection result of the camera device is obtained.

[0016] Optionally, performing correlation analysis on the image features according to the time sequence features of the image frames to obtain correlation feature information includes:

[0017] Splitting the image frame to obtain sub-images of the image frame;

[0018] According to the time sequence characteristics of the image frame, a sub-image in the image frame is encoded to obtain an image position code;

[0019] An association analysis is performed based on the image position code and the image features to obtain association feature information.

[0020] Optionally, performing association analysis based on the image position code and the image feature to obtain association feature information includes:

[0021] Fusing the image feature with the image position code to obtain an image fusion feature;

[0022] Feature association analysis is performed based on the image fusion features to obtain association feature information between multiple image features.

[0023] Optionally, fusing the image feature and the image position code to obtain an image fusion feature includes:

[0024] Performing dimension normalization processing on the image features and the image position codes;

[0025] The dimensionally normalized image features and image position codes are concatenated to obtain image fusion features.

[0026] Optionally, obtaining the occlusion detection result of the camera device based on the associated feature information includes:

[0027] Performing weight and / or bias calculation according to the associated feature information to obtain a first probability;

[0028] When the first probability is greater than a preset probability threshold, it is determined that the camera device is blocked.

[0029] Optionally, obtaining video information includes:

[0030] Controlling the camera device to collect video data of the environment and counting the video recording time;

[0031] If the video recording time reaches a preset time threshold, the video recording is terminated and the video information is obtained.

[0032] Optionally, the method further includes:

[0033] Controlling the camera device to collect new video information at a preset time interval;

[0034] The occlusion detection result of the camera device is determined according to the time sequence characteristics of the image frames in the new video information.

[0035] Optionally, the method further includes:

[0036] When it is determined that the camera device is blocked, extracting pixel feature parameters corresponding to at least two of the image frames;

[0037] Determining the pollution type and pollution level information corresponding to the camera device according to the pixel characteristic parameters;

[0038] According to the pollution type and the pollution level information, the camera device is controlled to perform self-cleaning, and the self-cleaning camera device is controlled to collect new video information.

[0039] According to a second aspect of the present application, a detection device is provided, which includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the above-mentioned camera device occlusion detection method.

[0040] According to a third aspect of the present application, a computer-readable storage medium is provided, comprising a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps of the above-mentioned camera device occlusion detection method.

[0041] According to the fourth aspect of the present application, a computer program product is provided, comprising a computer program, wherein the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of the above-mentioned camera device occlusion detection method.

[0042] According to a fifth aspect of the present application, a vehicle is provided, including a detection device, or a computer product.

[0043] In the camera device occlusion detection method, device, computer-readable storage medium, computer program product, and vehicle of the embodiments of the present application, the camera device can collect video information including at least two image frames, and then the camera device occlusion detection result can be determined based on the timing characteristics of each image frame in the video information. The present application analyzes multiple image frames based on the timing characteristics of multiple image frames collected by the camera device, thereby avoiding false detection of camera dirt caused by environmental interference, eliminating interference factors in camera device occlusion detection, and improving the accuracy of camera device occlusion detection.

[0044] Other features and advantages of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0046] In order to more completely understand the present application and its beneficial effects, the following description will be given in conjunction with the accompanying drawings, wherein the same figure numbers represent the same parts in the following description.

[0047] Figure 1 It is a schematic diagram of the camera dirt detection process provided in the background technology of this application;

[0048] Figure 2-1 is a first schematic diagram of camera dirtiness definition provided in an exemplary embodiment of the present application;

[0049] Figure 2-2 is a second schematic diagram of camera dirtiness definition provided in an exemplary embodiment of the present application;

[0050] Figure 3 is a schematic diagram of a dirty camera scene provided in an exemplary embodiment of the present application;

[0051] Figure 4 is a schematic diagram of a first process of dirt detection provided in an exemplary embodiment of the present application;

[0052] Figure 5 is a schematic diagram of a prediction model structure provided in an exemplary embodiment of the present application;

[0053] Figure 6 is a schematic diagram of a second process of dirt detection provided in an exemplary embodiment of the present application;

[0054] Figure 7It is a schematic diagram of the structure of a vehicle provided in one embodiment of the present application. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0056] Camera devices, such as car cameras, are very susceptible to contamination from extreme environments due to long-term exposure to the external environment. For example, raindrops, mud, weeds, etc. adhere to the lens surface, causing serious degradation of image quality and seriously affecting the safety of intelligent driving systems.

[0057] Therefore, accurately identifying whether the camera is dirty is of great significance to vehicle intelligent driving.

[0058] According to the background technology of this application, the current technology is generally limited to processing and recognizing a single frame of image, or recognizing multiple frames of image separately and then fusing the results. For example, Figure 1 The existing dirt recognition flow chart shown is actually based on image-based depth detection to identify whether the lens is dirty.

[0059] However, in some special scenarios, such as lens glare, snow, rain, etc., the image quality itself is very poor, not due to dirt attached to the camera surface. Therefore, using current technology for camera dirt detection is prone to false detection problems, such as mistakenly identifying snow in front of the lens as dirt attached to the lens surface.

[0060] Therefore, in order to solve the above-mentioned problems and improve the accuracy of lens dirt recognition, the present application proposes a camera device occlusion detection method, device, vehicle, computer-readable storage medium and computer program product. By collecting and constructing dirty video of the camera, the prediction model is trained. The prediction model in the present application can effectively utilize the timing information between image frames and reduce the false detection rate of camera dirt.

[0061] The vehicle in this application may be a fuel vehicle, a plug-in hybrid vehicle or a new energy vehicle, etc., and this application does not make any specific limitations on this.

[0062] Specifically, the camera device occlusion detection method in the embodiment of the present application can be applied to a vehicle, which is equipped with a camera device. After the camera device collects audio and video data, it can send it to the image processor of the vehicle, and the image processor executes each step of the camera device occlusion detection method in the present application. In addition, the camera device occlusion detection method in the present embodiment can also be applied to other devices equipped with a camera device, such as drones, etc., without specific limitation.

[0063] It is worth noting that in this embodiment, Figure 2-1 and Figure 2-2 As shown, a schematic diagram of a dirty camera is defined. Figure 2-1 In this case, the camera is not dirty. Figure 2-2 The camera is dirty. Figure 2-1 and Figure 2-2 The key difference is whether the occluder is attached to the lens surface. If the occluder is attached to the lens surface, the camera is considered to be dirty; if the occluder is not attached to the lens surface, the camera is considered to be clean. However, in actual scenes, it is impossible to make a direct judgment based on the external conditions of the camera. It can only be judged indirectly through the images captured by the camera, which will cause some misjudgments. It is very difficult to judge with only a single frame of image.

[0064] For example, Figure 3 In the various scenarios shown, the existing lens dirt detection method will identify that the lenses corresponding to all images are dirty. However, Figure 3 The first row of images shows that the camera is dirty. Its lens is contaminated by water stains, weeds and mud, resulting in the loss of environmental information. The second row of images shows that there is no dirt attached to the camera. The snow, rain and glare on the lens in the environment can easily be misjudged as a dirty camera in a single frame image. At this time, only by combining the timing information between the previous and next image frames and performing dynamic detection can it be recognized that there is no dirt on the lens surface and it is very clean.

[0065] On this basis, the camera device occlusion detection method in the embodiment of the present application is as follows: Figure 4 As shown, the following steps may be specifically included:

[0066] S10, acquiring video information including at least two image frames captured by a shooting device;

[0067] It should be noted that in this embodiment, while the vehicle is driving, the user can manually trigger a detection instruction to check whether the camera device (which may specifically refer to a camera in subsequent embodiments) is dirty, or the vehicle can automatically trigger the detection instruction according to a built-in program, for example, triggering the detection instruction at a preset time interval.

[0068] In this way, the vehicle can receive the above detection instruction, and according to the detection instruction, control the camera device to collect video information containing at least two image frames.

[0069] Specifically, for example, in this embodiment, the vehicle can control the camera device to collect the target video of the vehicle's environment. According to the above description, since this embodiment is based on the time series relationship between multiple images to determine whether the camera is contaminated and there is dirt on the camera surface, there are at least two image frames in the video collected in this embodiment, so that the vehicle can accurately identify whether the camera is dirty based on the dynamic image characteristics.

[0070] S20: Determine an occlusion detection result of the camera device according to a time sequence feature of the image frame in the video information.

[0071] In this embodiment, after acquiring multiple image frames, the vehicle can extract the time sequence features of the image frames, and further, determine the occlusion detection result of the camera device according to the time sequence features.

[0072] Specifically, for example, the vehicle can determine the image position code of each image frame based on the timing characteristics, wherein the image position code is related to the position of the image frame in the video, and can be used to characterize the position characteristics of each image frame in the video. By using the position code of each image frame, the image characteristics of multiple sequentially arranged image frames can be analyzed to accurately identify whether the camera is contaminated.

[0073] Furthermore, the vehicle can fuse the image features of the image frame with the corresponding image position codes, and identify camera dirtiness based on the fused features. For example, in this embodiment, the fused image features and image position codes of each image frame can be input into a pre-trained temporal attention transformer encoder. It can be understood that the fused features of multiple image frames are input into the above encoder in parallel. Through the temporal attention transformer encoder, dynamic and continuous analysis is performed in combination with multiple image frames to accurately identify whether there is dirt on the camera surface.

[0074] Therefore, in an embodiment of the present application, the vehicle can receive a detection instruction for the camera device, and according to the detection instruction, the camera device can be controlled to collect at least two frames of image frames. After acquiring multiple image frames, the vehicle can obtain the timing information of the image frame, and then, according to the sequence information, the timing characteristics of the image frame can be determined, and the detection result of the camera device can be determined according to the image characteristics of the image frame and the timing characteristics of the image frame where the image characteristics are located. Therefore, compared with the prior art that only collects a single image for analysis or simply analyzes multiple images, the present application can first determine the position coding of the collected multiple image frames, and then determine the detection result of the camera device according to the image characteristics and position coding of the image frame. The present application combines the timing characteristics on the basis of the image characteristics, and uses the timing characteristics between multiple image frames to continuously and dynamically analyze the image characteristics of multiple image frames, thereby avoiding the false detection of camera dirt caused by the environment, eliminating the interference factors of the camera device occlusion detection, and improving the accuracy of the camera device occlusion detection.

[0075] In one embodiment, before the above S10, “determining the occlusion detection result of the camera device according to the time sequence characteristics of the image frame in the video information”, the following may also be included:

[0076] S30, obtaining sequence information of the image frames in the video information;

[0077] S40, sequentially encoding the image frames according to the sequence information of the image frames to obtain time sequence features of the image frames.

[0078] In this embodiment, after the vehicle obtains video information including at least two image frames, it can obtain the sequence information of each frame image in the video information, and then, according to the sequence information, each image frame can be sequentially encoded to obtain the timing characteristics of each image frame.

[0079] It can be understood that the timing characteristics of the image frame are related to the position of the image frame in the image sequence, and can be used to characterize the position of each image frame in the image sequence. By utilizing the timing characteristics of the image frame, the image characteristics of multiple frames arranged in sequence can be analyzed to detect whether the camera is contaminated. The video captured by the camera device contains multiple image frames, and the multiple image frames constitute an image sequence. The timing characteristics of each image frame can characterize the position characteristics of the image frame in the image sequence.

[0080] Furthermore, in the above S20, “determining the occlusion detection result of the camera device according to the time sequence characteristics of the image frame in the video information” may include:

[0081] S201, extracting features from the image frames in the video information to obtain the image features; wherein the image features include: at least one of content features, shape features, structural features, pixel features, texture features, brightness features, and chromaticity features;

[0082] S202, performing correlation analysis on the image features according to the time sequence features of the image frames to obtain correlation feature information;

[0083] S203: Obtain an occlusion detection result of the camera device based on the associated feature information.

[0084] In this embodiment, the vehicle can perform feature extraction on each image frame to obtain image features corresponding to the image frame, wherein the image features include but are not limited to content features, shape features, structural features, pixel features, texture features, brightness features and chromaticity features.

[0085] Furthermore, the image features corresponding to the multiple image frames may be correlated and analyzed based on the time sequence features of the image frames to obtain correlation feature information, and the occlusion detection result of the camera device may be determined based on the correlation feature information.

[0086] Among them, in this embodiment, the above-mentioned associated feature information may include changes in image features in multiple consecutive image frames. For example, in two adjacent time sequences, image frame 1 and image frame 2, according to the above-mentioned image features, dirt is recognized on the camera device in image frame 1, but the dirt is not recognized at the same position in image frame 2. At this time, it can be determined that the occlusion detection result of the camera device is that the camera device is not obstructed, and there is no dirt on the lens surface of the camera device, which may be caused by factors such as environmental interference.

[0087] In a specific embodiment, in the above S202, “performing correlation analysis on the image features according to the temporal features of the image frames to obtain correlation feature information” may include:

[0088] S2021, splitting the image frame to obtain sub-images of the image frame;

[0089] S2022, encoding a sub-image in the image frame according to a temporal feature of the image frame to obtain an image position code;

[0090] S2023: Perform association analysis based on the image position code and the image feature to obtain association feature information.

[0091] In this embodiment, for each image frame, the vehicle can split the image frame to obtain sub-images of the image frame. When splitting the image frame, it can be split according to a preset image size requirement (such as an image resolution requirement), or the number of corresponding sub-images can be determined based on the dimension of the image features in the image frame, without specifically limiting the splitting method.

[0092] Furthermore, each sub-image in the image frame can be encoded according to the timing characteristics of the image frame to obtain an image position code, so that the image position code of the image frame can include the position characteristics of each sub-image in the image frame. Compared with directly analyzing the entire image frame, this embodiment effectively improves the image analysis accuracy, so that the image position code of the image frame can be used to merge with the image characteristics to accurately analyze whether there is dirt attached to the camera device, thereby avoiding the erroneous identification of snow, rain, glare, etc. as dirt attached to the surface of the camera lens.

[0093] Furthermore, the vehicle can perform association analysis based on the image position code and the corresponding image features to obtain associated feature information, so that the associated feature information can be used to determine the occlusion detection result of the camera device.

[0094] In one embodiment, the encoding dimension of the image position encoding of the image frame is related to the model feature parameters of the prediction model (for example, different model architectures require position encodings of different dimensions, and the model may require higher-dimensional encodings to capture more complex temporal relationships). If additional information needs to be embedded in the position encoding, such as image resolution, frame rate, or other metadata, more dimensions may be required to represent this information. Alternatively, different tasks may require position encodings of different dimensions. For example, for video analysis tasks, higher-dimensional encodings may be required to capture complex relationships between frames. Therefore, this embodiment does not specifically limit the value of the encoding dimension of the image position encoding.

[0095] On this basis, if t is the temporal feature of the image frame (i.e. the position of the image frame in the image sequence), is the image position code corresponding to it, d is the dimension of the position code, d≡ 0(mod 2) (this relationship indicates that d is an even number), then the function That is the position encoding function. The image position encoding can be specifically defined as:

[0096]

[0097] Among them, i is the dimension information of the image frame, It is a frequency parameter that can be adjusted as needed to ensure the diversity and discrimination of the encoding.

[0098] In another embodiment, if Figure 5As shown, the prediction model can also be used to identify the occlusion detection result of the camera device. The prediction model in this embodiment can include a convolutional neural network (CNN), a temporal attention encoder (Temporal Attention Encoder Transformer, Transformer) and a multilayer perceptron (MLP) connected in sequence, wherein the model hierarchical structure of the temporal attention transformer encoder and the multilayer perceptron MLP can refer to the existing Transformer hierarchical structure and MLP structure, which will not be elaborated on.

[0099] On this basis, after the vehicle has collected multiple image frames and determined the image position codes corresponding to the multiple image frames, the multiple image frames can be input into the convolutional neural network CNN in parallel, and the image features of the image frames can be extracted using the convolutional neural network CNN. Furthermore, the image features output by the convolutional neural network CNN can be fused with the image position codes corresponding to the image features, and the fused image features and image position codes can be input into the temporal attention transformer encoder, which calculates the associated feature information between the multiple image frames, such as the pixel feature comparison results of multiple consecutive image frames.

[0100] Among them, in this embodiment, when the image features and image position codes of the image frame are subjected to feature fusion, since the vector dimension of the image features extracted from the image frame by CNN and the dimension of the image position code are consistent, the feature fusion of the image features and the image position code can be achieved by directly adding the image features and the image position code.

[0101] Furthermore, the vehicle can input the correlation feature information between the multi-frame rate images output by the temporal attention transformer encoder into the multi-layer perceptron MLP in the prediction model, and use the multi-layer perceptron MLP to perform a nonlinear transformation on the input correlation feature information to obtain the dirt detection result of the camera device. Among them, the detection result in this embodiment can be the probability that the camera is contaminated and the probability that the camera is not contaminated, and finally the detection result with a higher probability is used as the final dirt detection result.

[0102] Furthermore, in the above S2023, “performing association analysis based on the image position code and the image feature to obtain association feature information” may include:

[0103] Step a, fusing the image feature and the image position code to obtain an image fusion feature;

[0104] Step b: performing feature association analysis based on the image fusion features to obtain association feature information between multiple image features.

[0105] In this embodiment, after acquiring the image features and image position codes of each image frame, the image features and image position codes may be fused so that the fused features may be image features with a time sequence, which facilitates correlation analysis of multiple image frames. For example, multiple images arranged in sequence may be analyzed. If it is glare, the glare generally will not exist in multiple image frames at the same time, or will not remain in the same position in the image frame continuously.

[0106] Furthermore, the vehicle can perform feature association analysis based on the above-mentioned image fusion features to analyze the associated feature information between multiple image frames. For example, according to the above-mentioned embodiment, the above-mentioned associated feature information may include changes in image features in multiple consecutive image frames, such as pixel feature comparison results between multiple consecutive image frames.

[0107] Furthermore, the above “fusing the image feature and the image position code to obtain an image fusion feature” may include:

[0108] Performing dimension normalization processing on the image features and the image position codes;

[0109] The dimensionally normalized image features and image position codes are concatenated to obtain image fusion features.

[0110] In this embodiment, before the image features and the image position codes are fused, the image features and the image position codes may be dimensionally normalized first, so that the dimensions of the image features and the image position codes after the dimension normalization remain consistent.

[0111] Furthermore, the image features and image position codes of the same dimension are concatenated (that is, the two can be added together) to obtain image fusion features, so that each image feature in the image fusion feature has a corresponding image position code, so that multiple image frames with a time sequence relationship can be processed subsequently.

[0112] It is worth noting that relying solely on a single-frame image, it is impossible to accurately distinguish whether it is snowing in the environment or snowflakes on the lens, or whether it is raining in the environment or water stains on the lens; or whether it is glare from an external light source on the lens or fogging up the lens. According to the definition of a dirty lens, the former is a clean lens, and only the latter lens needs to be cleaned. A single-frame image cannot make the distinction, and this embodiment can combine continuous multi-frame image analysis to accurately determine whether the surface of the camera lens is attached to dirt or is due to environmental interference.

[0113] Furthermore, in the above S203, “obtaining an occlusion detection result of the camera device based on the associated feature information” may include:

[0114] S2031, performing weight and / or bias calculation according to the associated feature information to obtain a first probability;

[0115] S2032: When the first probability is greater than a preset probability threshold, determine that the camera device is blocked.

[0116] In this embodiment, after performing feature association analysis based on the image fusion features to obtain associated feature information between multiple image features, the vehicle can perform weight and / or bias calculation based on the associated feature information to obtain a first probability.

[0117] It should be noted that, in this embodiment, the first probability mentioned above can be used to characterize the occlusion result of the camera device.

[0118] Furthermore, if the first probability is greater than a preset probability threshold, it can be determined that the camera device is blocked.

[0119] In another embodiment, the associated feature information can be input into a multi-layer perceptron in the prediction model to obtain the dirt detection result of the camera device. For example, the associated feature information can be input into a multi-layer perceptron in the prediction model to obtain the dirt detection result of the camera device. Figure 5 The multilayer perceptron MLP in the prediction model shown obtains a first probability and a second probability output by the multilayer perceptron MLP, wherein the first probability represents the probability that the camera device is dirty and the second probability represents the probability that the camera device is not dirty, and the dirt detection result of the camera device is determined based on the first probability and the second probability.

[0120] Specifically, for example, in this embodiment, according to the above description, the multi-layer perceptron MLP in the prediction model can perform nonlinear transformation on the input associated feature information and output a first probability and a second probability. It can be understood that the output of the multi-layer perceptron MLP is a binary probability of whether the camera is dirty, wherein the first probability can represent the probability that the camera surface is not contaminated and the surface is not dirty, and the second probability can represent the probability that the camera device is contaminated and the camera surface is dirty. Furthermore, the vehicle can compare the first probability with the second probability and use the detection result corresponding to the probability with a larger value as the final dirt detection result. For example, if the probability that the camera surface is not contaminated output by the multi-layer perceptron is 0.6 and the probability that the camera surface is contaminated is 0.4, then the label with the largest probability value is selected as the final conclusion of whether there is dirt, that is, the detection result that the camera is not dirty is output.

[0121] In this way, in this embodiment, it is possible to directly perform weight and / or bias calculations based on the associated feature information to determine whether the camera device is blocked, and to use a prediction model including a convolutional neural network CNN, a temporal attention transformer encoder, and a multi-layer perceptron MLP to calculate the attention mechanism, perform parallel calculations on multiple fused input image features and temporal features, and obtain the multi-dimensional relationships between image frames, thereby eliminating environmental interference with camera dirt detection, so that snow, rain, lens glare, etc. will not be identified as dirt attached to the camera surface, thereby achieving accurate detection of camera dirt, thereby ensuring driving safety, which is of great significance especially for intelligent driving.

[0122] In one embodiment, the camera device occlusion detection method in this embodiment may further include:

[0123] S50, controlling the camera device to collect video data of the environment, and counting the video recording time;

[0124] S60: If the video recording time reaches a preset time threshold, the video recording is terminated to obtain the video information.

[0125] It should be noted that in this embodiment, according to the above description, the technician can make dirt attached to the camera, for example, by using fingers, toothbrushes, etc. to attach different types and proportions of dirt such as mud, water droplets, and weeds to the lens surface.

[0126] On this basis, during the driving process of the vehicle, the contaminated camera device can be controlled to collect video data of the vehicle's environment, and detect whether the video recording time reaches a first preset time threshold (to ensure that a sufficient number of frame images are collected). When it is detected that the recording time reaches the first preset time threshold, the recording can be ended to obtain the first video data.

[0127] In addition, under the same driving conditions, the vehicle can control the uncontaminated camera device to collect video data of the environment in which the vehicle is located. Similarly, it detects whether the video recording time reaches the second preset time threshold. When it is detected that the recording time reaches the second preset time threshold, the recording can be ended to obtain the second video data. This embodiment does not specifically limit the first preset time threshold and the second preset time threshold, and can be flexibly adjusted according to actual needs.

[0128] Furthermore, a video segmentation rule may be preset to segment the first video data and the second video data into multiple segments, wherein the preset video segmentation rule includes ensuring that each segment of the video data is consistent with the video length that can be processed by the initial prediction model video.

[0129] In a specific embodiment, Figure 6As shown, the video data set collection process may include:

[0130] (1) First, dirt is attached to the target vehicle-mounted camera. For example, dirt of different types and proportions such as mud, water droplets, and weeds can be attached to the lens surface by using fingers or a toothbrush, and then step (2) is performed.

[0131] (2) After the lens dirt is created, the vehicle starts driving on the road and starts recording the video data of the target camera at the same time, and executes step (3). Liquid and solid-liquid mixture dirt, because of its adhesion, will leave obvious dirt marks on the camera surface even if it falls, and can be regarded as valid data. For dirt that does not leave any traces, the data can be filtered through manual review after collection;

[0132] (3) Determine whether to end the recording of the dirty data. If the duration of the video recording is greater than the time threshold T, it is yes and step (4) is executed. If the duration of the video recording is less than the time threshold T, it is no and step (2) is executed.

[0133] (4) End video recording and divide the video into multiple short videos according to a fixed duration T for storage. The duration T of the division needs to be consistent with the duration of the video that can be processed by the video classification neural network below. If the total duration of the original video cannot be divided by T, the last short video is directly deleted. Execute step (5);

[0134] (5) The vehicle stops, pulls over, and executes step (6);

[0135] (6) Determine whether to end all dirty video collection tasks. If the duration of all types of dirty videos meets the requirement, then yes, and then execute step (7). If the duration of a certain type of dirty video does not meet the requirement, then no, and then execute step (1).

[0136] (7) End the process;

[0137] Through the above process, a large number of dirty videos can be collected quickly and efficiently without labeling, and integrated with the video data collected by uncontaminated cameras to obtain a data set for dirty video classification neural network training and learning.

[0138] Therefore, in this embodiment, video data for training the initial prediction model can be collected in advance, and the temporal attention transformer encoder in the initial prediction model can be trained using the video data. The initial prediction model is trained by introducing the temporal information between image frames so that the obtained prediction model can pay attention to the temporal relationship between multiple frames of images. The trained prediction model can accurately identify whether the camera surface is contaminated, eliminates the interference of the surrounding environment on the camera detection, and reduces the false detection rate of camera dirt.

[0139] Furthermore, the camera device occlusion detection method in this embodiment may further include:

[0140] S70, controlling the camera device to collect new video information at a preset time interval;

[0141] S80, determining the occlusion detection result of the camera device according to the time sequence characteristics of the image frames in the new video information.

[0142] In this embodiment, the vehicle can control the camera device to collect new video information at preset time intervals during driving, and determine the occlusion detection result of the camera device based on the timing characteristics of the image frames in the new video information. For details, please refer to the above embodiment and will not be repeated here.

[0143] Through the above operations, this embodiment can continuously perform camera device dirtiness detection, thereby avoiding misidentification of road conditions caused by camera device contamination and reducing the safety of intelligent assisted driving.

[0144] In another embodiment, according to the above description, the prediction model may be directly used to determine the occlusion detection result of the camera device.

[0145] On this basis, the present embodiment can pre-construct an initial prediction model commonly used in the art, and use the pre-collected video data set to train the initial prediction model to obtain the prediction model in the present embodiment.

[0146] Specifically, for example, the vehicle can control the camera to collect video data sets multiple times during driving, wherein the video data sets include first video data collected by a contaminated camera device and second video data collected by an uncontaminated camera device. It is understandable that the technician can arrange dirt on the camera in advance so that the camera can collect video data with dirt.

[0147] Furthermore, a convolutional neural network can be used to extract multiple first image frames in the first video data and multiple second image frames in the second video data, and obtain the first image features of the first image frame, the first image position code of the first image frame, and the second image features of the second image frame, and the second image position code of the second image frame.

[0148] In this way, the vehicle can input the above-mentioned first image feature, the first image position code associated with the first image feature, the second image feature, and the second image position code associated with the second image feature into the temporal attention transformer encoder in the above-mentioned initial prediction model to train the temporal attention transformer encoder, optimize the encoder weight parameters, and obtain the prediction model, wherein this embodiment can pre-construct a cross-entropy loss function and iteratively optimize the encoder until the error between the predicted detection result output by the prediction model and the actual detection result is minimized.

[0149] In another embodiment, after extracting multiple first image frames from the first video data and multiple second image frames from the second video data, the multiple first image frames and the image position codes corresponding to the images, the second image frames, and the image position codes corresponding to the second image frames can be directly input into the initial prediction model, and the entire initial prediction model can be trained to obtain a prediction model.

[0150] In one embodiment, the camera device occlusion detection method in this embodiment may further include:

[0151] S90, when it is determined that the camera device is blocked, extracting pixel feature parameters corresponding to at least two image frames;

[0152] S100, determining the pollution type and pollution level information corresponding to the camera device according to the pixel characteristic parameters;

[0153] S110, controlling the camera device to perform self-cleaning according to the pollution type and the pollution level information, and controlling the self-cleaning camera device to collect new video information.

[0154] In this embodiment, after the vehicle detects the detection result of the camera, if the detection result is that the camera is contaminated and there is dirt on the surface, the corresponding pixel feature parameters in the multiple image frames can be extracted.

[0155] For example, texture features: dirt usually introduces additional textures and impurities, making the texture of the image complex. By analyzing the texture features of the image, such as texture details and texture contrast, the dirty area can be detected; color features: dirt usually changes the color distribution of the image. By analyzing the color features of the image, such as the color mean, variance or color saturation, the presence of dirt can be detected; edge features: dirt usually causes the image edge to become blurred or discontinuous. By analyzing the edge features of the image, such as the strength, number and continuity of the edge, the dirty area can be detected.

[0156] Furthermore, based on the above pixel characteristic parameters, the pollution type and pollution level information corresponding to the camera are identified. For example, if mud is identified to exist on the camera surface, the pollution level of the mud is determined to be low according to the preset pollution level mapping table.

[0157] At this time, the vehicle can control the camera device to perform self-cleaning, and re-capture new image frames after self-cleaning for camera detection to detect whether the camera has been cleaned.

[0158] In general, in a specific embodiment, when the vehicle is driving and the dirt detection function based on short video classification is turned on, when the video length recorded by the camera reaches T, the short video of this length T is split into image frames, the number of image frames is N, and the N frames of images are parallelly extracted using the convolutional neural network CNN to obtain the image features of each frame, and the image features of each frame are combined with the image position code and then sent to the temporal attention transformer encoder for calculation. Finally, the output result of the encoder is sent to the multi-layer perceptron MLP for a nonlinear transformation, and the probability values ​​of the two categories of dirt and no dirt are output. By selecting the label with a larger probability, the final output result of the model can be obtained, wherein the final output of the neural network is a binary classification probability of whether there is dirt. For example, the network finally outputs a probability of dirt of 0.4 and a probability of no dirt of 0.6. The label with the largest probability value is selected as the final conclusion of whether there is dirt, that is, the conclusion of no dirt on the camera is output.

[0159] In this way, the present application collects and constructs a video of camera dirtiness to train the prediction model, so that the prediction model can effectively utilize the timing information between image frames to prevent the occurrence of false detection. The prior art is limited to the use of single-frame image information and ignores the timing information. The present application reduces the false detection rate of camera dirtiness by introducing the timing information between image frames.

[0160] Accordingly, the embodiment of the present application also provides a vehicle, such as Figure 7 As shown, Figure 7 A schematic diagram of the structure of a vehicle provided in an embodiment of the present application. The vehicle 1100 includes a processor 1101 having one or more processing cores, a memory 1102 having one or more computer-readable storage media, and a computer program stored in the memory 1102 and executable on the processor. The processor 1101 is electrically connected to the memory 1102. It will be understood by those skilled in the art that the vehicle structure shown in the figure does not constitute a limitation on the vehicle, and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0161] The processor 1101 is the control center of the vehicle 1100, and uses various interfaces and lines to connect various parts of the entire vehicle 1100. By running or loading software programs and / or units stored in the memory 1102, and calling data stored in the memory 1102, it executes various functions of the vehicle 1100 and processes data, thereby monitoring the vehicle 1100 as a whole. The processor 1101 can be a processor (Central Processing Unit, CPU), a graphics processing unit (graphics processing unit, GPU), a network processor (Network Processor, NP), etc., and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application.

[0162] In the embodiment of the present application, the processor 1101 in the vehicle 1100 will load instructions corresponding to the processes of one or more application programs into the memory 1102 according to the following steps, and the processor 1101 will run the application programs stored in the memory 1102 to implement various functions, such as:

[0163] Acquire video information including at least two image frames collected by a shooting device;

[0164] The occlusion detection result of the camera device is determined according to the time sequence characteristics of the image frame in the video information.

[0165] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0166] Optional, such as Figure 7 As shown, the vehicle 1100 further includes: a touch screen 1103, a radio frequency circuit 1104, an audio circuit 1105, an input unit 1106, and a power supply 1107. The processor 1101 is electrically connected to the touch screen 1103, the radio frequency circuit 1104, the audio circuit 1105, the input unit 1106, and the power supply 1107, respectively. Those skilled in the art will appreciate that Figure 7 The vehicle structure shown in the figure does not constitute a limitation on the vehicle, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0167] The touch display screen 1103 can be used to display a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface. The touch display screen 1103 may include a display panel and a touch panel. Among them, the display panel can be used to display information input by the user or information provided to the user and various graphical user interfaces of the vehicle, which can be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel can be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light-emitting diode (OLED, Organic Light-Emitting Diode), etc. The touch panel can be used to collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 1101, and can receive the command sent by the processor 1101 and execute it. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 1101 to determine the type of touch event, and then the processor 1101 provides a corresponding visual output on the display panel according to the type of touch event. In an embodiment of the present application, the touch panel and the display panel can be integrated into the touch display screen 1103 to realize the input and output functions. However, in some embodiments, the touch panel and the touch panel can be used as two independent components to realize the input and output functions. That is, the touch display screen 1103 can also be used as a part of the input unit 1106 to realize the input function.

[0168] The RF circuit 1104 may be used to send and receive RF signals to establish wireless communication with network devices or other vehicles through wireless communication, and to send and receive signals with network devices or other vehicles.

[0169] The audio circuit 1105 can be used to provide an audio interface between the user and the vehicle through a speaker and a microphone. The audio circuit 1105 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 1105 and converted into audio data, and then the audio data is output to the processor 1101 for processing, and then sent to another vehicle through the radio frequency circuit 1104, or the audio data is output to the memory 1102 for further processing. The audio circuit 1105 may also include an earplug jack to provide communication between an external headset and the vehicle.

[0170] The input unit 1106 may be configured to receive input numbers, character information, or user feature information (such as fingerprint, iris, facial information, etc.), and generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0171] The power supply 1107 is used to supply power to various components of the vehicle 1100. Optionally, the power supply 1107 can be logically connected to the processor 1101 through a power management device, so that the power management device can manage charging, discharging, and power consumption. The power supply 1107 can also include one or more DC or AC power supplies, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0172] although Figure 7 Not shown, the vehicle 1100 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.

[0173] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0174] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0175] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of computer programs are stored. The computer program can be loaded by a processor to execute any camera device occlusion detection method provided in the embodiment of the present application. The computer program can execute the following steps of the camera device occlusion detection method:

[0176] Acquire video information including at least two image frames collected by a shooting device;

[0177] The occlusion detection result of the camera device is determined according to the time sequence characteristics of the image frame in the video information.

[0178] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0179] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0180] Since the computer program stored in the computer-readable storage medium can execute any camera device occlusion detection method provided in the embodiments of the present application, the beneficial effects that can be achieved by any camera device occlusion detection method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0181] In the above-mentioned detection device, computer-readable storage medium, computer program product, and vehicle, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the above-described detection device, computer-readable storage medium, computer program product, vehicle and its corresponding units can refer to the description of the camera device occlusion detection method in the above embodiment, and will not be repeated here.

[0182] The above are only preferred embodiments of the present application and do not constitute any form of limitation to the present application. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. A camera device occlusion detection method, characterized in that: The method comprises: Acquire video information including at least two image frames collected by a shooting device; Determining an occlusion detection result of the camera device according to a time sequence feature of the image frame in the video information; Before the step of determining the time sequence characteristics of the image frame in the video information, the method further comprises: Acquire sequence information of the image frames in the video information; Sequentially encoding the image frames according to the sequence information of the image frames to obtain time sequence features of the image frames; The determining, according to the time sequence characteristics of the image frames in the video information, the occlusion detection result of the camera device comprises: Inputting multiple image frames in parallel into a convolutional neural network (CNN) in a prediction model, and extracting image features of the image frames through the convolutional neural network (CNN); Splitting the image frame to obtain sub-images of the image frame; According to the time sequence characteristics of the image frame, a sub-image in the image frame is encoded to obtain an image position code; Performing feature fusion on the image feature and the image position code corresponding to the image feature to obtain an image fusion feature, and inputting the image fusion feature and the image position code into a temporal attention transformer encoder in the prediction model, and calculating the associated feature information between multiple image frames through the temporal attention transformer encoder; the associated feature information includes the change of camera dirtiness in multiple consecutive image frames; Based on the associated feature information, it is identified whether the camera is dirty, and an occlusion detection result of the camera device is obtained.

2. The camera device occlusion detection method according to claim 1, characterized in that: The image feature includes at least one of a content feature, a shape feature, a structure feature, a pixel feature, a texture feature, a brightness feature, and a chromaticity feature.

3. The camera device occlusion detection method according to claim 1, characterized in that: The step of fusing the image feature with the image position code to obtain an image fusion feature includes: Performing dimension normalization processing on the image features and the image position codes; The dimensionally normalized image features and image position codes are concatenated to obtain image fusion features.

4. The camera device occlusion detection method according to claim 1, characterized in that: The step of identifying whether the camera is dirty based on the associated feature information and obtaining an occlusion detection result of the camera device includes: Performing weight and / or bias calculation according to the associated feature information to obtain a first probability; When the first probability is greater than a preset probability threshold, it is determined that the camera device is blocked.

5. The camera device occlusion detection method according to claim 1, characterized in that: The obtaining of video information including at least two image frames collected by a shooting device includes: Controlling the camera device to collect video data of the environment and counting the video recording time; If the video recording time reaches a preset time threshold, the video recording is terminated and the video information is obtained.

6. The camera device occlusion detection method according to claim 5, characterized in that: The method further comprises: Controlling the camera device to collect new video information at a preset time interval; The occlusion detection result of the camera device is determined according to the time sequence characteristics of the image frames in the new video information.

7. The camera device occlusion detection method according to any one of claims 1 to 6, characterized in that: The method further comprises: When it is determined that the camera device is blocked, extracting pixel feature parameters corresponding to at least two of the image frames; Determining the pollution type and pollution level information corresponding to the camera device according to the pixel characteristic parameters; According to the pollution type and the pollution level information, the camera device is controlled to perform self-cleaning, and the self-cleaning camera device is controlled to collect new video information.

8. A detection device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: It comprises a computer program, which, when running on a vehicle, is used to cause the vehicle to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that It includes a computer program, which is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device executes any one of the methods described in claims 1 to 7.

11. A vehicle, characterized in that: Includes the detection device according to claim 8, or the computer program product according to claim 10.

Images