Camera dirt detection method, apparatus, and vehicle

Abstract

A camera dirt detection method, device and vehicle, the method comprising: inputting an image collected by a camera to be detected into a trained encoder to obtain an image code output by the encoder; extracting a latent variable feature of the image code based on a trained latent variable feature layer; obtaining a dirt evaluation index according to the latent variable feature; and determining that the camera has dirt when the dirt evaluation index meets a preset condition. The method, device and vehicle detect camera dirt based on the trained encoder and the trained latent variable feature layer, thereby improving the accuracy of camera dirt detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and more particularly, to a camera dirty detection method and device and a vehicle. BACKGROUND

[0002] A camera is an important component of an automatic driving sensor suite. The camera is directly exposed to the external environment and is prone to dirt. Compared with other sensors, the performance of the camera decreases much more due to dirt. Therefore, it is crucial to accurately detect dirt on the camera, especially for higher levels of automatic driving.

[0003] In the camera dirty detection based on the deep learning method, a supervised deep learning method is mainly used. This method needs to collect image data of the dirty camera in advance for model training. Due to the difficulty in image collection, the workload required for labeling is large, and the actual application effect is not ideal, thereby resulting in a low accuracy of camera dirty detection. SUMMARY

[0004] A series of simplified concepts are introduced in the summary section, which will be further described in detail in the specific embodiment section. The summary section of the present application does not mean to attempt to limit the key features and necessary technical features of the claimed technical solution, and even less to determine the protection scope of the claimed technical solution.

[0005] In view of the deficiencies of the prior art, the first aspect of the embodiment of the present application proposes a camera dirty detection method, which comprises: inputting an image collected by a to-be-detected camera into a trained encoder to obtain an image code output by the encoder; extracting a latent variable feature of the image code based on a trained latent variable feature layer; obtaining a dirty evaluation index according to the latent variable feature, and determining that the camera has dirt when the dirty evaluation index meets a preset condition.

[0006] In one embodiment, the obtaining of the dirty evaluation index according to the latent variable feature comprises: determining a mean vector and a variance vector of the latent variable feature; obtaining a KL divergence according to the mean vector and the variance vector, and the dirty evaluation index comprises the KL divergence.

[0007] In an embodiment, the determining that the to-be-detected camera is dirty when the dirtiness evaluation index meets a preset condition comprises: if the KL divergence of a single frame of image collected by the to-be-detected camera is greater than a first preset threshold, it is determined that the to-be-detected camera is dirty; and / or if the KL divergences of at least two frames of image collected by the to-be-detected camera are all less than or equal to a first preset threshold and greater than a second preset threshold, it is determined that the to-be-detected camera is dirty, wherein the second preset threshold is less than the first preset threshold.

[0008] In an embodiment, the training process of the encoder and the latent variable feature layer comprises: inputting an original image collected by a camera into the encoder to obtain an original image code output by the encoder; extracting a latent variable feature of the original image code based on the latent variable feature layer; obtaining a sampled latent variable according to the latent variable feature of the original image code; inputting the sampled latent variable into a decoder corresponding to the encoder to obtain a reconstructed image output by the decoder; updating parameters of the encoder and the latent variable feature layer based on a loss function to obtain the trained encoder and the trained latent variable feature layer, wherein the loss function at least comprises a reconstruction loss determined according to the original image and the reconstructed image.

[0009] In an embodiment, the obtaining a sampled latent variable according to the latent variable feature of the original image code comprises: determining a mean vector and a variance vector of the latent variable feature of the original image code, wherein the mean vector and the variance vector are subject to a normal distribution; and obtaining the sampled latent variable according to the mean vector, the variance vector and a normal distribution, wherein a dimension of the normal distribution sampling is the same as a dimension of the mean vector, and the normal distribution sampling is a random number satisfying the normal distribution.

[0010] In an embodiment, the obtaining a sampled latent variable according to the mean vector, the variance vector and a normal distribution comprises: determining a product of the normal distribution sampling and the variance vector, and obtaining the sampled latent variable according to a sum of the product and the mean vector.

[0011] In an embodiment, the loss function further comprises a KL divergence loss determined based on the latent variable feature of the original image code.

[0012] In an embodiment, the original image collected by the camera is an image collected by a camera without dirt.

[0013] According to another aspect of the present application, a camera dirt detection device is provided, which comprises a memory and a processor, the memory having stored thereon a computer program to be run by the processor, the computer program, when executed by the processor, performing the camera dirt detection method.

[0014] According to still another aspect of the present application, a vehicle is provided, which comprises the camera dirt detection device.

[0015] According to another aspect of the present application, a computer storage medium is provided, which has stored thereon a computer program, the computer program, when executed, implementing the camera dirt detection method.

[0016] According to another aspect of the present application, a computer program is provided, the computer program, when executed, implementing the camera dirt detection method.

[0017] The camera dirt detection method, device and vehicle of the present application detect camera dirt based on a trained encoder and a trained latent variable feature layer, thereby improving the accuracy of camera dirt detection. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout the figures. The drawings provided are for illustrative purposes only and, as such, should not be construed to limit the present application. In the drawings:

[0019] Figure 1 a schematic flow chart of a camera dirt detection method according to an embodiment of the present application;

[0020] Figure 2 a schematic flow chart of a training method of an encoder and a latent variable feature extraction layer for camera dirt detection according to an embodiment of the present application;

[0021] Figure 3 a comparison chart of images captured by a camera without dirt and images captured by a dirty camera according to an embodiment of the present application;

[0022] Figure 4 a schematic diagram of an encoder according to an embodiment of the present application;

[0023] Figure 5 a schematic diagram of a latent variable feature layer according to an embodiment of the present application;

[0024] Figure 6 a schematic diagram of a decoder according to an embodiment of the present application;

[0025] Figure 7 A schematic block diagram of a camera dirt detection apparatus according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the objects, technical solutions and advantages of the present application clearer, the following will describe the example embodiments according to the present application in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. It should be understood that the present application is not limited to the described example embodiments. Based on the embodiments of the present application described in the present application, any other embodiments obtained by a person of ordinary skill in the art without creative effort should fall within the protection scope of the present application.

[0027] In the following description, a large number of specific details are given in order to provide a more thorough understanding of the present application. However, it should be apparent to a person of ordinary skill in the art that the present application can be implemented without one or more of these details. In other instances, some well-known features have not been described in order to avoid obscuring the present application.

[0028] It should be understood that the present application can be implemented in different forms and should not be interpreted as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that the disclosure will be thorough and complete and will fully convey the scope of the present application to those skilled in the art.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising", when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0030] Most early camera dirty detection methods use traditional image analysis methods, that is, by measuring the gray value of the image collected by the camera to be detected, and comparing it with the gray value of the reference image under the same illumination, if the gray value of the image to be detected does not meet the range of the gray value of the reference image, it is determined that there is dirt. In the case of uniform image background, this algorithm can obtain good detection effect, but due to the influence of equipment and environment and other factors, vignetting phenomenon will inevitably exist during image acquisition, which will cause the center area of the image to be brighter and the edge area to be darker, and the gray scale range of the image will be expanded. For some not very serious or located in the vignetting area, it is difficult to detect all of them. Another method of dirty detection is based on fast difference filtering method, by traversing the image to be detected, and subtracting the average gray value of the reference pixel at a specific position around the current pixel from the gray value of the current pixel, the difference image is obtained. The more serious the dirt, the larger the difference value. Among them, the selection of reference pixels still needs to rely on subjective definition, and the interval between reference pixels and current pixels should be far enough to ensure that they are located outside the dirty area; but if the distance is too far, the gray scale attenuation is serious, and the accuracy cannot be guaranteed. In addition, due to the different colors, sizes and shapes of camera dirt, even the same dirt under different conditions, the appearance is different, so it is difficult to have almost consistent dirty data, which cannot guarantee the accuracy.

[0031] In addition, some methods use supervised deep learning method based on segmentation to realize camera dirty detection, which needs to collect image data of dirty camera in advance for model training. The imaging effect is different when the camera dirty degree is different, but the data of camera dirty is often difficult to collect, and there is deviation between the simulated data and the real data. Labeling the image data also needs a large work flow, which further leads to insufficient generalization ability of the model, and the actual application effect is not ideal, so as to cause low detection rate of camera dirty.

[0032] In order to thoroughly understand the present application, detailed structures will be proposed in the following description in order to explain the technical solutions proposed by the present application. The optional embodiments of the present application are described in detail as follows, however, in addition to these detailed descriptions, the present application can also have other implementation manners.

[0033] Below, the camera dirty detection method 100 proposed by the embodiments of the present application will be described with reference to the accompanying drawings. Figure 1 The camera dirty detection method 100 proposed by the embodiments of the present application can include the following steps. Figure 1 The schematic flow chart of the camera dirty detection method 100 according to the embodiments of the present application is shown. As shown in Figure 1 The camera dirty detection method 100 according to the embodiments of the present application can include the following steps.

[0034] In step S110, the image collected by the to-be-detected camera is input into the trained encoder to obtain image encoding output by the encoder.

[0035] In step S120, the latent variable feature of the image encoding is extracted based on the trained latent variable feature layer.

[0036] In step S130, the dirtiness evaluation index is obtained according to the latent variable feature, and when the dirtiness evaluation index meets a preset condition, it is determined that the to-be-detected camera has dirtiness.

[0037] In the embodiment of the present application, first, the image collected by the to-be-detected camera is input into the trained encoder, and the output image encoding can be obtained. Then, the latent variable feature of the image encoding is extracted according to the trained latent variable feature extraction layer. After the training of the encoder and the latent variable feature extraction layer, when the image collected by the camera without dirtiness (i.e., the normal camera) is input into the encoder and the latent variable feature layer, the latent variable feature meeting the normal distribution can be obtained, and when the image collected by the dirty camera is input into the encoder and the latent variable feature layer, the latent variable feature meeting the normal distribution cannot be obtained. Therefore, finally, the latent variable feature of the image collected by the to-be-detected camera can be extracted based on the trained encoder and the trained latent variable feature extraction layer, the dirtiness evaluation index of the to-be-detected camera is obtained according to the latent variable feature, and when it is determined that the dirtiness evaluation index meets a certain preset condition, it can be determined that the to-be-detected camera has dirtiness.

[0038] Therefore, the camera dirtiness detection method 100 according to the embodiment of the present application detects the camera dirtiness based on the trained encoder and the trained latent variable feature layer, and improves the accuracy of camera dirtiness detection.

[0039] In the embodiment of the present application, in step S110, the image collected by the to-be-detected camera is input into the trained encoder to obtain the image encoding output by the encoder, and in step S120, the latent variable feature of the image encoding is extracted based on the trained latent variable feature layer. Wherein, the encoder and the latent variable feature extraction layer need to be trained to obtain the trained encoder and the trained latent variable feature extraction layer.

[0040] In the embodiment of the present application, the training method 200 of the encoder and the latent variable feature layer for camera dirtiness detection will be described below with reference to Figure 2 Figure 2 ​As shown in step S210, the raw image captured by the camera is input into the encoder to obtain the raw image code output by the encoder. The camera used to capture the raw image includes but is not limited to a vehicle-mounted camera. The embodiment of the present application does not need to collect image data of the camera in various dirty scenes in the training stage, but only needs to obtain image data collected by the camera without dirt, thereby reducing the difficulty of image acquisition; and the trained model can be tested by using the data collected by the dirty camera. Figure 3 A comparison chart of the image collected by the camera without dirt and the image collected by the dirty camera is shown. The camera without dirt is a normal camera.

[0041] After obtaining the raw image, it is input into the encoder for encoding. The encoder is a kind of neural network, and its function is to learn the representation of the input information by taking the input information as the learning target. The encoder is a neural network similar to CNN (convolutional neural network), which receives input and produces a much smaller representation (image code) that contains enough information to provide the required output format for the next part of the network.

[0042] Referring to Figure 4 An exemplary network structure of the encoder is shown, which is composed of multiple convolutional layers, maximum pooling layers, average pooling layers, etc. Figure 4 The network structure shown is simple in operator, easy to deploy, strong in migration, and applicable to different development versions to solve the problem of unsupported operators. However, it should be noted that the encoder of the embodiment of the present application can also use any other suitable network structure.

[0043] In step S220, the latent variable feature of the raw image code is extracted based on the latent variable feature layer. The latent variable feature is a low-dimensional representation of the raw image in the latent space, which includes multiple elements, and the multiple elements are used to indicate the features of the raw image in multiple image indicators. Each element is used to describe a certain attribute of the raw image. Optionally, the multiple image indicators correspond to multiple directions in the latent space, and the elements of each image indicator are numerical values of the raw image in one direction in the latent space.

[0044] Referring to Figure 5 The network structure of the latent variable feature layer of one embodiment of the present application is shown. Figure 5 The latent variable feature layer includes convolutional layers, fully connected layers (FC), batch normalization layers (BN), and linear rectifier functions (ReLU), etc. It should be noted that the latent variable feature layer of the embodiment of the present application can also use any other suitable network structure.

[0045] At step S230, the sampling latent variable is obtained according to the latent variable feature of the original image coding. The sampling latent variable has a similar distribution with the original image, and the sampling latent variable is decoded by the decoder to generate a new image, i.e., the reconstructed image, which has a similar distribution with the original image, thereby improving the accuracy of the reconstructed image.

[0046] Exemplarily, the sampling latent variable is obtained according to the latent variable feature of the original image coding, including: determining a mean vector and a variance vector of the latent variable feature of the original image coding; and obtaining the sampling latent variable according to the mean vector, the variance vector and normal distribution sampling. The normal distribution sampling has the same dimension as the mean vector, i.e., the same dimension as the variance vector.

[0047] The mean vector and the variance vector of the latent variable feature of the original image coding are subject to normal distribution, and can represent normal distribution information of the original image. The plurality of mean features and the plurality of variance features correspond to the normal distribution of the original image on a plurality of image indicators. The normal distribution sampling is a random number satisfying the normal distribution. The introduction of the normal distribution sampling can embed noise in the latent variable feature, so that the adaptability of the model is better.

[0048] Exemplarily, the sampling latent variable is obtained according to the mean vector, the variance vector and the normal distribution sampling, specifically including: determining the product of the normal distribution sampling ε and the variance vector σ, and obtaining the sampling latent variable z according to the sum of the product and the mean vector μ, i.e.:

[0049] z = μ + ε × σ (Formula 1)

[0050] Then, at step S240, the sampling latent variable is input to the decoder corresponding to the encoder for decoding to obtain the reconstructed image. The purpose of the encoding and decoding in the embodiment of the application is to learn as much as possible the original image that is not obtained, without enumerating the dirty scene image, thereby reducing the difficulty of image acquisition.

[0051] The decoder performs up-sampling processing on the sampling latent variable to obtain the reconstructed image with the same size as the original image. The reconstructed image is reconstructed based on the sampling latent variable, and the sampling latent variable has a similar distribution with the original image, so the reconstructed image has a similar distribution with the original image. The decoder corresponds to each layer of the encoder, and an exemplary network structure thereof is shown in Figure 6 .

[0052] At step S250, parameters of the encoder and the latent variable feature layer are updated according to a loss function, to obtain the trained encoder and the trained latent variable feature layer. The loss function at least includes a reconstruction loss determined according to the original image and the reconstructed image, i.e., the reconstruction loss indicates a gap between the original image and the reconstructed image. By optimizing the parameters of the encoder and the latent variable feature layer with the objective of minimizing the reconstruction loss, the original image and the reconstructed image can be made to approach each other.

[0053] Specifically, the reconstruction loss can be calculated according to an MSE (Mean Square Error) value between the original image and the reconstructed image, i.e.,

[0054] loss(x,y)=1 / nΣ(x i -y i ) 2 (Formula 2)

[0055] wherein x i is a pixel value in the original image, and y i is a pixel value in the reconstructed image.

[0056] Further, the loss function further includes a KL divergence loss determined based on the latent variable feature encoded from the original image. That is, the total loss function can be a sum of the reconstruction loss and the KL divergence loss. The KL divergence loss measures a difference between two probability distributions, and the KL divergence loss encourages the encoder to uniformly distribute all encodings around the center of the latent space. Minimizing the KL divergence loss means ensuring that the distribution of the latent variable feature encoded from the original image is as close to a normal distribution as possible.

[0057] Specifically, a KL divergence between the distribution of the latent variable feature and a normal distribution constitutes the KL divergence loss. That is, the KL divergence loss is expressed as:

[0058]

[0059] wherein μ (i) and σ (i) are components of a mean vector and a variance vector respectively, and d is a dimension of the mean vector and the variance vector.

[0060] It can be understood that during the training process, a plurality of original images are obtained, and the above training steps are iteratively performed based on the plurality of original images until a preset requirement is met. The preset requirement can be that the loss value of the loss function reaches a preset loss value, or the loss value reaches convergence, or the number of iterations reaches a preset number, etc., which is not specifically limited here. After the training is completed, when the original image collected by the normal camera (i.e., the camera without dirt) is input into the encoder and the latent variable feature layer, the latent variable feature satisfying the normal distribution can be obtained; if the original image collected by the dirty camera is input into the encoder and the latent variable feature layer, the latent variable feature satisfying the normal distribution cannot be obtained. Therefore, the camera dirt can be detected based on the trained encoder and the latent variable feature layer.

[0061] Exemplarily, the training method of the encoder and the latent variable feature layer for camera dirt detection according to the embodiment of the present application can be implemented in a device, apparatus or system having a memory and a processor.

[0062] Exemplarily, the training method of the encoder and the latent variable feature layer for camera dirt detection according to the embodiment of the present application can be deployed at a personal terminal, such as a smart phone, a tablet computer, a personal computer, etc. Alternatively, the training method for camera dirt detection according to the embodiment of the present application can also be deployed at a server end (or cloud end). Alternatively, the training method of the encoder and the latent variable feature layer for camera dirt detection according to the embodiment of the present application can also be distributedly deployed at the server end (or cloud end) and the personal terminal. Alternatively, the training method of the encoder and the latent variable feature layer for camera dirt detection according to the embodiment of the present application can also be distributedly deployed at different personal terminals.

[0063] In summary, since the camera dirty data is difficult to collect and the dirty scene data is difficult to reproduce, the training method of the encoder and the latent variable feature layer for camera dirt detection according to the embodiment of the present application is trained based on an unsupervised deep learning method, without the need to exhaust the dirty scene image and without the need for manual annotation of the image, thereby reducing the difficulty of data acquisition, saving the workload required for image annotation, and reducing the problem of insufficient generalization performance due to lack of annotation data.

[0064] In the embodiments of the present application, after the encoder and the latent variable feature layer are trained according to the training method described above, when an image captured by a camera without dirt (i.e., a normal camera) is input into the encoder and the latent variable feature layer, a latent variable feature satisfying a normal distribution can be obtained; if an original image captured by a dirty camera is input into the encoder and the latent variable feature layer, a latent variable feature satisfying a normal distribution cannot be obtained. Therefore, the latent variable feature of an image captured by a to-be-detected camera can be extracted based on the trained encoder and the latent variable feature layer, and whether the to-be-detected camera is dirty can be determined according to the latent variable feature.

[0065] Exemplarily, the to-be-detected camera can be a vehicle-mounted camera. When a vehicle-mounted system is started, an image captured by a vehicle-mounted camera is acquired, at this time, the vehicle is often not driving, and therefore, whether the camera is normal can be predicted in advance before the vehicle drives, reminding a user to wipe the camera, and avoiding the problem of reduced accuracy of a downstream detection task caused by dirty camera to affect the performance of an auxiliary driving system.

[0066] After the image captured by the to-be-detected camera is acquired, the image is input into the trained encoder to obtain an image code output by the encoder; then, the latent variable feature of the image code is extracted based on the trained latent variable feature layer. The network structure of the encoder and the latent variable feature can refer to the foregoing. Since the distribution of the latent variable feature is related to whether the camera is dirty, a dirty evaluation index can be acquired according to the latent variable feature, and when the dirty evaluation index satisfies a preset condition, it is determined that the to-be-detected camera is dirty.

[0067] In the embodiments of the present application, in step S130, the dirty evaluation index is acquired according to the latent variable feature, and when the dirty evaluation index satisfies a preset condition, it is determined that the camera is dirty. Wherein, the dirty evaluation index is acquired according to the latent variable feature, including: determining a mean vector μ and a variance vector σ of the latent variable feature; obtaining a KL divergence according to the mean vector μ and the variance vector σ, and the dirty evaluation index includes the KL divergence.

[0068] Exemplarily, the KL divergence is expressed as:

[0069]

[0070] Wherein, μ (i) and σ (i) are components of the mean vector and the variance vector respectively, and d is the dimension of the mean vector and the variance vector.

[0071] Then, whether the image is a dirty image is determined according to the KL divergence, that is, whether the to-be-detected camera is dirty is determined.

[0072] Specifically, if the KL divergence of a single frame image captured by the camera under test is greater than the first preset threshold T1, it is directly determined that the camera under test is dirty. The first preset threshold T1 is set to be much larger than the second preset threshold T2 to ensure the accuracy of the detection results.

[0073] If the KL divergence of at least two frames captured by the camera under test is less than or equal to a first preset threshold T1 and greater than a second preset threshold T2, then the camera under test is determined to be dirty, wherein the second preset threshold T2 is less than the first preset threshold T2. Specifically, if the KL divergence of one frame is less than or equal to the first preset threshold T1, the next frame is selected to calculate the KL divergence. After multiple repeated calculations, a voting mechanism is used to determine whether the camera is dirty. The number of repeated calculations is determined based on a trade-off between real-time calculation speed and detection accuracy. In one example, if the KL divergence of N consecutive frames captured by the camera under test is greater than the second preset threshold T2, then the camera is determined to be dirty, where N≥2. In another example, if the KL divergence of N frames out of M consecutive frames captured by the camera is greater than the second preset threshold T2, then the camera is determined to be dirty, where N≥2 and M>N. This embodiment of the invention uses multiple judgments, which can improve accuracy and reduce false positives.

[0074] In summary, the camera dirt detection method of this invention improves the accuracy of camera dirt detection by detecting camera dirt based on a trained encoder and a trained latent variable feature layer.

[0075] The following is combined with Figure 7 Describes a camera dirt detection device provided according to another aspect of this application. Figure 7 A schematic structural block diagram of a camera dirt detection device 700 according to an embodiment of this application is shown. Figure 7 As shown, the camera dirt detection device 700 includes a memory 710 and a processor 720. The memory 710 stores a computer-executable program that runs on the processor 720. When the computer-executable program is run by the processor 720, it causes the processor 720 to execute the aforementioned camera dirt detection method 100. Those skilled in the art can understand the structure and specific operation of each module in the camera dirt detection device 700 according to the embodiments of this application based on the foregoing description; for simplicity, further details are omitted here. Therefore, the camera dirt detection device according to the embodiments of this application detects camera dirt based on a trained encoder and a trained latent variable feature layer, improving the accuracy of camera dirt detection.

[0076] Furthermore, according to embodiments of this application, a vehicle is also provided, which may include the aforementioned camera dirt detection device 700.

[0077] Further, the present application also provides a computer storage medium, which stores a computer program. The computer program is executed to implement the camera dirty detection method 100 of the embodiments of the present application. The computer storage medium may, for example, include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer readable storage medium can be any combination of one or more computer readable storage media.

[0078] According to the embodiments of the present application, a computer program is also provided. The computer program is executed to implement the camera dirty detection method 100 described above.

[0079] Based on the above description, the camera dirty detection method, the device and the vehicle of the embodiments of the present application detect the camera dirty based on the trained encoder and the trained latent variable feature layer, and the accuracy of the camera dirty detection is improved.

[0080] Although the example embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the example embodiments are merely exemplary and are not intended to limit the scope of the present application. Those of ordinary skill in the art can make various changes and modifications without departing from the scope and spirit of the present application. All such changes and modifications are intended to be included within the scope of the present application as claimed in the appended claims.

[0081] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be realized by electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed by hardware or software depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0082] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be omitted or not executed.

[0083] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0084] Similarly, it is to be understood that the various features of the application can sometimes be used to advantage together, but each of the features can be used independently of one another. Furthermore, various embodiments of the present application have been described herein in terms of processes, procedures, and techniques. It should be understood that the embodiments of the application can be implemented in software, firmware, hardware, or any combination thereof. It should also be understood that the various features of the application can be used in any combination or sub-combination thereof.

[0085] Those skilled in the art will appreciate that all features described herein (including all accompanying claims, abstract and drawings), and processes or steps of any method or of any device described herein can be combined in any combination, except where features are mutually exclusive. Each feature disclosed in this description (including any accompanying claims, abstract and drawings) can be replaced by alternative features serving the same, equivalent or a similar purpose, unless expressly stated otherwise.

[0086] Furthermore, those skilled in the art will recognize that references in the specification to "one embodiment", "an embodiment", "an example embodiment", etc., mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment. Further, where a particular feature, structure, or characteristic is stated to have been added in an embodiment, it will be understood that to the extent possible, this particular feature, structure or characteristic has been added to that embodiment and all following embodiments.

[0087] Various component embodiments of the application can be implemented in hardware, or as software modules running in one or more processors, or in combinations thereof. Skilled persons will appreciate that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functionality of some of the modules according to embodiments of the application. The application can also be implemented as a program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such program(s) can be stored on a computer readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier medium, or in any other form.

[0088] It should be noted that the foregoing examples have been provided merely for the purpose of explanation and are in no way to be construed as limiting of the present application. While the application has been described with reference to preferred embodiments and illustrative examples, the words which have been used herein are words of description, and thus are used in a descriptive sense and not restrictive. In this detailed description, numerous specific details have been set forth in order to provide a thorough understanding of the application. However, it will be apparent to one skilled in the art that the application can be practiced without the specific details. In other instances, well-known methods have not been described in detail in order not to unnecessarily overshadow an aspect of the application. Accordingly, the scope of the application is defined only by the appended claims and equivalents thereto.

[0089] The specification and drawings should be considered in a descriptive sense only and not limiting on the scope of the present application. Furthermore, the description should not be construed as an exclusion of any further modifications in the scope of the claims. The objects and advantages of the application will become more fully understood from the following detailed description, along with the appended claims and from the drawings.

Claims

1. A camera dirt detection method, characterized by, The method comprises: inputting an image captured by a camera to be detected into a trained encoder to obtain image encoding output by the encoder; extracting latent variable features of the image encoding based on a trained latent variable feature layer; obtaining a dirtiness evaluation index according to the latent variable features, comprising: determining a mean vector and a variance vector of the latent variable features; obtaining a KL divergence according to the mean vector and the variance vector, the dirtiness evaluation index comprising the KL divergence; when the dirtiness evaluation index meets a preset condition, determining that the camera has dirtiness.

2. The method of claim 1, wherein, The method comprises: if the KL divergence of a single frame of image captured by the camera to be detected is greater than a first preset threshold, it is determined that the camera to be detected has dirtiness; and / or, if the KL divergence of at least two frames of image captured by the camera to be detected are all less than or equal to a first preset threshold and greater than a second preset threshold, it is determined that the camera to be detected has dirtiness, wherein the second preset threshold is less than the first preset threshold.

3. The method of claim 1, wherein, The training process of the encoder and the latent variable feature layer comprises: inputting an original image captured by a camera into an encoder to obtain original image encoding output by the encoder; extracting latent variable features of the original image encoding based on a latent variable feature layer; obtaining a sampling latent variable according to the latent variable features of the original image encoding; inputting the sampling latent variable into a decoder corresponding to the encoder to obtain a reconstructed image output by the decoder; updating parameters of the encoder and the latent variable feature layer based on a loss function to obtain a trained encoder and a trained latent variable feature layer, the loss function comprising at least a reconstruction loss determined according to the original image and the reconstructed image.

4. The method of claim 3, wherein, The method comprises: determining a mean vector and a variance vector of the latent variable features of the original image encoding, the mean vector and the variance vector conforming to a normal distribution; obtaining the sampling latent variable according to the mean vector, the variance vector and a normal distribution sampling, wherein the dimension of the normal distribution sampling is the same as that of the mean vector, and the normal distribution sampling is a random number satisfying the normal distribution.

5. The method of claim 4, wherein, The method comprises: determining a product of the normal distribution sampling and the variance vector, and obtaining the sampling latent variable according to a sum of the product and the mean vector.

6. The method of claim 3, wherein, The loss function further comprises a KL divergence loss determined based on the latent variable features of the original image encoding.

7. The method of claim 3, wherein, The original image captured by the camera is an image captured by a camera without dirtiness.

8. A camera dirt detection apparatus, characterized by, The camera dirtiness detection device comprises a memory and a processor, the memory stores a computer program run by the processor, and the computer program performs the camera dirtiness detection method of any one of claims 1-7 when run by the processor.

9. A vehicle characterized by comprising: The vehicle comprises the camera dirt detection device according to claim 8.

10. A computer storage medium having stored thereon a computer program, characterized in that The computer program, when executed, implements the camera dirt detection method according to any one of claims 1-7.

11. A computer program product, characterised in that, The computer program product is stored in a computer readable storage medium, and the computer program product, when executed, implements the camera dirt detection method according to any one of claims 1-7.

Images