Image restoration method, device, equipment and storage medium

Abstract

The present application relates to an image restoration method, apparatus, device and storage medium. The method comprises: calling a preset vehicle-mounted camera to capture a video stream; for each frame of a basic image in the video stream, generating a mask corresponding to the basic image; wherein the mask is used to identify the damaged area and the normal area of ​​the basic image; for each frame of a basic image in the video stream, according to the mask corresponding to the basic image, sequentially performing classification noise addition processing and classification denoising processing on the basic image, and, in the process of performing the classification denoising processing, repairing the damaged area of ​​the basic image according to the basic images of the previous and next frames of the basic image. The present application provides an image restoration method that focuses on the damaged area of ​​the basic image and integrates the contextual information of the damaged area. Compared with the traditional image restoration method, the present application can repair the damaged area in the basic image, and the restored image is more natural and realistic and can significantly reduce the distortion and artifacts in the restored image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image restoration method, apparatus, device and storage medium. Background Art

[0002] With the advancement of science and technology, computer vision technology has been greatly developed in many fields. For example, in the field of automotive technology, computer vision technology has been used to realize the function of electronic rearview mirror. That is, the image captured by the camera outside the vehicle is captured within the field of view of the traditional rearview mirror, and the image captured by the camera is displayed on the display inside the vehicle.

[0003] Currently, in electronic rearview mirror application scenarios, in order to enable users to view high-quality images, the basic image captured by the camera is repaired before the image is displayed. Existing image repair methods mainly focus on adjusting and optimizing the basic image, including: adjusting the contrast of the image, and adjusting the image brightness. However, only adjusting and optimizing the basic image lacks the ability to adaptively process different scenes, especially for the problem of image damage in complex scenes. There is a lack of effective repair methods. For example: for image damage caused by strong light, low light, rain, snow, dirt, etc., if only the image is adjusted and optimized, it is difficult to repair the damaged area of ​​the image, and the image quality cannot be improved. It may even cause the image quality to further deteriorate. As a result, in actual applications, the image quality of electronic rearview mirrors often cannot be substantially improved. Summary of the Invention

[0004] The present application provides an image restoration method, apparatus, device and storage medium to solve the problem that existing image restoration methods only adjust and optimize basic images and cannot repair damaged images.

[0005] In order to solve the above technical problems, the technical solution of this application is solved through the following embodiments:

[0006] An embodiment of the present application provides an image restoration method, comprising: calling a preset vehicle-mounted camera to capture a video stream; generating a mask corresponding to each frame of a basic image in the video stream; wherein the mask is used to identify damaged areas and normal areas of the basic image; for each frame of a basic image in the video stream, sequentially performing classified denoising processing and classified denoising processing on the basic image according to the mask corresponding to the basic image, and, during the classification denoising processing, repairing the damaged area of ​​the basic image according to the basic images of the previous and next frames of the basic image.

[0007] Wherein, for each frame of basic image in the video stream, generating a mask corresponding to the basic image includes: sequentially acquiring each frame of the basic image in the video stream; inputting the currently acquired basic image into a pre-trained mask generation model, and acquiring a mask output by the mask generation model and corresponding to the currently acquired basic image; wherein, the mask includes: at least one damaged channel; each damaged channel corresponds to a damage type, and each damaged type corresponds to a damaged area; and the mask generation model is used to: when detecting the damaged area and the normal area of ​​the basic image, identify the damaged type of the damaged area; and identify the damaged area and the normal area corresponding to the damaged type in the damaged channel corresponding to the damaged type.

[0008] In which, according to the mask corresponding to the basic image, the classified denoising process and the classified denoising process are sequentially performed on the basic image, including: in the classified denoising process stage, according to the mask corresponding to the basic image, according to a first preset time step, multiple steps of adding weighted Gaussian noise are added to the damaged area of ​​the basic image and multiple steps of adding Gaussian noise are added to the normal area of ​​the basic image until the basic image meets the preset complete Gaussian noise condition; in the classified denoising process stage, according to the mask corresponding to the basic image, according to a second preset time step, multiple denoising processes are performed on the damaged area and the normal area of ​​the basic image that meet the complete Gaussian noise condition respectively; wherein the number of denoising steps performed in the classified denoising process stage is the same as the number of denoising steps performed in the classified denoising process stage.

[0009] The step-weighted Gaussian noise is added multiple times to the damaged area of ​​the base image, including: the step-weighted Gaussian noise added each time can be calculated using the following formula:

[0010]

[0011] Among them, I t 受损 represents the step-weighted Gaussian noise added for the tth time; I0 受损 represents the preset initial Gaussian noise; w(M) represents the weight function; represents the cumulative Gaussian noise amount of the tth time;

[0012]

[0013] Wherein, w_I, w_G and w_A are all preset weights; f_I(ΔI) represents the pixel intensity difference coefficient between the damaged area and the normal area in the base image; represents the mask gradient coefficient of the damaged area in the basic image; f_A(A(M)) represents the mask area coefficient of the damaged area in the basic image.

[0014] Among them, the completely Gaussian noise condition includes: the number of noise addition steps performed in the classification noise addition processing stage reaches a preset step threshold; or the damaged area and the normal area of ​​the basic image are independent and identically distributed and obey the same standard Gaussian distribution.

[0015] Wherein, in the process of executing the classification denoising processing, the damaged area of ​​the basic image is repaired according to the basic images of the previous and next frames of the basic image, including: in the classification denoising processing stage, in the process of each denoising processing, for each pixel point in the damaged area of ​​the basic image, the local context feature value within a preset range of the corresponding position of the pixel point is determined in the normal area of ​​the basic images of the previous and next frames of the basic image; based on the local context feature value within the preset range of the corresponding position of the pixel point, the repair pixel value corresponding to the pixel point is determined and the original pixel value of the pixel point is replaced by the repair pixel value.

[0016] The local context feature value is a weighted average feature value calculated based on the pixel values ​​of each pixel within a preset range of the corresponding position of the pixel point in the normal area of ​​the previous and next frames of the basic image; the weighted average feature value is calculated using the following formula:

[0017]

[0018] Among them, μ context represents the weighted average eigenvalue; I(x′, y′) represents the pixel value of the pixel point within the preset range of the corresponding position of the pixel point I(x, y) in the normal area of ​​the previous and next frames of the basic image; w(xx′, yy′) represents the spatial weight coefficient of I(x′, y′); ∑ (x′，y′)：M(x′，y′)＝0 Indicates cumulative calculation;

[0019] The repaired pixel value corresponding to the pixel point is determined using the following formula:

[0020] I recovered (x, y) = α·μ context +(1-α)·I(x, y);

[0021] Among them, I recovered (x, y) represents the repaired pixel value corresponding to the pixel point I(x, y); α is the preset fusion coefficient.

[0022] An embodiment of the present application also provides an image repair device, including: a calling module, used to call a preset vehicle-mounted camera to capture a video stream; a generating module, used to generate a mask corresponding to each frame of basic image in the video stream; wherein the mask is used to identify the damaged area and normal area of ​​the basic image; a repair module, used to sequentially perform classified noise addition processing and classified denoising processing on each frame of basic image in the video stream according to the mask corresponding to the basic image, and, during the process of performing the classified denoising processing, repair the damaged area of ​​the basic image according to the basic images of the previous and next frames of the basic image.

[0023] An embodiment of the present application also provides an image restoration device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to: execute an image restoration program stored in the memory to implement any of the above-mentioned image restoration methods.

[0024] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed to implement any of the above-mentioned image restoration methods.

[0025] The above technical solution provided by the embodiment of the present application has the following advantages compared with the existing technology: the method provided by the embodiment of the present application can call a preset vehicle-mounted camera to capture a video stream; for each frame of the basic image in the video stream, a mask corresponding to the basic image is generated; wherein the mask is used to identify the damaged area and the normal area of ​​the basic image; for each frame of the basic image in the video stream, the basic image is sequentially subjected to classification noise addition processing and classification denoising processing according to the mask corresponding to the basic image, and, in the process of performing the classification denoising processing, the damaged area of ​​the basic image is repaired according to the basic images of the previous and next frames of the basic image. Based on this, the embodiment of the present application provides an image restoration method that focuses on the damaged area of ​​the basic image and integrates the contextual information of the damaged area. Compared with the traditional image restoration method, the embodiment of the present application can repair the damaged area in the basic image, and the restored image is more natural and realistic and can significantly reduce the distortion and artifacts in the restored image. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0028] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0029] Figure 1 is a flowchart of an image restoration method according to an embodiment of the present application;

[0030] Figure 2 is a flowchart of image restoration steps according to an embodiment of the present application;

[0031] Figure 3 is a structural diagram of an image restoration device according to an embodiment of the present application;

[0032] Figure 4 2 is a structural diagram of an image restoration device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0034] The disclosure below provides many different embodiments or examples for implementing different structures of the present application. In order to simplify the disclosure of the present application, the components and settings of specific examples are described below. Of course, these are merely examples and are not intended to limit the present application. In addition, the present application may repeat reference numbers and / or letters in different examples. Such repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0035] The embodiment of the present application provides an image repair method. Figure 1 FIG. 1 is a flow chart of an image restoration method according to an embodiment of the present application.

[0036] Step S110: calling a preset vehicle-mounted camera to capture a video stream.

[0037] Types of vehicle-mounted cameras include, but are not limited to, vehicle-mounted cameras for electronic rearview mirrors and vehicle-mounted cameras for intelligent driving.

[0038] The preset type of vehicle-mounted camera is called upon demand to collect a video stream, wherein the video stream is a continuous video frame composed of basic images collected frame by frame in chronological order from the earliest to the latest.

[0039] Step S120 : generating a mask corresponding to each frame of the basic image in the video stream; wherein the mask is used to identify a damaged area and a normal area of ​​the basic image.

[0040] The damaged area refers to an area in the base image where image damage occurs.

[0041] Normal areas refer to areas in the base image that do not show any damage.

[0042] Specifically, as the vehicle-mounted camera sequentially captures base images, a mask corresponding to each frame is generated. The base image and its mask have the same size. The damaged and normal areas are identified in the mask, with the locations of the damaged areas in the mask corresponding to the damaged areas in the base image, and the locations of the normal areas in the mask corresponding to the normal areas in the base image.

[0043] Furthermore, the sizes of the basic images of each frame sequentially captured by the vehicle-mounted camera and the masks corresponding to each basic image frame are the same. Of course, the basic images of each frame and / or the masks corresponding to each basic image frame can also be preprocessed so that the sizes of the basic images of each frame and the masks corresponding to each basic image frame are the same.

[0044] Step S130, for each frame of the basic image in the video stream, sequentially perform classification noise addition processing and classification denoising processing on the basic image according to the mask corresponding to the basic image, and in the process of performing the classification denoising processing, repair the damaged area of ​​the basic image according to the basic images of the previous and next frames of the basic image.

[0045] Classification noise processing refers to adding different Gaussian noises to the damaged area and normal area of ​​the base image in multiple steps.

[0046] Classification denoising processing refers to denoising the damaged area and normal area of ​​the basic image with Gaussian noise added in multiple steps.

[0047] The previous and next frame basic images include: a previous frame basic image and / or a next frame basic image of the current basic image.

[0048] Specifically, based on the previous frame basic image and / or the next frame basic image of the basic image, the local context feature value of the damaged area of ​​the current basic image is determined; based on the local context feature value of the damaged area of ​​the current basic image, the damaged area of ​​the current basic image is repaired.

[0049] Furthermore, the sequentially obtained restored images can be displayed in sequence on a display so that a user can view the restored video stream.

[0050] In an embodiment of the present application, a preset vehicle-mounted camera is called to capture a video stream; for each frame of a basic image in the video stream, a mask corresponding to the basic image is generated; wherein the mask is used to identify the damaged area and the normal area of ​​the basic image; for each frame of a basic image in the video stream, classified noise addition processing and classified denoising processing are sequentially performed on the basic image according to the mask corresponding to the basic image, and, in the process of performing the classified denoising processing, the damaged area of ​​the basic image is repaired according to the basic images of the previous and next frames of the basic image. Based on this, an embodiment of the present application provides an image restoration method that focuses on the damaged area of ​​a basic image and integrates the contextual information of the damaged area. Compared with traditional image restoration methods, the embodiment of the present application can repair the damaged area in the basic image, and the restored image is more natural and realistic and can significantly reduce the distortion and artifacts in the restored image.

[0051] In order to make the embodiments of the present application easier to understand, the image restoration method of the embodiments of the present application will be further described below.

[0052] In the embodiment of the present application, generating a mask corresponding to a base image is the basis of image restoration. The process of generating a mask corresponding to a base image is described below.

[0053] Specifically, when a preset vehicle-mounted camera is used to capture a video stream, each frame of the base image is sequentially acquired from the video stream. The currently acquired base image is input into a pre-trained mask generation model, and a mask corresponding to the currently acquired base image, output by the mask generation model, is obtained. The mask includes at least one damaged channel, each damaged channel corresponding to a damage type, and each damage type corresponding to a damaged area. In other words, each damaged channel can identify a damaged area.

[0054] The embodiment of the present application pre-trains a mask generation model. The mask generation model is used to: when detecting damaged areas and normal areas of the base image, identify the damage type of the damaged area; and identify the damaged area and normal area corresponding to the damage type in the damaged channel corresponding to the damage type.

[0055] When choosing a mask generation model, you can choose a pre-trained model. A pre-trained model is one that has been trained using a large-scale dataset and has the ability to detect damaged areas and generate masks. Pre-trained models include, but are not limited to, Mask R-CNN, U-Net, and other image segmentation models.

[0056] In the training of the mask generation model, the pre-trained model is first loaded to ensure that the weight parameters of the pre-trained model remain unchanged. Then, the pre-trained model is trained again using the real base image generated in the real task and the real mask set corresponding to the real base image, so that the parameters in the pre-trained model can be fine-tuned according to the image characteristics of the real task, and finally the mask generation model is obtained. Compared with the pre-trained model, the mask output by the mask generation model obtained after fine-tuning the parameters is more accurate. Furthermore, since the pre-trained model has been trained on a large-scale dataset, it only needs to be fine-tuned on the new task, and there is no need to train from scratch, which can greatly reduce the model training time and computing resources. The training of the mask generation model can be performed on a cloud server.

[0057] Before using the mask generation model, the base image captured by the vehicle camera is preprocessed before being fed into the mask generation model. This preprocessing includes adjusting the size of the base image, unifying the base image size, and ensuring that the base image and the mask are the same size; normalizing the pixel values ​​in the base image, unifying the numerical ranges to which the pixel values ​​belong, and ensuring that the pixel values ​​of different base images fall within the same numerical range. The preprocessed base image is then fed into the mask generation model, which performs inference calculations on the base image and, based on the weights and structural features within the mask generation model, automatically identifies damaged and normal areas of the base image and distinguishes the types of damage corresponding to the damaged areas.

[0058] Furthermore, the mask is a binary or multi-class labeled image of the same size as the base image. The mask generation model can perform pixel-by-pixel segmentation, predicting the damage type for each pixel in the base image and labeling it with the value corresponding to the damage type. Pixel values ​​corresponding to damaged areas are assigned the value corresponding to their damage type, while pixel values ​​corresponding to normal areas are assigned the value corresponding to non-damaged areas, thereby forming a mask.

[0059] Furthermore, the mask is a binary mask image or a multi-channel mask image. In a binary mask image, the pixels in the damaged area are assigned a non-zero value (such as white or a specific value), indicating that they are part of the damaged area; and the non-damaged area (normal area) is assigned a zero value (such as black or another specific value), indicating that it is a normal and undamaged area. Of course, there may be multiple types of damage in the base image. For multiple types of damage, the mask can contain multiple damaged channels, each damaged channel corresponds to a type of damage, and the pixel value of the damaged area corresponding to each damage type on the corresponding channel is a non-zero value, and the other areas are zero values. The non-zero values ​​corresponding to different damage types may be different.

[0060] For example: for an overexposed area, the damaged category code corresponding to the overexposed area is marked in the damaged channel corresponding to the overexposed area; for a highlight overflow area, the damaged category code corresponding to the highlight overflow area is marked in the damaged channel corresponding to the highlight overflow area.

[0061] Furthermore, the damage type can reflect information such as the damage cause, damage scenario, damage classification, etc. For example, the damage type and its specific description are shown in Table 1. Of course, those skilled in the art should understand that Table 1 is only used to illustrate this embodiment and is not used to limit this embodiment.

[0062]

[0063]

[0064]

[0065]

[0066] Table 1

[0067] In the embodiments of the present application, a mask can be used as an indicator of the damaged area of ​​the base image. This mask can provide a clear target and scope for subsequent image restoration steps. Based on the mask corresponding to the base image, the damaged area in the base image that needs to be restored can be accurately identified, focusing on the damaged area specified by the mask and attempting to restore the image content in the damaged area, thus performing targeted image restoration instead of indiscriminately processing the entire base image. This allows for accurate and efficient image restoration.

[0068] Specifically, after generating the mask corresponding to the base image, the classification noise addition processing and the classification denoising processing are sequentially performed on the base image according to the mask corresponding to the base image. Moreover, during the classification denoising processing, the damaged area of ​​the base image is repaired according to the base images of the previous and next frames of the base image.

[0069] like Figure 2FIG. 1 is a flowchart of image restoration steps according to an embodiment of the present application.

[0070] Step S210, in the classification and noise addition processing stage, according to the mask corresponding to the basic image, according to the first preset time step, step-weighted Gaussian noise is added multiple times in the damaged area of ​​the basic image and step-weighted Gaussian noise is added multiple times in the normal area of ​​the basic image until the basic image meets the preset complete Gaussian noise condition.

[0071] The first preset time step is an empirical value or a value obtained through experiments.

[0072] At every first preset time step, step-weighted Gaussian noise is added once to the damaged area of ​​the base image and once to the normal area of ​​the base image, and it is detected whether the current base image meets the complete Gaussian noise condition. If the current base image does not meet the complete Gaussian noise condition, step-weighted Gaussian noise is continued to be added to the damaged area of ​​the base image and step-weighted Gaussian noise is continued to be added to the normal area of ​​the base image at intervals of the first preset time step.

[0073] Step S220, in the classification denoising stage, according to the mask corresponding to the basic image, according to the second preset time step, multiple denoising processes are performed on the damaged area and normal area of ​​the basic image that meets the complete Gaussian noise condition; wherein, in each denoising process, the damaged area of ​​the basic image is repaired once according to the previous and next frame basic images of the basic image and the mask corresponding to the basic image.

[0074] The second preset time step is an empirical value or a value obtained through experiments. The first preset time step and the second preset time step are the same.

[0075] In each step of the classification noise addition process, in the damaged area of ​​the base image, the step-weighted Gaussian noise added each time can be calculated using the following formula:

[0076]

[0077] Among them, I t 受损 represents the step-weighted Gaussian noise added for the tth time; I0 受损 represents the preset initial Gaussian noise; w(M) represents the weight function; Indicates the cumulative Gaussian noise amount of the tth time. N i (0,σi 2 ) represents zero mean and the noise variance is σi 2 Gaussian noise.

[0078] w(M) is used to adjust the amount or characteristics of noise added for the tth time. The larger the value of w(M), the more step-weighted Gaussian noise is added to the damaged area. This simulates noise diffusion while more realistically reflecting the impact of damage on image quality, allowing for more accurate restoration of the original information of the damaged area during subsequent inverse processing. The size of w(M) can be determined based on the current base image. Furthermore, w(M) reflects the characteristic strength of the damaged area, including pixel intensity differences, mask gradients, and mask areas. w(M) is calculated by operating on these characteristic strengths.

[0079]

[0080] Wherein, w_I, w_G and w_A are all preset weights; f_I(ΔI) represents the pixel intensity difference coefficient between the damaged area and the normal area in the base image; represents the mask gradient coefficient of the damaged area in the basic image; f_A(A(M)) represents the mask area coefficient of the damaged area in the basic image.

[0081] w_I is the weight for the pixel intensity difference coefficient. Its initial value can be set to a small positive value, such as 0.2. If the pixel intensity difference in the damaged area is important for noise injection, this weight can be increased appropriately. Through experimental verification and adjustment, a value that effectively reflects pixel differences while balancing other factors can be found.

[0082] w_G is the weight for the mask gradient coefficient. Its initial value can be set to a moderately positive value, such as 0.5. The mask gradient reflects the edge strength of the damaged area. Stronger edges may require more noise processing. If edge information is crucial for the noise injection effect, w_G can be increased.

[0083] w_A is the weight for the mask area coefficient. Its initial value can be set to a small positive value, such as 0.3. The mask area reflects the size of the damaged area. Larger damaged areas require more attention, but at the same time, it is important to prevent noise contamination caused by overly large areas. In practice, w_A can be adjusted based on the size and importance of the damaged area.

[0084] f_I(ΔI)=1 / (1+exp(-k I *mean(ΔI));

[0085] Where, mean(ΔI) represents the average value of the pixel intensity difference between the damaged area and the normal area, which reflects the overall intensity change of the damaged area. exp represents the exponential function. k Ik is an adjustable parameter (positive scalar coefficient) used to control the steepness of the f_I(ΔI) curve. I The larger it is, the more sensitive f_I(ΔI) is to changes in mean(ΔI), and even small intensity differences can lead to significant changes in the output.

[0086]

[0087] in, It is the L2 norm of the mask gradient of the damaged area, which reflects the edge strength or mutation degree of the damaged area. The larger the value, the more obvious the boundary between the damaged area and the surrounding environment, and the greater the possibility of being disturbed by strong light or the degree of impact. G is an adjustable parameter (positive scalar coefficient) that controls The size of k G The bigger, Giving a higher weight to the mask gradient means that the amount of noise added in the damaged area with strong edges may be greater; conversely, k G The smaller, The response to the mask gradient will be weakened, and the amount of noise added will be reduced accordingly.

[0088]

[0089] Where A(M) is the area of ​​the damaged area mask, reflecting the proportion of the damaged area in the base image. A_total is the area of ​​the base image, which is used to normalize the area of ​​the damaged area to ensure that the result of f_A(A(M)) is between 0 and 1. A is an adjustable parameter (positive power exponential coefficient) that controls the size of f_A(A(M)), which determines the sensitivity of f_A(A(M)) to the size of the damaged area. Furthermore, if the area of ​​the damaged area is larger than the preset large area threshold, the large damaged area can be processed more intensively, that is, k is set A > 1, in this case, f_A(A(M)) gives a higher weight to the large damaged area, and the large damaged area will add more noise; if the area of ​​the damaged area is smaller than the small area threshold and the damaged area includes the preset key image (such as license plate, traffic sign, etc.), the value of kA can be appropriately reduced to focus on the restoration of the small damaged area, that is, set 0 < k A When ≤1, the weight of f_A(A(M)) on large damaged areas will be reduced, and more emphasis will be placed on noise processing in small but critical areas.

[0090] Furthermore, in each step of the classification noise addition process, in the normal area of ​​the basic image, the stepped Gaussian noise added each time can be calculated using the following formula:

[0091]

[0092] Among them, I t 正常 represents the stepped Gaussian noise added for the tth time; I0 受损 Represents the preset initial Gaussian noise; Indicates the cumulative Gaussian noise amount at the tth time.

[0093] Furthermore, at each step of the classification and noise addition process, a determination is made as to whether the base image satisfies a completely Gaussian noise condition. Furthermore, the completely Gaussian noise condition includes: the number of noise addition steps performed in the classification and noise addition process reaches a preset threshold; or the damaged and normal regions of the base image are independent and identically distributed and follow the same standard Gaussian distribution.

[0094] Complying with independent and identical distribution means that the damaged area and the normal area of ​​the basic image do not affect each other and obey the same probability distribution. Among them, no influence means that the probability distributions of the damaged area and the normal area do not depend on each other, that is: p(I i , I j )=p(I i )p(I j ); where p(I i , I j ) represents the probability distribution of the basic image, p(I i ) represents the probability distribution of the damaged area, p(I j ) represents the probability distribution of the normal area. Obeying the same probability distribution means p(I i ) and p(I j ) have the same distribution function, distribution shape, distribution parameters, and have the same expectation and variance.

[0095] Obeying the same standard Gaussian distribution means that the mean, variance, and histogram of the damaged and normal areas of the base image have the following characteristics:

[0096] Mean: For a standard Gaussian distribution, the expected mean μ should be close to 0, that is, μ≈E[x]≈0;

[0097] Variance: For a standard Gaussian distribution, the expected variance σ 2 Close to 1, that is: σ 2 ≈Var[x]≈1;

[0098] Histogram: For a standard Gaussian distribution, the probability density function conforms to the following formula:

[0099]

[0100] Furthermore, the meaning of a calculation result being close to (approximately equal to) a numerical value is that the calculation result is within a preset numerical range of the numerical value, which is a smaller range.

[0101] Furthermore, the base image that meets the fully Gaussian noise condition is input into a preset denoising network, which performs classification denoising. The denoising network can be a deep neural network. For example, the denoising network can be a convolutional neural network (CNN) or a Transformer. The denoising network can be used to restore the conditional distribution of the original image. Each time the denoising network performs a denoising operation, it uses the following formula to remove Gaussian noise from the base image.

[0102] I t-1 受损 =DNC t (I t 受损 , M);

[0103] I t-1 正常 =DNC t (I t 正常 );

[0104] Among them, I t-1 受损 Denotes the denoising network DNC t The Gaussian noise removed in the damaged area for the tth time; I t 受损 Represents the current Gaussian noise in the damaged area; I t-1 正常 Denotes the denoising network DNC t The Gaussian noise removed in the normal area for the tth time; I t 正常 Indicates the current Gaussian noise in the normal area.

[0105] In an embodiment of the present application, a mask M is introduced into the denoising network. The mask is used to enhance the characteristic response of the damaged area, so that the denoising network can perform targeted and refined denoising operations on the damaged area according to the mask, rather than performing indiscriminate denoising operations on the entire basic image. This can greatly improve the efficiency and accuracy of denoising, especially for images that are only partially damaged, the denoising is more accurate.

[0106] In the embodiment of the present application, the number of denoising steps performed in the classification denoising process is the same as the number of denoising steps performed in the classification denoising process. In other words, the number of times Gaussian noise is added to the base image is the same as the number of times the denoising process is performed.

[0107] Specifically, the step thresholds corresponding to the number of denoising steps and the number of denoising steps can be preset, or, in each step executed in the classification denoising processing stage, Gaussian noise (including step-weighted Gaussian noise and step-weighted Gaussian noise) is added to the base image. As the Gaussian noise gradually increases, the base image gradually loses its original structure and content. When the base image meets the complete Gaussian noise condition, stop adding Gaussian noise to the base image again, record the number of times Gaussian noise is added to the base image (this number is the number of denoising steps), and perform denoising processing this number of times in the classification denoising processing stage.

[0108] In an embodiment of the present application, the damaged area in the base image can be gradually repaired in the classification and denoising processing stage. Since there is often a high degree of correlation between adjacent frames, the continuity of adjacent frames in time can be used to repair the damaged frame. The context features corresponding to the damaged area of ​​the current frame can be extracted in the frames before and after the current frame. Context features include but are not limited to: the motion trajectory of dynamic objects, the stable part of the background, changes in lighting conditions, etc. By comparing and analyzing these temporal context features, it is inferred that the state of the damaged area in the current frame in the continuous frames should be restored, and a more reasonable and natural repaired image can be repaired.

[0109] For example, in a night scene with an electronic rearview mirror, if the current frame is affected by strong light and some areas are blurred, the damaged area of ​​the current frame can be repaired by referring to the image information at the same position in the previous or next frame. In other words, the true content of the missing or overexposed part can be inferred based on the trajectory of the moving object and the invariance of the scene.

[0110] Specifically, in the classification denoising processing stage, in each denoising process, for each pixel point in the damaged area of ​​the basic image, the local context feature value within a preset range of the corresponding position of the pixel point is determined in the normal areas of the basic images of the previous and next frames of the basic image; based on the local context feature value within the preset range of the corresponding position of the pixel point, the repair pixel value corresponding to the pixel point is determined and the original pixel value of the pixel point is replaced with the repair pixel value.

[0111] The local context feature value is: a weighted average feature value calculated based on the pixel values ​​of each pixel point within a preset range corresponding to the pixel point in the normal area of ​​the previous and next frame basic images (the previous frame and the next frame) of the basic image.

[0112] The weighted average eigenvalue is calculated using the following formula:

[0113]

[0114] Among them, μ contextrepresents the weighted average eigenvalue; I(x′, y′) represents the pixel value of the pixel point within the preset range of the corresponding position of the pixel point I(x, y) in the normal area of ​​the previous and next frames of the basic image; w(xx′, yy′) represents the spatial weight coefficient of I(x′, y′); ∑ (x′，y′)：M(x′，y′)＝0 Indicates cumulative calculation. For ease of understanding, I(x′, y′) represents a pixel point in description, and I(x′, y′) represents the pixel value of the pixel point in calculation.

[0115] Furthermore, the size of the preset range can be an empirical value. The distance or similarity between pixel point I(x, y) and pixel point I(x', y') is positively correlated with the spatial weight coefficient of pixel point I(x', y'). For example: multiple distance intervals are pre-set, each distance interval corresponds to a spatial weight coefficient and as the end value of the distance interval increases, the value of the spatial weight coefficient increases. In this way, the spatial weight coefficient of I(x', y') can be determined based on the distance interval to which the distance between I(x, y) and I(x', y') belongs.

[0116] Furthermore, the denominator of the weighted average eigenvalue can be obtained as follows: for each pixel within the damaged area of ​​the current base image, determine the pixel corresponding to the position of the pixel in the previous frame of the base image. If the pixel corresponding to the position is in the normal area of ​​the previous frame of the base image, obtain the pixel values ​​of all pixels within a preset range of the pixel corresponding to the position in the normal area of ​​the previous frame of the base image, determine the distance between each pixel within the preset range and the pixel corresponding to the position, determine the spatial weight coefficient corresponding to the distance, and calculate the sum of the pixel value of each pixel within the preset range and the spatial weight coefficient corresponding to the pixel. Perform the same steps for the next frame of the base image. Accumulate the sum values ​​corresponding to all pixels within the preset range in the previous frame of the base image and the sum values ​​corresponding to all pixels within the preset range in the next frame of the base image to obtain the denominator of the weighted average eigenvalue.

[0117] Furthermore, the numerator part of the weighted average eigenvalue can be obtained in the following way: accumulate the spatial weight coefficients corresponding to all pixel points within a preset range in the previous frame basic image and the spatial weight coefficients corresponding to all pixel points within a preset range in the next frame basic image to obtain the numerator value of the weighted average eigenvalue.

[0118] The repaired pixel value corresponding to the pixel point is determined using the following formula:

[0119] I recovered (x, y) = α·μ context +(1-α)·I(x, y);

[0120] Among them, I recovered (x, y) represents the repaired pixel value corresponding to the pixel point I(x, y); α is the preset fusion coefficient.

[0121] In an embodiment of the present application, the spatial weight coefficient is first used for feature fusion, that is: in the local feature collection stage, the spatial weight coefficient is introduced, and the weight is assigned according to the distance or similarity between adjacent pixels, so that the pixels that are closer to the damaged area or have higher texture similarity contribute more to the repair. This weight-based feature fusion method improves the quality and accuracy of the repair results. The embodiment of the present application dynamically mixes the original damaged pixels and local context features through the fusion coefficient, that is: in the feature propagation and repair stage, the fusion coefficient is used to balance the degree of fusion between the newly filled local context features and the original damaged pixel values. This dynamic mixing method makes the repair process more flexible, and can be adaptively adjusted according to the specific circumstances of the damaged pixels to achieve better repair effects.

[0122] In an embodiment of the present application, an image restoration model can be used to implement the above-mentioned image restoration steps. Furthermore, during the process of capturing a video stream by an on-board camera, the video stream and the mask corresponding to each frame of the basic image in the video stream can be input into a pre-trained image restoration model, so that the image restoration model sequentially performs classification noise addition processing and classification denoising processing on each frame of the basic image according to the mask corresponding to each frame of the basic image, and during the classification denoising processing, the damaged area of ​​the basic image is restored based on the previous and next frame basic images of each frame of the basic image.

[0123] Specifically, the image restoration model is a conditional diffusion model. The conditional diffusion model can simulate the noise diffusion and denoising process, gradually resolving a blurred image and thus restoring the image. The conditional diffusion model in the embodiment of the present application comprehensively considers the damaged areas in the image during both the positive and negative diffusion processes and performs targeted processing on them, thereby achieving high accuracy in image restoration.

[0124] Furthermore, the image restoration model includes a forward diffusion network and a reverse sampling network (i.e., a denoising network). The forward diffusion network performs a classification denoising process on each frame of the base image based on a mask corresponding to each frame of the base image, and the reverse sampling network performs a classification denoising process on each frame of the base image based on a mask corresponding to each frame of the base image. During the classification denoising process, the damaged area of ​​the base image is restored based on the previous and next base images of each frame of the base image, thereby obtaining a restored image.

[0125] Furthermore, during the forward diffusion process, the forward diffusion network references the damaged areas in the base image and performs targeted processing on them, allowing the image restoration model to pay more attention to the details of these damaged areas. In other words, the network strengthens the learning of the feature responses of the masks of these damaged areas. During the reverse sampling process, the reverse sampling network performs refined denoising operations on the damaged areas based on the learned feature responses. Specifically, this involves correcting the pixel values ​​of the damaged areas and reconstructing the texture and structure of the damaged areas. The reverse sampling network considers the additional information provided by the masks of the damaged areas and places greater emphasis on the denoising performance of the damaged areas during the fitting process. Therefore, the reverse sampling network can more accurately predict the image content of these damaged areas when they were not damaged.

[0126] Furthermore, if a pre-trained image restoration model is used for image restoration, it should be noted that in actual applications, due to various factors (such as numerical precision limitations, computing resource constraints, etc.), the forward diffusion network of the image restoration model may not strictly reach a completely Gaussian noise state when performing classification and noise processing on the base image. For example, each pixel may still contain tiny information of the original image, so model training is required to enable the image restoration model to be able to add noise to the base image to a state close to completely Gaussian noise.

[0127] In terms of model training, this is reflected in the continuous adjustment of the parameters of the image restoration model and the continuous optimization of its structure, so that the image restoration model can implement the image restoration process of the embodiment of the present application. The image restoration model can include the above-mentioned algorithm formula, and the weights and coefficients in the above-mentioned algorithm formula can be used as parameters to be trained in the image restoration model.

[0128] In the embodiments of the present application, multiple sample groups are required before training the image restoration model. Each sample group includes a damaged image and a normal image corresponding to the same image content, as well as a mask corresponding to the damaged image. The damaged image refers to a base image containing both damaged and normal areas. The normal image refers to an undamaged base image.

[0129] For example: 正常图像 =I 受损图像 +L; where I 正常图像 Represents a normal image; I 受损图像 Indicates a damaged image; L indicates noise or distortion caused by image damage. 正常图像 , I 受损图像 Both L and L represent pixels in the image in the form of vector matrices.

[0130] During model training, the image restoration model is trained using multiple sample groups. For each sample group, the damaged image and its corresponding mask in the sample group are input into the image restoration model to obtain a restored image output by the image restoration model. A preset loss function is used to determine the loss value between the restored image and the normal image in the sample group. When the loss value is greater than a preset loss threshold, the parameters in the image restoration model are adjusted. When the loss value is less than or equal to the loss threshold for a preset number of consecutive times, the image restoration model is determined to have converged. Types of loss functions include, but are not limited to, mean square error function, L1 norm, L2 norm, or perceptual loss function.

[0131] For example: the loss function is:

[0132]

[0133] Among them, Loss t Indicates the loss value; I 修复图像 Indicates the repaired image; Represents the L2 norm. By continuously iterating and minimizing the loss, the image restoration model is trained to recover the original image from noise.

[0134] In an embodiment of the present application, a forward diffusion network and a reverse sampling network in an image restoration model are used to perform classified denoising and classified denoising on a base image. During the classification denoising process, the damaged areas in the base image are restored to obtain a restored base image. During the restoration of the base image, the reverse sampling network can analyze the global structure of the base image, including features such as edges, textures, and color distribution. Combining this contextual information, the reverse sampling network gradually propagates information from the undamaged area to the damaged area through interpolation, diffusion, or learning.

[0135] For example, in image restoration of an electronic rearview mirror, if the brightness of a certain area is too high, resulting in information loss, the reverse sampling network can use the pixels with moderate or low brightness around the area (normal area) in the previous and next frames, starting from the neighborhood of the affected pixels, to analyze the brightness, color, and texture information of the surrounding areas not affected by strong light. By propagating the features of these neighboring pixels, the overexposed areas are gradually compensated, so that the restored image remains consistent with the surrounding environment.

[0136] For example, in a road image, if the damaged area is located on the road, the reverse sampling network can use the structure and texture information of the surrounding normal road to infer the content of the damaged area. By combining this contextual feature information, it can produce more reasonable and natural restoration results.

[0137] In an embodiment of the present application, the training of the image restoration model can be performed on a cloud server. After the training of the image restoration model is completed, the image restoration model is set in the vehicle.

[0138] In an embodiment of the present application, the Generative Adversarial Network (GAN) technology can be combined to improve the authenticity and naturalness of the restoration results.

[0139] Furthermore, after the image restoration model training is completed, an adversarial network is constructed. The adversarial network includes: a generator and a discriminator. In an embodiment of the present application, the trained image restoration model is used as a generator in the adversarial network. In this way, the generator is used to output a restored image corresponding to the damaged image. The restored image is input to the discriminator. The discriminator is used to make a true or false judgment on the input restored image, that is, to judge whether the input restored image is a real image or a restored image generated by the generator. In the process of training the adversarial network, the generator and the discriminator are adversarially trained by minimizing the loss function of the adversarial network, encouraging the discriminator to correctly distinguish real images, and requiring the generator to deceive the discriminator so that the discriminator determines that the input restored image is a real image. In this way, the generator gradually improves the quality of the generated image and makes it more realistic.

[0140] In an embodiment of the present application, the image restoration model can be pruned to reduce the number of model parameters and computational complexity while maintaining model performance. Furthermore, the image restoration model is divided into multiple functional modules, and each time a functional module is stopped, the image quality of the restored image output by the image restoration model is determined. If the image quality of the restored image has not changed compared to before the functional module was stopped, the stopped functional module is removed from the image restoration model. Otherwise, the stopped functional module is retained in the image restoration model. By fine-tuning the pruning ratio and quantization accuracy of the model, a higher inference speed can be achieved while ensuring the restoration quality.

[0141] In an embodiment of the present application, in order to improve the image restoration speed, parallel processing technology can be introduced, and multiple GPUs (graphics processing units) can be used to jointly run the image restoration model. For example, the image restoration model is divided into multiple parts, and each part runs on a GPU.

[0142] In an embodiment of the present application, in order to improve the quality of image restoration, a feedback mechanism can be introduced. After the restored image is displayed to the user (driver), the user's image quality evaluation information of the restored image is collected, and the image restoration model is retrained based on the image quality evaluation information.

[0143] Furthermore, drivers can evaluate and provide feedback on the restoration results. This feedback can be used to further optimize the image restoration model and improve restoration effectiveness. Specifically, after the image restoration model completes the restoration of the base image, drivers can rate the restoration results or provide a written evaluation. Written evaluations provide drivers with more room for expression, allowing them to, for example, describe in detail the strengths and weaknesses of the restoration results. After collecting driver feedback, this data can be used to further optimize the image restoration model. First, driver ratings and comments can be analyzed to identify common problems or shortcomings in the restoration results, and the image restoration model can be retrained to address these issues. For example, if drivers generally report poor restoration results for a certain type of image, additional training data can be added or model parameters adjusted for these images. Second, driver feedback can be used to establish a feedback loop, allowing the image restoration model to improve over time. For example, driver feedback can be used as new training data to retrain the image restoration model. Through continuous iteration and optimization, the image restoration model can gradually adapt to various complex scenarios and changes, improving the stability and accuracy of the restoration results.

[0144] For example: Setting up a feedback interface: providing drivers with a feedback interface that allows them to easily provide feedback on the repair results, including: a slider, a five-star system, a rating system, or a simple questionnaire. Data analysis and processing: analyzing the collected feedback information and extracting key indicators, such as satisfaction scores, common problems, etc. These indicators will help identify the strengths and weaknesses of the model. Model optimization: adjusting and optimizing the image repair model based on feedback information, including but not limited to: adjusting model parameters, improving algorithm processes, or introducing new technologies to improve repair results. Iterative update: redeploying the optimized image repair model to the vehicle system and conducting real vehicle tests to verify the improved performance. Continuous improvement: establishing a continuous improvement mechanism, regularly collecting feedback information and optimizing the model to ensure that the algorithm always remains in the best state to meet the needs of the driver.

[0145] The embodiments of the present application target complex special scenarios, such as strong light, weak light, rain, snow, partial dirt caused by mud, vibration, impact, internal reflection, and other special scenarios that may be encountered during vehicle driving. The mask-based damaged area detection can quickly locate the damaged area; by constructing a conditional diffusion model based on contextual information and using deep learning methods, automatic image repair is performed. Compared with traditional image processing methods and threshold settings, this method is more intelligent and more generalized and can adapt to different driving scenarios; the embodiments of the present application lightweight the network structure of the model through model pruning, and can achieve real-time repair and display of video streams, thereby improving the safety of the driving process; the embodiments of the present application introduce a feedback mechanism to further optimize the model and improve the repair effect.

[0146] Furthermore, in the embodiment of the present application, the computational complexity and inference speed of the image restoration model are optimized for the special needs of the electronic rearview mirror application scenario, so that it can meet the requirements of real-time image restoration. The driver can instantly obtain clear, high-quality images while driving. By repairing the image in the electronic rearview mirror in real time, the visual interference of the driver in complex environments can be reduced, and the driving comfort and safety can be improved. In addition, due to the introduction of the user feedback mechanism, this solution can also continuously optimize the restoration effect according to the real-time feedback of the driver, further enhancing the user experience. Moreover, the embodiment of the present application is not only applicable to the electronic rearview mirror application scenario, but can also be expanded to other application scenarios that require real-time image restoration, such as automatic driving systems, scenarios that require the use of vehicle-mounted cameras to capture images, etc.

[0147] The present application also provides an image restoration device. Figure 3 , which is a structural diagram of an image restoration device according to an embodiment of the present application.

[0148] The image restoration device comprises:

[0149] The calling module 310 is used to call a preset vehicle-mounted camera to collect a video stream.

[0150] The generating module 320 is configured to generate a mask corresponding to each frame of the basic image in the video stream, wherein the mask is used to identify a damaged area and a normal area of ​​the basic image.

[0151] The repair module 330 is used to sequentially perform classified noise addition processing and classified denoising processing on each frame of the basic image in the video stream according to the mask corresponding to the basic image, and, during the classification denoising processing, repair the damaged area of ​​the basic image according to the basic images of the previous and next frames of the basic image.

[0152] The functions of the device described in the embodiment of the present application have been described in the above method embodiment. Therefore, for any details not fully described in the description of this embodiment, please refer to the relevant description in the above embodiment and will not be repeated here.

[0153] The present application also provides an image restoration device, such as Figure 4 , which is a structural diagram of an image restoration device according to an embodiment of the present application.

[0154] The image restoration device includes a processor 410 , a communication interface 420 , a memory 430 , and a communication bus 440 . The processor 410 , the communication interface 420 , and the memory 430 communicate with each other via the communication bus 440 .

[0155] The memory 430 is used to store computer programs.

[0156] In one embodiment of the present application, the processor 410, when executing the program stored on the memory 430, implements the image repair method provided by any of the aforementioned method embodiments, including: calling a preset vehicle-mounted camera to capture a video stream; for each frame of basic image in the video stream, generating a mask corresponding to the basic image; wherein the mask is used to identify the damaged area and normal area of ​​the basic image; for each frame of basic image in the video stream, according to the mask corresponding to the basic image, sequentially performing classified noise addition processing and classified denoising processing on the basic image, and, in the process of executing the classified denoising processing, repairing the damaged area of ​​the basic image according to the basic images of the previous and next frames of the basic image.

[0157] Wherein, for each frame of basic image in the video stream, generating a mask corresponding to the basic image includes: sequentially acquiring each frame of the basic image in the video stream; inputting the currently acquired basic image into a pre-trained mask generation model, and acquiring a mask output by the mask generation model and corresponding to the currently acquired basic image; wherein, the mask includes: at least one damaged channel; each damaged channel corresponds to a damage type, and each damaged type corresponds to a damaged area; and the mask generation model is used to: when detecting the damaged area and the normal area of ​​the basic image, identify the damaged type of the damaged area; and identify the damaged area and the normal area corresponding to the damaged type in the damaged channel corresponding to the damaged type.

[0158] In which, according to the mask corresponding to the basic image, the classified denoising process and the classified denoising process are sequentially performed on the basic image, including: in the classified denoising process stage, according to the mask corresponding to the basic image, according to a first preset time step, multiple steps of adding weighted Gaussian noise are added to the damaged area of ​​the basic image and multiple steps of adding Gaussian noise are added to the normal area of ​​the basic image until the basic image meets the preset complete Gaussian noise condition; in the classified denoising process stage, according to the mask corresponding to the basic image, according to a second preset time step, multiple denoising processes are performed on the damaged area and the normal area of ​​the basic image that meet the complete Gaussian noise condition respectively; wherein the number of denoising steps performed in the classified denoising process stage is the same as the number of denoising steps performed in the classified denoising process stage.

[0159] The step-weighted Gaussian noise is added multiple times to the damaged area of ​​the base image, including: the step-weighted Gaussian noise added each time can be calculated using the following formula:

[0160]

[0161] Among them, I t 受损 represents the step-weighted Gaussian noise added for the tth time; I0 受损 represents the preset initial Gaussian noise; w(M) represents the weight function; represents the cumulative Gaussian noise amount of the tth time;

[0162]

[0163] Wherein, w_I, w_G and w_A are all preset weights; f_I(ΔI) represents the pixel intensity difference coefficient between the damaged area and the normal area in the base image; represents the mask gradient coefficient of the damaged area in the basic image; f_A(A(M)) represents the mask area coefficient of the damaged area in the basic image.

[0164] Among them, the completely Gaussian noise condition includes: the number of noise addition steps performed in the classification noise addition processing stage reaches a preset step threshold; or the damaged area and the normal area of ​​the basic image are independent and identically distributed and obey the same standard Gaussian distribution.

[0165] Wherein, in the process of executing the classification denoising processing, the damaged area of ​​the basic image is repaired according to the basic images of the previous and next frames of the basic image, including: in the classification denoising processing stage, in the process of each denoising processing, for each pixel point in the damaged area of ​​the basic image, the local context feature value within a preset range of the corresponding position of the pixel point is determined in the normal area of ​​the basic images of the previous and next frames of the basic image; based on the local context feature value within the preset range of the corresponding position of the pixel point, the repair pixel value corresponding to the pixel point is determined and the original pixel value of the pixel point is replaced by the repair pixel value.

[0166] The local context feature value is a weighted average feature value calculated based on the pixel values ​​of each pixel within a preset range of the corresponding position of the pixel point in the normal area of ​​the previous and next frames of the basic image; the weighted average feature value is calculated using the following formula:

[0167]

[0168] Among them, μ context represents the weighted average eigenvalue; I(x′, y′) represents the pixel value of the pixel point within the preset range of the corresponding position of the pixel point I(x, y) in the normal area of ​​the previous and next frames of the basic image; w(xx′, yy′) represents the spatial weight coefficient of I(x′, y′); ∑ (x′，y′)：M(x′，y′)＝0 Indicates cumulative calculation;

[0169] The repaired pixel value corresponding to the pixel point is determined using the following formula:

[0170] I recovered (x, y) = α·μ context +(1-α)·I(x, y);

[0171] Among them, I recovered (x, y) represents the repaired pixel value corresponding to the pixel point I(x, y); α is the preset fusion coefficient.

[0172] The present application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the image restoration method provided in any of the aforementioned method embodiments. Since the image restoration method has been described in detail above, any details not fully described in this embodiment are referred to the relevant descriptions in the aforementioned embodiments and are not further elaborated here.

[0173] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0174] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, or of course, by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiment.

[0175] It should be understood that the terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "comprise", "include", "contain" and "have" are inclusive and therefore specify the presence of stated features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be performed in the specific order described or illustrated, unless the order of execution is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0176] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

Claims

1. An image restoration method, characterized in that: include: Call the preset vehicle camera to collect video stream; For each frame of a basic image in the video stream, generating a mask corresponding to the basic image; wherein the mask is used to identify a damaged area and a normal area of ​​the basic image; For each frame of a basic image in the video stream, sequentially performing a classification noise addition process and a classification denoising process on the basic image according to a mask corresponding to the basic image, and during the classification denoising process, repairing a damaged area of ​​the basic image according to previous and next basic images of the basic image; wherein, according to the mask corresponding to the base image, sequentially performing classified denoising processing and classified denoising processing on the base image, comprising: in a classified denoising processing stage, according to the mask corresponding to the base image, adding step-weighted Gaussian noise multiple times to the damaged area of ​​the base image and adding step-weighted Gaussian noise multiple times to the normal area of ​​the base image according to a first preset time step, until the base image meets a preset complete Gaussian noise condition; in a classified denoising processing stage, according to the mask corresponding to the base image, performing denoising processing multiple times on the damaged area and the normal area of ​​the base image that meet the complete Gaussian noise condition according to a second preset time step; wherein the number of denoising steps performed in the classified denoising processing stage is the same as the number of denoising steps performed in the classified denoising processing stage; The step-weighted Gaussian noise is added multiple times to the damaged area of ​​the base image, including: the step-weighted Gaussian noise added each time can be calculated using the following formula: Among them, I t 受损 represents the step-weighted Gaussian noise added for the tth time; I0 受损 represents the preset initial Gaussian noise; w(M) represents the weight function; represents the cumulative Gaussian noise amount of the tth time; Wherein, w_I, w_G and w_A are all preset weights; f_I(ΔI) represents the pixel intensity difference coefficient between the damaged area and the normal area in the base image; represents the mask gradient coefficient of the damaged area in the basic image; f_A(A(M)) represents the mask area coefficient of the damaged area in the basic image.

2. The method according to claim 1, characterized in that The step of generating a mask corresponding to each frame of a basic image in the video stream includes: Sequentially acquiring each frame of the basic image in the video stream; Inputting the currently acquired basic image into a pre-trained mask generation model, and obtaining a mask output by the mask generation model corresponding to the currently acquired basic image; Wherein, the mask includes: at least one damaged channel; each damaged channel corresponds to a damage type, and each damaged type corresponds to a damaged area; Furthermore, the mask generation model is used to: identify the damage type of the damaged area when detecting the damaged area and the normal area of ​​the basic image; and identify the damaged area and the normal area corresponding to the damage type in the damaged channel corresponding to the damage type.

3. The method according to claim 1, characterized in that The completely Gaussian noise condition includes: The number of noise addition steps performed in the classification noise addition processing stage reaches a preset step threshold; or, The damaged area and the normal area of ​​the basic image are independent and identically distributed and obey the same standard Gaussian distribution.

4. The method according to claim 1, wherein The method of repairing the damaged area of ​​the basic image according to the previous and next basic images of the basic image during the classification denoising process includes: In the classification denoising stage, during each denoising process, for each pixel in the damaged area of ​​the base image, a local context feature value within a preset range corresponding to the pixel point in the normal areas of the base images of the previous and next frames of the base image is determined; According to the local context feature value within a preset range of the position corresponding to the pixel point, the repair pixel value corresponding to the pixel point is determined and the original pixel value of the pixel point is replaced with the repair pixel value.

5. The method according to claim 4, characterized in that The local context feature value is: a weighted average feature value calculated based on the pixel values ​​of each pixel point within a preset range corresponding to the pixel point in the normal area of ​​the previous and next frames of the basic image; The weighted average eigenvalue is calculated using the following formula: Among them, μ context represents the weighted average eigenvalue; I(x′, y′) represents the pixel value of the pixel point within the preset range of the corresponding position of the pixel point I(x, y) in the normal area of ​​the previous and next frames of the basic image; w(xx′, yy′) represents the spatial weight coefficient of I(x′, y′); ∑ (x′,y′):M(x′,y′)＝0 Indicates cumulative calculation; The repaired pixel value corresponding to the pixel point is determined using the following formula: I recovered (x,y)=a·m context +(1-α)·I(x,y); Among them, I recoeered (x, y) represents the repaired pixel value corresponding to the pixel point I(x, y); α is the preset fusion coefficient.

6. An image restoration device, characterized in that: include: The calling module is used to call the preset vehicle-mounted camera to collect video stream; A generating module, configured to generate a mask corresponding to each frame of a basic image in the video stream; wherein the mask is used to identify a damaged area and a normal area of ​​the basic image; A restoration module is configured to sequentially perform classified denoising and classified denoising on each frame of a basic image in the video stream according to a mask corresponding to the basic image, and, during the classification denoising process, repair a damaged area of ​​the basic image according to previous and next basic images of the basic image. The sequentially performing classified denoising and classified denoising on the basic image according to the mask corresponding to the basic image comprises: in a classification denoising stage, according to the mask corresponding to the basic image, adding step-weighted Gaussian noise multiple times to the damaged area of ​​the basic image and adding step-weighted Gaussian noise multiple times to the normal area of ​​the basic image according to a first preset time step, until the basic image meets a preset perfect Gaussian noise condition; in a classification denoising stage, according to the mask corresponding to the basic image, performing denoising on the damaged area and the normal area of ​​the basic image that meet the perfect Gaussian noise condition multiple times according to a second preset time step; wherein the number of denoising steps performed in the classification denoising stage is the same as the number of denoising steps performed in the classification denoising stage; The step-weighted Gaussian noise is added multiple times to the damaged area of ​​the base image, including: the step-weighted Gaussian noise added each time can be calculated using the following formula: Among them, I t 受损 represents the step-weighted Gaussian noise added for the tth time; I0 受损 represents the preset initial Gaussian noise; w(M) represents the weight function; represents the cumulative Gaussian noise amount of the tth time; Wherein, w_I, w_G and w_A are all preset weights; f_I(ΔI) represents the pixel intensity difference coefficient between the damaged area and the normal area in the base image; represents the mask gradient coefficient of the damaged area in the basic image; f_A(A(M)) represents the mask area coefficient of the damaged area in the basic image.

7. An image restoration device, characterized in that: include: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor coupled to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to: execute an image restoration program stored in the memory to implement the image restoration method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed to implement the image restoration method according to any one of claims 1 to 5.

Images