Image snow removal method and system based on zero sample learning

Abstract

The invention relates to the technical field of deep learning, and particularly discloses an image snow removal method and system based on zero sample learning, and the method comprises the steps: training a preset zero sample learning model for image snow removal based on a plurality of original sample snow images in combination with a snow image model, and obtaining a target zero sample learning model; and inputting a to-be-processed snow image into the target zero sample learning model to obtain a target mask image and a target snowflake foreground image corresponding to the to-be-processed snow image, and inputting the target mask image and the target snowflake foreground image into the snow image model to obtain a target snow removal image corresponding to the to-be-processed snow image. According to the method, image snow removal processing is carried out by constructing the lightweight zero sample learning model, and the video image snow removal effect and processing efficiency are improved while the problems that a data set is difficult to obtain and threshold deviation is generated by a synthetic data set training network are avoided.

Description

Background Art

[0002] In current deep learning-based video image snow removal networks, they are all end-to-end deep networks. By training the network with a large number of synthetic images, the designed network can achieve good results when processing synthetic images, but it is difficult to obtain good results for images captured in real snow weather. In addition, although the current convolutional neural network method can adaptively extract features during the snow removal process and also reduces the requirements for estimating the scene, snow grain size, and snow density in the image, the current deep learning-based method has a cumbersome network design and a large number of parameters, and there is still room for improvement in processing speed.

[0003] Therefore, there is an urgent need to provide a technical solution to solve the above problems. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides an image snow removal method and system based on zero-shot learning.

[0005] In a first aspect, the present invention provides an image snow removal method based on zero-shot learning. The technical solution of this method is as follows:

[0006] S1. Based on multiple original sample snow images and in combination with a snow image model, train a preset zero-shot learning model for image snow removal to obtain a target zero-shot learning model; wherein, the preset zero-shot learning model includes: a mask estimation network, a snowflake foreground estimation network, and a clear image estimation network; the snow image model is used to characterize the correlation between the mask image extracted by the mask estimation network, the snowflake foreground image extracted by the snowflake foreground estimation network, the clear image extracted by the clear image estimation network, and the snow image;

[0007] S2. Input the snow image to be processed into the target zero-shot learning model, obtain the target mask image and the target snowflake foreground image corresponding to the snow image to be processed and input them into the snow image model to obtain the target snow removal image corresponding to the snow image to be processed.

[0008] The beneficial effects of an image snow removal method based on zero-shot learning of the present invention are as follows:

[0009] The method of the present invention performs image snow removal processing by constructing a lightweight zero-shot learning model, which avoids the difficulty of obtaining a dataset and the threshold deviation problem caused by training the network with a synthetic dataset, and at the same time improves the video image snow removal effect and processing efficiency.

[0010] On the basis of the above solution, an image snow removal method based on zero-shot learning of the present invention can also be improved as follows.

[0011] In an optional manner, S1 includes:

[0012] S11. Input any original snow image into the preset zero - shot learning model to obtain the first sample mask image, the first sample snow foreground image, and the first sample clear image corresponding to the any original snow image, and input them into the snow image model to obtain the reconstructed sample snow image corresponding to the any original snow image;

[0013] S12. Input the reconstructed sample snow image corresponding to the any original snow image into the preset zero - shot learning model to obtain the second sample mask image, the second sample snow foreground image, and the second sample clear image corresponding to the any original snow image;

[0014] S13. According to the first sample mask image, the first sample snow foreground image, the first sample clear image, the second sample mask image, the second sample snow foreground image, and the second sample clear image corresponding to the any original snow image, and in combination with the objective loss function of the preset zero - shot learning model, obtain the loss value corresponding to the any original snow image until the loss values corresponding to each original snow image are obtained;

[0015] S14. Use all the loss values to optimize the network parameters of the preset zero - shot learning model to obtain an optimized zero - shot learning model. Take the optimized zero - shot learning model as the preset zero - shot learning model, and return to execute S11 until when the optimized zero - shot learning model meets the preset iterative training condition, determine the optimized zero - shot learning model as the target zero - shot learning model.

[0016] In an optional manner, S11 includes:

[0017] S111. Input the any original snow image into the mask estimation network, the snow foreground estimation network, and the clear image estimation network respectively to obtain the first sample mask image, the first sample snow foreground image, and the first sample clear image corresponding to the any original snow image;

[0018] S112. Input the first sample mask image, the first sample snow foreground image, and the first sample clear image corresponding to the any original snow image into the snow image model to obtain the reconstructed sample snow image corresponding to the any original snow image; where the snow image model is: I 2 (x) = Z 1 (x)×S 1 (x)+(1 - Z 1 (x))×J 1 (x); Z 1 (x) is the first sample mask image, S 1(x) is the first sample snowflake foreground image, J 1 (x) is the first sample clear image, I 2 (x) is the reconstructed sample snow image.

[0019] In an alternative manner, according to the first sample mask image, the first sample snowflake foreground image, the first sample clear image, the second sample mask image, the second sample snowflake foreground image, and the second sample clear image corresponding to any one of the original sample snow images, and in combination with the target loss function of the preset zero-shot learning model, the step of obtaining the loss value corresponding to any one of the original sample snow images includes:

[0020] Input the first sample mask image, the first sample snowflake foreground image, the first sample clear image, the second sample mask image, the second sample snowflake foreground image, and the second sample clear image corresponding to any one of the original sample snow images into the target loss function to obtain the loss value corresponding to any one of the original sample snow images; wherein, the target loss function is: is the loss value, is the image similarity loss, is the snowflake foreground image loss, is the image saturation penalty loss, is the mask image loss, ω 1 、ω 2 、ω 3 and ω 4 are loss coefficients.

[0021] In an alternative manner, the mask image is: an image after secondary masking.

[0022] In a second aspect, the present invention provides an image desnowing system based on zero-shot learning, and the technical solution of the system is as follows:

[0023] Including: a training module and a running module;

[0024] The training module is used for: training a preset zero-shot learning model for image desnowing based on a plurality of original sample snow images and in combination with a snow image model to obtain a target zero-shot learning model; wherein, the preset zero-shot learning model includes: a mask estimation network, a snowflake foreground estimation network, and a clear image estimation network; the snow image model is used to represent the correlation between the mask image extracted by the mask estimation network, the snowflake foreground image extracted by the snowflake foreground estimation network, the clear image extracted by the clear image estimation network, and the snow image;

[0025] The running module is used to: input the snow image to be processed into the target zero-shot learning model, obtain the target mask image and the target snow foreground image corresponding to the snow image to be processed, and input them into the snow image model to obtain the target snow-removed image corresponding to the snow image to be processed.

[0026] The beneficial effects of an image snow-removing system based on zero-shot learning of the present invention are as follows:

[0027] The system of the present invention performs image snow-removing processing by constructing a lightweight zero-shot learning model, which not only avoids the difficulty of obtaining a data set and the problem of threshold deviation caused by training a network with a synthetic data set, but also improves the snow-removing effect and processing efficiency of video images.

[0028] On the basis of the above solution, an image snow-removing system based on zero-shot learning of the present invention can also be improved as follows.

[0029] In an optional manner, the training module includes: a first training module, a second training module, a third training module, and an iterative training module;

[0030] The first training module is used to: input any original sample snow image into the preset zero-shot learning model, obtain the first sample mask image, the first sample snow foreground image, and the first sample clear image corresponding to the any original sample snow image, and input them into the snow image model to obtain the reconstructed sample snow image corresponding to the any original sample snow image;

[0031] The second training module is used to: input the reconstructed sample snow image corresponding to the any original sample snow image into the preset zero-shot learning model, obtain the second sample mask image, the second sample snow foreground image, and the second sample clear image corresponding to the any original sample snow image;

[0032] The third training module is used to: according to the first sample mask image, the first sample snow foreground image, the first sample clear image, the second sample mask image, the second sample snow foreground image, and the second sample clear image corresponding to the any original sample snow image, and in combination with the target loss function of the preset zero-shot learning model, obtain the loss value corresponding to the any original sample snow image until the loss values corresponding to each original sample snow image are obtained;

[0033] The iterative training module is used to: optimize the network parameters of the preset zero-shot learning model by using all the loss values, obtain an optimized zero-shot learning model, use the optimized zero-shot learning model as the preset zero-shot learning model, and return to call the first training module. Until the optimized zero-shot learning model meets the preset iterative training conditions, the optimized zero-shot learning model is determined as the target zero-shot learning model.

[0034] In an alternative embodiment, the first training module is specifically configured to:

[0035] Input any of the original snow images into the mask estimation network, the snow foreground estimation network, and the clear image estimation network respectively, to obtain a first sample mask image, a first sample snow foreground image, and a first sample clear image corresponding to any of the original snow images;

[0036] Input the first sample mask image, the first sample snow foreground image, and the first sample clear image corresponding to any of the original snow images into the snow image model to obtain a reconstructed sample snow image corresponding to any of the original snow images; where the snow image model is: I 2 (x) = Z 1 (x) × S 1 (x) + (1 - Z 1 (x)) × J 1 (x); Z 1 (x) is the first sample mask image, S 1 (x) is the first sample snow foreground image, J 1 (x) is the first sample clear image, I 2 (x) is the reconstructed sample snow image.

[0037] In a third aspect, a technical solution of an electronic device according to the present invention is as follows:

[0038] It includes a memory, a processor, and a program stored on the memory and running on the processor. When the processor executes the program, it implements the steps of the zero-shot learning-based image snow removal method of the present invention.

[0039] In a fourth aspect, a technical solution of a computer-readable storage medium provided by the present invention is as follows:

[0040] Instructions are stored in the computer-readable storage medium. When the computer-readable storage medium reads the instructions, the computer-readable storage medium executes the steps of the zero-shot learning-based image snow removal method of the present invention.

[0041] The above description is only an overview of the technical solution of the present invention. In order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the specification. And in order to make the above and other objects, features and advantages of the present invention more obvious and understandable, the specific embodiments of the present invention are specifically exemplified below. Brief Description of the Drawings

[0042] The drawings are only used to illustrate the embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0043] Figure 1 is a schematic flowchart of an embodiment of an image snow removal method based on zero-shot learning of the present invention;

[0044] Figure 2 is a schematic structural diagram of a preset zero-shot learning model;

[0045] Figure 3 is a schematic structural diagram of the overall network;

[0046] Figure 4 is a schematic diagram of the training principle;

[0047] Figure 5 is a schematic diagram of the principle of the snow image model;

[0048] Figure 6 is a schematic structural diagram of an embodiment of an image snow removal system based on zero-shot learning of the present invention;

[0049] Figure 7 is a schematic structural diagram of an embodiment of an electronic device of the present invention. Detailed Description of the Embodiments

[0050] Hereinafter, the exemplary embodiments of the present invention will be described in more detail with reference to the drawings. Although the exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein.

[0051] Figure 1FIG. 0 shows a schematic flowchart of an embodiment of an image snow removal method based on zero - shot learning provided by the present invention. The image snow removal method based on zero - shot learning can be executed by an electronic device such as a terminal device or a server. Among them, the terminal device can be any fixed or mobile terminal such as a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle - mounted device, a wearable device, etc. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can implement the image snow removal method based on zero - shot learning by a processor calling computer - readable instructions stored in a memory. As Figure 1 shown, the method includes the following steps:

[0052] S1. Based on a plurality of original sample snow images and in combination with a snow image model, train a preset zero - shot learning model for image snow removal to obtain a target zero - shot learning model.

[0053] Among them, as Figure 2 shown, the preset zero - shot learning model includes: a mask estimation network, a snowflake foreground estimation network, and a clear image estimation network. The snow image model is used to characterize the correlation relationship between the mask image extracted by the mask estimation network, the snowflake foreground image extracted by the snowflake foreground estimation network, the clear image extracted by the clear image estimation network, and the snow image. The mask image in this embodiment is an image after secondary masking.

[0054] Among them, the original sample snow image is a randomly selected snow image for model training, and the snow image refers to an image containing snowflakes and / or snow scenes. The target zero - shot learning model is a trained zero - shot learning model.

[0055] It should be noted that zero - shot learning means that the model can directly predict or classify new and unseen categories without having seen any labeled examples of the target categories.

[0056] S2. Input the snow image to be processed into the target zero - shot learning model to obtain a target mask image and a target snowflake foreground image corresponding to the snow image to be processed, and input them into the snow image model to obtain a target snow - removed image corresponding to the snow image to be processed.

[0057] Among them, the snow image to be processed is the snow image that needs to be de - snowed in this embodiment. The target de - snowed image is the clear image obtained after de - snowing the snow image to be processed. The target mask image is the mask image obtained after processing the image to be processed by the target zero - shot learning model. The target snowflake foreground image is the snowflake foreground image obtained after processing the image to be processed by the target zero - shot learning model.

[0058] In an alternative way, S1 includes:

[0059] S11. Input any original sample snow image into the preset zero - shot learning model to obtain the first sample mask image, the first sample snowflake foreground image and the first sample clear image corresponding to the any original sample snow image, and input them into the snow image model to obtain the reconstructed sample snow image corresponding to the any original sample snow image.

[0060] Figure 3 Shows the overall network structure diagram of this embodiment. As Figure 3 shown: ① The clear image estimation network (J - Net) includes eleven convolutional layers connected in sequence. Among them, the first ten convolutional layers are all 3×3 convolutional layers, and the last convolutional layer is a 1×1 convolutional layer. The third convolutional layer is also connected to the eighth convolutional layer, the fourth convolutional layer is also connected to the seventh convolutional layer, and the fifth convolutional layer is also connected to the sixth convolutional layer. The first convolutional layer receives the original sample image, and the eleventh convolutional layer outputs the sample clear image. ② The snowflake foreground estimation network (S - Net) includes eleven convolutional layers connected in sequence. Among them, the first ten convolutional layers are all 3×3 convolutional layers, and the last convolutional layer is a 1×1 convolutional layer. The third convolutional layer is also connected to the eighth convolutional layer, the fourth convolutional layer is also connected to the seventh convolutional layer, and the fifth convolutional layer is also connected to the sixth convolutional layer. The first convolutional layer receives the original sample image, and the eleventh convolutional layer outputs the sample snowflake foreground image. ③ The mask estimation network (z - Net) includes the first four convolutional layers, a feature representation layer, and the last four convolutional layers connected in sequence. Among them, the first four convolutional layers and the first three of the last four convolutional layers are all 5×5 convolutional layers, and the last convolutional layer is a 1×1 convolutional layer. The feature representation layer first extracts useful features from the original data. These features can be numerical values, text, images, or other forms of data, depending on the type of data and the requirements of the task. The process of feature extraction usually involves various algorithms and techniques, such as principal component analysis (PCA), factor analysis (FA), convolutional neural network (CNN), etc. These algorithms can identify and extract the most important features from the original data.

[0061] It should be noted that the mask estimation network, the snow foreground estimation network, and the clear image estimation network are not completely isolated from each other, and some of the feature data in the middle is shared. The specific structures of the mask estimation network, the snow foreground estimation network, and the clear image estimation network can also be adjusted according to the actual situation, and no restrictions are set here.

[0062] S12. Input the reconstructed sample snow image corresponding to any one of the original sample snow images into the preset zero-shot learning model to obtain a second sample mask image, a second sample snow foreground image, and a second sample clear image corresponding to any one of the original sample snow images.

[0063] Among them, the first sample mask image, the first sample snow foreground image, and the first sample clear image are images generated based on the sample snow image before reconstruction, and the second sample mask image, the second sample snow foreground image, and the second sample clear image are images generated based on the sample snow image after reconstruction. The generation methods before and after reconstruction are the same, and will not be elaborated here.

[0064] S13. According to the first sample mask image, the first sample snow foreground image, the first sample clear image, the second sample mask image, the second sample snow foreground image, and the second sample clear image corresponding to any one of the original sample snow images, and in combination with the target loss function of the preset zero-shot learning model, obtain the loss value corresponding to any one of the original sample snow images until the loss values corresponding to each original sample snow image are obtained.

[0065] S14. Use all the loss values to optimize the network parameters of the preset zero-shot learning model to obtain an optimized zero-shot learning model. Take the optimized zero-shot learning model as the preset zero-shot learning model, and return to execute S11 until the optimized zero-shot learning model meets the preset iterative training conditions, and then determine the optimized zero-shot learning model as the target zero-shot learning model.

[0066] Among them, the preset iterative training conditions are default set to: reaching the maximum number of iterations or loss convergence, and can also be set according to the actual situation, and no restrictions are set here.

[0067] In an alternative way, S11 includes:

[0068] S111. Input any one of the original sample snow images into the mask estimation network, the snow foreground estimation network, and the clear image estimation network respectively to obtain a first sample mask image, a first sample snow foreground image, and a first sample clear image corresponding to any one of the original sample snow images.

[0069] S112. Input the first sample mask image, the first sample snowflake foreground image, and the first sample clear image corresponding to any one of the original sample snow images into the snow image model to obtain the reconstructed sample snow image corresponding to any one of the original sample snow images.

[0070] Among them, the snow image model is: I 2 (x) = Z 1 (x) × S 1 (x) + (1 - Z 1 (x)) × J 1 (x); Z 1 (x) is the first sample mask image, S 1 (x) is the first sample snowflake foreground image, J 1 (x) is the first sample clear image, I 2 (x) is the reconstructed sample snow image.

[0071] In an optional manner, the step of obtaining the loss value corresponding to any one of the original sample snow images according to the first sample mask image, the first sample snowflake foreground image, the first sample clear image, the second sample mask image, the second sample snowflake foreground image, and the second sample clear image corresponding to any one of the original sample snow images, and in combination with the target loss function of the preset zero-shot learning model, includes:

[0072] Input the first sample mask image, the first sample snowflake foreground image, the first sample clear image, the second sample mask image, the second sample snowflake foreground image, and the second sample clear image corresponding to any one of the original sample snow images into the target loss function to obtain the loss value corresponding to any one of the original sample snow images.

[0073] Among them, the target loss function is: is the loss value, is the image similarity loss, is the snowflake foreground image loss, is the image saturation penalty loss, is the mask image loss, ω 1 、ω 2 、ω 3 and ω 4 are loss coefficients.

[0074] It should be noted that, as Figure 4 shows, specifically represents the original sample snow image I 1 (x) and the reconstructed sample snow image I 2 (x), as well as the first sample clear image J 1 (x) and the second sample clear image J(x) 2(x) similarity loss. Specifically represents the first sample snow foreground image S 1 (x) and the second sample snow foreground image S 2 (x) loss, Specifically represents the first sample mask image Z 1 (x) and the second sample mask image Z 2 (x) loss. Specifically represents the original sample snow image I 1 (x) and the reconstructed sample snow image I 2 (x) loss for the difference in image saturation. Such as Figure 5 As shown, the snow image model can also be deformed into: In the inference (detection) stage, the required clear image can be restored according to the mask image, snow foreground image and snow image.

[0075] The technical solution of this embodiment performs image desnowing by constructing a lightweight zero-shot learning model, which avoids the difficulty of obtaining a dataset and the threshold bias problem caused by training the network with a synthetic dataset, and at the same time improves the video image desnowing effect and processing efficiency.

[0076] Figure 6 Shows a schematic structural diagram of an embodiment of an image desnowing system 200 based on zero-shot learning provided by the present invention. Such as Figure 6 As shown, the system 200 includes: a training module 210 and a running module 220;

[0077] The training module 210 is used for: based on a plurality of original sample snow images and in combination with a snow image model, training a preset zero-shot learning model for image desnowing to obtain a target zero-shot learning model; wherein, the preset zero-shot learning model includes: a mask estimation network, a snow foreground estimation network and a clear image estimation network; the snow image model is used to characterize the association relationship between the mask image extracted by the mask estimation network, the snow foreground image extracted by the snow foreground estimation network, the clear image extracted by the clear image estimation network and the snow image;

[0078] The running module 220 is used for: inputting the snow image to be processed into the target zero-shot learning model, obtaining the target mask image and target snow foreground image corresponding to the snow image to be processed and inputting them into the snow image model, and obtaining the target desnowed image corresponding to the snow image to be processed.

[0079] In an optional manner, the training module 210 includes: a first training module, a second training module, a third training module and an iterative training module;

[0080] The first training module is used to: input any original snow image into the preset zero-shot learning model, obtain the first sample mask image, the first sample snow foreground image, and the first sample clear image corresponding to the any original snow image, and input them into the snow image model to obtain the reconstructed sample snow image corresponding to the any original snow image;

[0081] The second training module is used to: input the reconstructed sample snow image corresponding to the any original snow image into the preset zero-shot learning model, obtain the second sample mask image, the second sample snow foreground image, and the second sample clear image corresponding to the any original snow image;

[0082] The third training module is used to: according to the first sample mask image, the first sample snow foreground image, the first sample clear image, the second sample mask image, the second sample snow foreground image, and the second sample clear image corresponding to the any original snow image, and in combination with the target loss function of the preset zero-shot learning model, obtain the loss value corresponding to the any original snow image until the loss values corresponding to each original snow image are obtained;

[0083] The iterative training module is used to: use all the loss values to optimize the network parameters of the preset zero-shot learning model to obtain an optimized zero-shot learning model, use the optimized zero-shot learning model as the preset zero-shot learning model, and return to call the first training module until the optimized zero-shot learning model meets the preset iterative training conditions, and determine the optimized zero-shot learning model as the target zero-shot learning model.

[0084] In an optional manner, the first training module is specifically used to:

[0085] Input the any original snow image into the mask estimation network, the snow foreground estimation network, and the clear image estimation network respectively to obtain the first sample mask image, the first sample snow foreground image, and the first sample clear image corresponding to the any original snow image;

[0086] Input the first sample mask image, the first sample snow foreground image, and the first sample clear image corresponding to the any original snow image into the snow image model to obtain the reconstructed sample snow image corresponding to the any original snow image; where the snow image model is: I 2 (x) = Z 1 (x) × S 1 (x) + (1 - Z 1 (x)) × J 1 (x); Z 1 (x) is the first sample mask image, S1 (x) is the first sample snowflake foreground image, J 1 (x) is the first sample clear image, I 2 (x) is the reconstructed sample snow image.

[0087] In an alternative manner, the third training module is specifically configured to:

[0088] Input the first sample mask image, the first sample snowflake foreground image, the first sample clear image, the second sample mask image, the second sample snowflake foreground image, and the second sample clear image corresponding to any one of the original sample snow images into the target loss function to obtain the loss value corresponding to any one of the original sample snow images; wherein, the target loss function is: is the loss value, is the image similarity loss, is the snowflake foreground image loss, is the image saturation penalty loss, is the mask image loss, ω 1 、ω 2 、ω 3 and ω 4 are loss coefficients.

[0089] In an alternative manner, the mask image is: the image after secondary masking.

[0090] It should be noted that the beneficial effects of the image deraining system based on zero-shot learning provided in the above embodiments are the same as those of the image deraining method based on zero-shot learning, and will not be elaborated here. In addition, when the system provided in the above embodiments implements its functions, only the above-mentioned division of each functional module is used as an example for illustration. In practical applications, the above functions can be allocated to different functional modules according to needs, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be seen in the method embodiments, which will not be elaborated here.

[0091] Among them, the image deraining system based on zero-shot learning of the present invention can be a computer program (including program code) running in a computer device. For example, the image deraining system based on zero-shot learning of the present invention is an application software, which can be used to execute the corresponding steps in the image deraining method based on zero-shot learning of the present invention.

[0092] In some embodiments, the zero-shot learning-based image de-snowing system of the present invention can be implemented in a combination of software and hardware. As an example, the zero-shot learning-based image de-snowing system of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the zero-shot learning-based image de-snowing method of the present invention. For example, a processor in the form of a hardware decoding processor can employ one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0093] Among them, the modules involved in the embodiments of the present invention can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation to the module itself in some cases.

[0094] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned zero-shot learning-based image de-snowing method is implemented. That is to say, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the zero-shot learning-based image de-snowing method shown in any embodiment of the present invention by calling the computer program.

[0095] In an alternative embodiment, an electronic device is provided, as Figure 7 shown, Figure 7 the electronic device 4000 shown includes: a processor 4001 and a memory 4003. Among them, the processor 4001 and the memory 4003 are connected, such as connected through a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, and the transceiver 4004 can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data, etc. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation to the embodiments of the present invention.

[0096] The processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logical blocks, modules, and circuits described in connection with the disclosure of the present invention. The processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0097] The bus 4002 may include a path for transferring information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 7 only a thick line is used to represent the bus 4002 in the figure, but it does not mean that there is only one bus or one type of bus.

[0098] The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, or it may also be an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0099] The memory 4003 is used to store the application program code (computer program) for implementing the solution of the present invention, and is controlled by the processor 4001 for execution. The processor 4001 is used to execute the application program code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0100] Among them, the electronic device may also be a terminal device. The terminal device may be any terminal device that can install an application and access a web page through the application, including at least one of a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart TV, and a smart vehicle-mounted device.

[0101] It should be noted that Figure 7 the electronic device shown is only an example, and should not impose any limitation on the functions and usage scope of the embodiments of the present invention.

[0102] A computer-readable storage medium according to an embodiment of the present invention has a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned image snow removal method based on zero-shot learning is implemented.

[0103] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0104] In an exemplary embodiment, there is also provided a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the above-mentioned image snow removal method based on zero-shot learning.

[0105] Computer program code for performing the operations of the present invention may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0106] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0107] The computer-readable storage medium provided by the embodiments of the present invention may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0108] The above computer-readable storage medium stores one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0109] The above description is only a preferred embodiment of the present invention and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of disclosure involved in the present invention is not limited to the technical solutions formed by the specific combination of the above technical features, but also covers other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) having similar functions disclosed in the present invention.

[0110] It should be noted that the terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and represent a limitation on a specific order or sequence. In appropriate cases, the order of use of similar objects can be interchanged so that the embodiments of the present application described here can be implemented in an order other than the illustrated or described order.

[0111] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms: it can be completely hardware, or completely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to as "circuit", "module" or "system" in this article. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, which contains computer-readable program code.

[0112] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for removing snow from an image based on zero-shot learning, characterized in that: include: S1. Based on multiple original sample snow images and in combination with a snow image model, a preset zero-sample learning model for image snow removal is trained to obtain a target zero-sample learning model; wherein the preset zero-sample learning model includes: a mask estimation network, a snowflake foreground estimation network and a clear image estimation network; the snow image model is used to characterize the association between the mask image extracted by the mask estimation network, the snowflake foreground image extracted by the snowflake foreground estimation network, the clear image extracted by the clear image estimation network and the snow image; S2. Input the snow image to be processed into the target zero-sample learning model, obtain the target mask image and target snowflake foreground image corresponding to the snow image to be processed, and input them into the snow image model to obtain the target snow-removed image corresponding to the snow image to be processed.

2. The image snow removal method based on zero-shot learning according to claim 1, characterized in that: S1 includes: S11, inputting any original sample snow image into the preset zero-sample learning model, obtaining a first sample mask image, a first sample snowflake foreground image and a first sample clear image corresponding to any original sample snow image, and inputting them into the snow image model, to obtain a reconstructed sample snow image corresponding to any original sample snow image; S12, inputting the reconstructed sample snow image corresponding to any one of the original sample snow images into the preset zero-sample learning model to obtain a second sample mask image, a second sample snowflake foreground image and a second sample clear image corresponding to any one of the original sample snow images; S13, according to the first sample mask image, the first sample snowflake foreground image, the first sample clear image, the second sample mask image, the second sample snowflake foreground image and the second sample clear image corresponding to any of the original sample snow images, and in combination with the target loss function of the preset zero-sample learning model, obtaining the loss value corresponding to any of the original sample snow images, until the loss value corresponding to each original sample snow image is obtained; S14. Utilize all loss values ​​to optimize the network parameters of the preset zero-sample learning model to obtain an optimized zero-sample learning model, use the optimized zero-sample learning model as the preset zero-sample learning model, and return to execute S11 until the optimized zero-sample learning model meets the preset iterative training conditions, and then determine the optimized zero-sample learning model as the target zero-sample learning model.

3. The image snow removal method based on zero-shot learning according to claim 2 is characterized in that: S11 includes: S111, inputting any one of the original sample snow images into the mask estimation network, the snowflake foreground estimation network and the clear image estimation network respectively, to obtain a first sample mask image, a first sample snowflake foreground image and a first sample clear image corresponding to any one of the original sample snow images; S112. Input the first sample mask image, the first sample snowflake foreground image and the first sample clear image corresponding to any of the original sample snow images into the snow image model to obtain a reconstructed sample snow image corresponding to any of the original sample snow images; wherein the snow image model is: I2(x)=Z1(x)×S1(x)+(1-Z1(x))×J1(x); Z1(x) is the first sample mask image, S1(x) is the first sample snowflake foreground image, J1(x) is the first sample clear image, and I2(x) is the reconstructed sample snow image.

4. The image snow removal method based on zero-shot learning according to claim 2, characterized in that: The step of obtaining a loss value corresponding to any original sample snow image according to the first sample mask image, the first sample snowflake foreground image, the first sample clear image, the second sample mask image, the second sample snowflake foreground image and the second sample clear image corresponding to any original sample snow image, and in combination with the target loss function of the preset zero-sample learning model, comprises: The first sample mask image, the first sample snowflake foreground image, the first sample clear image, the second sample mask image, the second sample snowflake foreground image and the second sample clear image corresponding to any of the original sample snow images are input into the target loss function to obtain the loss value corresponding to any of the original sample snow images; wherein the target loss function is: is the loss value, is the image similarity loss, is the snow foreground image loss, Penalize loss for image saturation, is the mask image loss, ω1, ω2, ω3 and ω4 are the loss coefficients.

5. The image snow removal method based on zero-shot learning according to any one of claims 1 to 4, characterized in that: The mask image is: an image after secondary masking.

6. An image snow removal system based on zero-shot learning, characterized in that: include: Training module and running module; The training module is used to: train a preset zero-sample learning model for image snow removal based on multiple original sample snow images and in combination with a snow image model to obtain a target zero-sample learning model; wherein the preset zero-sample learning model includes: a mask estimation network, a snowflake foreground estimation network and a clear image estimation network; the snow image model is used to characterize the association between the mask image extracted by the mask estimation network, the snowflake foreground image extracted by the snowflake foreground estimation network, the clear image extracted by the clear image estimation network and the snow image; The operation module is used to: input the snow image to be processed into the target zero-sample learning model, obtain the target mask image and target snowflake foreground image corresponding to the snow image to be processed and input them into the snow image model to obtain the target snow-removed image corresponding to the snow image to be processed.

7. The image snow removal system based on zero-shot learning according to claim 6, characterized in that: The training modules include: a first training module, a second training module, a third training module and an iterative training module; The first training module is used to: input any original sample snow image into the preset zero-sample learning model, obtain a first sample mask image, a first sample snowflake foreground image and a first sample clear image corresponding to the any original sample snow image, and input them into the snow image model to obtain a reconstructed sample snow image corresponding to the any original sample snow image; The second training module is used to: input the reconstructed sample snow image corresponding to any of the original sample snow images into the preset zero-sample learning model to obtain a second sample mask image, a second sample snowflake foreground image and a second sample clear image corresponding to any of the original sample snow images; The third training module is used to obtain the loss value corresponding to any original sample snow image according to the first sample mask image, the first sample snowflake foreground image, the first sample clear image, the second sample mask image, the second sample snowflake foreground image and the second sample clear image corresponding to any original sample snow image, and in combination with the target loss function of the preset zero-sample learning model, until the loss value corresponding to each original sample snow image is obtained; The iterative training module is used to: use all loss values ​​to optimize the network parameters of the preset zero-sample learning model to obtain an optimized zero-sample learning model, use the optimized zero-sample learning model as the preset zero-sample learning model, and return to call the first training module until the optimized zero-sample learning model meets the preset iterative training conditions, and then determine the optimized zero-sample learning model as the target zero-sample learning model.

8. The image snow removal system based on zero-shot learning according to claim 7, characterized in that: The first training module is specifically used for: Inputting any one of the original sample snow images into the mask estimation network, the snowflake foreground estimation network and the clear image estimation network respectively, to obtain a first sample mask image, a first sample snowflake foreground image and a first sample clear image corresponding to any one of the original sample snow images; The first sample mask image, the first sample snowflake foreground image and the first sample clear image corresponding to any of the original sample snow images are input into the snow image model to obtain a reconstructed sample snow image corresponding to any of the original sample snow images; wherein the snow image model is: I2(x)=Z1(x)×S1(x)+(1-Z1(x))×J1(x); Z1(x) is the first sample mask image, S1(x) is the first sample snowflake foreground image, J1(x) is the first sample clear image, and I2(x) is the reconstructed sample snow image.

9. An electronic device, characterized in that: The electronic device includes a processor, the processor is coupled to a memory, at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the image snow removal method based on zero-sample learning as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the computer-readable storage medium implements the image snow removal method based on zero-sample learning as described in any one of claims 1 to 5.

Images