Image processing, weather image inpainting and image data processing method

Abstract

Embodiments of the present specification provide image processing, weather image restoration and image data processing methods, wherein the image processing method comprises: obtaining a to-be-processed image, inputting the to-be-processed image into an image processing model, and obtaining a target image, wherein the image processing model is obtained by training a sample image set, the sample image set is obtained by performing image mapping on at least one initial image in an initial image set by using a preset mapping relationship, the preset mapping relationship is a mapping relationship between a historical source image and a historical target image, and the historical source image and the historical target image are used to pre-train the image processing model. Based on the mapping relationship between the historical source image and the historical target image determined in advance, the characteristics of the historical image are restored, a large number of historical images are avoided to be stored, a sample image set containing current knowledge and historical knowledge is obtained, the image processing model is trained, the comprehensiveness and accuracy of image processing are ensured, the storage cost is reduced, and the processing efficiency is improved.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to an image processing method. Background Technology

[0002] Image processing models are primarily trained using sample image sets. These models continuously learn and update their functionality through iterative training; however, new training often leads to the forgetting of historical knowledge. Therefore, by sampling historical images and adding them to the current training image set, the sample image set is reconstructed, enabling more comprehensive and accurate model training. This allows the trained image processing model to learn current knowledge while retaining historical knowledge, resulting in a more comprehensive and accurate image processing model.

[0003] However, sampling historical images and reconstructing the current training sample image set requires pre-storing a large number of historical images. Given the huge number of training samples for image processing models, pre-storing and always retaining a large number of historical images increases data storage costs and reduces processing efficiency. Therefore, there is an urgent need for a comprehensive, accurate, low-storage-cost, and efficient image processing method. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide an image processing method. One or more embodiments of this specification also relate to another image processing method, a weather image restoration method, an image data processing method, an image processing apparatus, another image processing apparatus, a weather image restoration apparatus, an image data processing apparatus, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, an image processing method is provided, comprising:

[0006] Obtain the image to be processed;

[0007] The image to be processed is input into the image processing model to obtain the target image. The image processing model is trained using a sample image set. The sample image set is obtained by mapping at least one initial image in the initial image set using a preset mapping relationship. The preset mapping relationship is the mapping relationship between historical source images and historical target images. The historical source images and historical target images are used to train the image processing model in advance.

[0008] According to a second aspect of the embodiments of this specification, an image processing method is provided, applied to a server, comprising:

[0009] Receive an image processing request sent by the client, wherein the image processing request carries the image to be processed;

[0010] The image to be processed is input into the image processing model to obtain the target image. The image processing model is trained using a sample image set. The sample image set is obtained by mapping at least one initial image in the initial image set using a preset mapping relationship. The preset mapping relationship is the mapping relationship between historical source images and historical target images. The historical source images and historical target images are used to train the image processing model in advance.

[0011] The target image is fed back to the client so that the client can render the target image.

[0012] According to a third aspect of the embodiments of this specification, a weather image restoration method is provided, comprising:

[0013] Acquire the weather image to be repaired;

[0014] The weather image to be repaired is input into the image repair model to obtain the target weather image. The image repair model is trained using a sample weather image set. The sample weather image set is obtained by mapping at least one initial sample weather image in the initial weather image set using a preset mapping relationship. The preset mapping relationship is the mapping relationship between historical source weather images and historical target weather images. The historical source weather images and historical target weather images are used to train the image repair model in advance.

[0015] According to a fourth aspect of the embodiments of this specification, an image data processing method is provided, applied to a cloud-side device, comprising:

[0016] Obtain an initial image set, which includes multiple initial images;

[0017] Using a preset mapping relationship, at least one initial image in the initial image set is mapped to obtain a sample image set, wherein the initial image is either an initial source image or an initial target image;

[0018] The image processing model is trained using a sample image set to obtain the trained image processing model.

[0019] The trained image processing model parameters are sent to the edge device.

[0020] According to a fifth aspect of the embodiments of this specification, an image processing apparatus is provided, comprising:

[0021] The first acquisition module is configured to acquire the image to be processed.

[0022] The first processing module is configured to input the image to be processed into the image processing model to obtain the target image. The image processing model is trained using a sample image set. The sample image set is obtained by mapping at least one initial image in the initial image set using a preset mapping relationship. The preset mapping relationship is the mapping relationship between historical source images and historical target images. The historical source images and historical target images are used to pre-train the image processing model.

[0023] According to a sixth aspect of the embodiments of this specification, an image processing apparatus is provided, applied to a server, comprising:

[0024] The receiving module is configured to receive image processing requests sent by the client, wherein the image processing request carries the image to be processed;

[0025] The second processing module is configured to input the image to be processed into the image processing model to obtain the target image. The image processing model is trained using a sample image set. The sample image set is obtained by mapping at least one initial image in the initial image set using a preset mapping relationship. The preset mapping relationship is the mapping relationship between historical source images and historical target images. The historical source images and historical target images are used to train the image processing model in advance.

[0026] The feedback module is configured to send the target image back to the client so that the client can render the target image.

[0027] According to a seventh aspect of the embodiments of this specification, a weather image restoration apparatus is provided, comprising:

[0028] The second acquisition module is configured to acquire the weather image to be repaired;

[0029] The repair module is configured to input the weather image to be repaired into the image repair model to obtain the target weather image. The image repair model is trained using a sample weather image set. The sample weather image set is obtained by mapping at least one initial sample weather image in the initial weather image set using a preset mapping relationship. The preset mapping relationship is the mapping relationship between historical source weather images and historical target weather images. The historical source weather images and historical target weather images are used to pre-train the image repair model.

[0030] According to an eighth aspect of the embodiments of this specification, an image data processing apparatus is provided, applied to a cloud-side device, comprising:

[0031] The third acquisition module is configured to acquire an initial image set, wherein the initial image set includes multiple initial images;

[0032] The mapping module is configured to use a preset mapping relationship to perform image mapping on at least one initial image in the initial image set to obtain a sample image set, wherein the initial image is an initial source image or an initial target image;

[0033] The first training module is configured to train the image processing model using a set of sample images to obtain the trained image processing model.

[0034] The sending module is configured to send the model parameters of the trained image processing model to the edge device.

[0035] According to a ninth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0036] Memory and processor;

[0037] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the above-mentioned image processing method, weather image restoration method, or image data processing method.

[0038] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-described image processing, weather image restoration, or image data processing methods.

[0039] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described image processing, weather image restoration, or image data processing method.

[0040] In one or more embodiments of this specification, an image to be processed is acquired, and the image to be processed is input into an image processing model to obtain a target image. The image processing model is trained using a sample image set, which is obtained by mapping at least one initial image in the initial image set using a preset mapping relationship. The preset mapping relationship is a mapping relationship between historical source images and historical target images. The historical source images and historical target images are used to pre-train the image processing model. Based on the pre-determined preset mapping relationship between historical source images and historical target images, a large number of historical images are avoided from being stored in advance. During the training of the image processing model, the features of historical images are restored using the preset mapping relationship, resulting in a sample image set containing current and historical knowledge. This is used to train the image processing model, resulting in a more comprehensive and accurate image processing model. While ensuring the comprehensiveness and accuracy of image processing, storage costs are reduced and processing efficiency is improved. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating an image processing method provided in one embodiment of this specification;

[0042] Figure 2 This is a flowchart of another image processing method provided in one embodiment of this specification;

[0043] Figure 3 This is a flowchart illustrating a weather image restoration method provided in one embodiment of this specification;

[0044] Figure 4 This is a flowchart illustrating an image data processing method provided in one embodiment of this specification;

[0045] Figure 5 This is a schematic flowchart of an image processing method provided in one embodiment of this specification;

[0046] Figure 6 This is a schematic diagram of the front-end interface of an image processing method provided in one embodiment of this specification;

[0047] Figure 7 This is a flowchart illustrating the processing steps of an image processing method for road image restoration under severe weather conditions, provided in one embodiment of this specification.

[0048] Figure 8 This is a schematic diagram of the structure of an image processing apparatus provided in one embodiment of this specification;

[0049] Figure 9 This is a schematic diagram of the structure of another image processing apparatus provided in one embodiment of this specification;

[0050] Figure 10 This is a schematic diagram of the structure of a weather image restoration device provided in one embodiment of this specification;

[0051] Figure 11 This is a schematic diagram of the structure of an image data processing apparatus provided in one embodiment of this specification;

[0052] Figure 12 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0053] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0054] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0055] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0056] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0057] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0058] Bayes' Theorem: A probability transformation law used to describe the relationship between two conditional probabilities, specifically P(A|B)=P(B|A)*P(A) / P(B).

[0059] The Gaussian distribution, also known as the normal distribution, is defined as follows: if a random variable X follows a normal distribution with expected value μ and variance σ², it is denoted as N(μ, σ²). Its probability density function is given by the normal distribution. The expected value μ determines the location of the distribution, and the standard deviation σ determines its amplitude. The normal distribution when μ = 0 and σ = 1 is the standard normal distribution.

[0060] The Laplace approximation uses a Gaussian distribution to approximate a complex distribution. The solution process is the same as finding the expectation μ and variance σ² of the normal distribution.

[0061] The Monte Carlo method, also known as statistical simulation or statistical experiment, is a numerical simulation method that takes probabilistic phenomena as its research object. It is a calculation method that uses sampling surveys to obtain statistical values ​​to estimate unknown characteristic quantities.

[0062] Jensen's Inequality: An inequality that characterizes the relationship between the values ​​of convex functions and the integral values ​​of convex functions.

[0063] The Fisher Information Matrix is ​​a generalization of Fisher information quantity from single-parameter to multi-parameter cases. Fisher information quantity represents the average amount of information about the state parameters that a sample of a random variable can provide in a certain sense.

[0064] CNN (Convolutional Neural Networks): A multi-convolutional neural network model with forward and backward propagation.

[0065] VAE (Variational Auto-Encoder) model: A probabilistic model based on variational inference (Variational Bayesian Methods), it is a neural network model with image generation capabilities.

[0066] GAN (Generative Adversarial Network) model: A neural network model for image generation, which includes a generator and a discriminator. By training the generator and discriminator alternately, a high-accuracy generator is obtained to generate images.

[0067] Diffusion model: an image generation model that utilizes a forward process (adding noise) and a backward process (denoising).

[0068] Currently, image processing models are trained by learning from a large number of historical images and then using the generated experiential knowledge. However, effectively learning knowledge in a manner similar to human learning remains challenging, especially in new training tasks. When training image processing models with entirely new sample image sets, the models exhibit a "forgetting" phenomenon after continuous learning. This means that while the model learns current knowledge from the new sample image set, this current knowledge interferes with previously learned historical knowledge. This "forgetting" phenomenon is similar to human forgetting, but the degree of forgetting exceeds that of humans, causing the image processing models trained for new tasks to lose comprehensiveness and accuracy. For example, in an image processing model used for image restoration, the i-th training iteration trains the model's ability to restore rainy images, and the (i+1)-th training iteration trains the model's ability to restore snowy images. The (i+1)-th training iteration interferes with the model's ability to restore rainy images.

[0069] To address the "forgetting" problem in image processing models, it's necessary to reconstruct the sample image set for each training task. This ensures that the trained image processing model not only "learns" current knowledge but also "reviews and reinforces" historical knowledge in the current training task. However, such sample image set reconstruction is time-consuming. Currently, there are two methods to address the "forgetting" phenomenon in image processing models: one method is to construct a new sample image set from all historical and current images. For example, if there are 1 million historical images and 100,000 current images, the sample image set for the current training task is constructed using these 1.1 million images. This method is time-consuming in terms of historical image collection, sample image set construction, and model training, resulting in insufficient model training efficiency. The other method is to sample historical images and mix them with current images to obtain the sample image set for the current training task, and then train the image processing model using these 150,000 images. For example, if there are 1 million historical images, 50,000 are extracted from them, and if there are 100,000 current images, the sample image set for the current training task is constructed using these 150,000 images. This approach, as the model's functionality is continuously updated, requires an increasing number of historical images to be stored in advance. Furthermore, a large number of historical images need to be continuously stored for future updates of new features, increasing data storage costs and reducing processing efficiency.

[0070] To address the aforementioned problems, this specification provides an image processing method. This specification also relates to another image processing method, a weather image restoration method, an image data processing method, an image processing apparatus, another image processing apparatus, a weather image restoration apparatus, an image data processing apparatus, a computing device, a computer-readable storage medium, and a computer program, which will be described in detail in the following embodiments.

[0071] See Figure 1 , Figure 1 A flowchart of an image processing method according to an embodiment of this specification is shown, including the following specific steps:

[0072] Step 102: Obtain the image to be processed.

[0073] The embodiments in this specification apply to the client and / or server of applications, web pages, or mini-programs with image processing functions. These clients and / or servers deploy image processing models to implement image processing functions. These image processing functions include, but are not limited to: image inpainting, image super-resolution, image denoising, and image style transfer.

[0074] The image to be processed is a visual image that can be a real image captured by an image acquisition device, such as a photograph, video frame, or remote sensing image. It can also be an artificially generated image, such as a drawing or animation, or a virtual image generated by an image generation algorithm, such as a drawing or animation. No limitation is made here.

[0075] The acquisition of images to be processed can be either received from a client, such as images uploaded by a user through an application's front-end interface on a mobile terminal, or from an image database. Image databases can include local and remote databases, such as open-source databases or cloud databases. For example, video images of area A recorded by a camera over the past 24 hours could be obtained from a cloud database. Alternatively, images can be received from image acquisition devices. For instance, image sensors on a car might send images from four locations to a server-side application in the vehicle's computer that performs image stitching. The server then stitches these images together to obtain an image of the car's surrounding environment. This is not a limitation of the method described here.

[0076] For example, retrieve 100 road images of road A under rainy weather conditions within 24 hours from the local database, ARoad.Image_j(j∈[1,100]), where the 100 road images are photos of road A within 24 hours.

[0077] Acquiring the image to be processed provides the foundation for subsequent image processing.

[0078] Step 104: Input the image to be processed into the image processing model to obtain the target image. The image processing model is trained using a sample image set. The sample image set is obtained by mapping at least one initial image in the initial image set using a preset mapping relationship. The preset mapping relationship is the mapping relationship between historical source images and historical target images. The historical source images and historical target images are used to train the image processing model in advance.

[0079] Image processing models are neural network models with image processing capabilities. Classified by processing function, they include, but are not limited to: image inpainting models, image super-resolution models, image denoising models, and image style transfer models. Classified by model structure, they include, but are not limited to: CNN models, VAE models, GAN models, and Diffusion models. An image processing model is trained using the current training task, which is the task used to train the current image processing model. Historical training tasks are those prior to the current training task.

[0080] The target image is the visual image obtained by processing the image to be processed. For example, if the image to be processed is a low-resolution visual image, the image processing is image super-resolution, and the target image is a high-resolution visual image. Alternatively, if the image to be processed contains an anime style image, the image processing is style transfer recognition, and the target image is a realistic style image.

[0081] The sample image set is a collection of reconstructed sample images used for the current training task. The sample image set includes multiple sample images. Sample images are the target sample images used for the current training task. Based on image source, sample images are divided into initial images for the current training task and historical feature images obtained through image mapping. Initial images are the initial sample images used for the current training task. Historical feature images are mapped images containing features from historical images; they can be understood as a type of generated sample image. For example, if the sample image set contains 100,000 initial images, image mapping is performed on 40,000 of these initial images to obtain 40,000 historical feature images, resulting in a sample image set of 60,000 initial images and 40,000 historical feature images. Based on image function, sample images are divided into source images and target images corresponding to the source images. Source images are the training sample images in the current training task. Target images are the label sample images in the current training task. The sample source images are used as input to the image processing model in the current training task to obtain the predicted images corresponding to the sample source images. Based on the sample target images and the predicted images, the loss value is determined to adjust the model parameters of the image processing model. It can be understood that the sample image set is a knowledge set containing current and historical knowledge, which has the characteristics of being comprehensive and accurate.

[0082] Historical source images and historical target images are historical images from the historical image set. Historical source images are training sample images from the historical training task. Historical target images are label sample images from the historical training task. Historical source images are input into the image processing model in the historical training task to obtain the corresponding historical prediction images. Based on the historical target images and historical prediction images, the loss value is determined, thus completing the adjustment of the model parameters of the image processing model.

[0083] The preset mapping relationship is the mapping relationship between historical source images and historical target images. It is a quantifiable feature mapping relationship. For example, the image features of the historical source image are X, and the image features of the historical target image are Y. but Represented as a predefined mapping relationship, image mapping can be understood as a method of image feature transformation, and a predefined mapping relationship can be understood as a quantifiable feature transformation function. Furthermore, a predefined mapping relationship is a directional mapping relationship, and its inverse mapping is also a predefined mapping relationship. For example, if the image features of the historical source image are X, and the image features of the historical target image are Y, then... It also represents a pre-defined mapping relationship. For example, if the historical source image is a low-resolution visual image and the historical target image is a high-resolution visual image, the function that predicts the chromaticity value of each pixel is... If it is achieved, then and This refers to a preset mapping relationship. The preset mapping relationship can be pre-extracted or extracted during image processing. During the current training task, if the preset mapping relationship is pre-extracted, it is read from the hard drive and written to the memory cache; if the preset mapping relationship is extracted during image processing, it is written to the memory cache.

[0084] The image to be processed is input into the image processing model to obtain the target image. Specifically, the image to be processed is input into the image processing model, and image processing is performed on the image to be processed to obtain the target image.

[0085] For example, 100 road images ARoad.Image_j (j∈[1,100]) are input into the image inpainting model to perform image inpainting on the 100 road images of road A, and obtain 100 road inpainted images ARoad.TargetImage_j (j∈[1,100]) of road A with raindrops removed from the images.

[0086] In the embodiments of this specification, an image to be processed is acquired, input into an image processing model, and a target image is obtained. The image processing model is trained using a sample image set, which is obtained by mapping at least one initial image in the initial image set using a preset mapping relationship. This preset mapping relationship is a mapping relationship between historical source images and historical target images, used to pre-train the image processing model. Based on the pre-determined preset mapping relationship between historical source images and historical target images, a large number of historical images are avoided from being stored in advance. During the training of the image processing model, the features of historical images are restored using the preset mapping relationship, resulting in a sample image set containing both current and historical knowledge. This allows for training of the image processing model, leading to a more comprehensive and accurate image processing model. This approach reduces storage costs and improves processing efficiency while ensuring the comprehensiveness and accuracy of image processing.

[0087] In one optional embodiment of this specification, the following specific steps are included before step 104:

[0088] Obtain an initial image set, which includes multiple initial images;

[0089] Using a preset mapping relationship, at least one initial image in the initial image set is mapped to obtain a sample image set, wherein the initial image is either an initial source image or an initial target image;

[0090] The image processing model is trained using a sample image set to obtain the trained image processing model.

[0091] The initial image set is a collection of initial sample images used for the current training task. It includes multiple initial images, which are used as initial sample images for the current training task. Based on image function, the initial images are divided into initial source images and initial target images. Initial source images are the initial training sample images for the current training task. Initial target images are the initial label sample images for the current training task.

[0092] Obtain the initial image set by acquiring initial images from an image library. The image library can be an open-source image library, a historical project database, or a combination of both; no specific restrictions are imposed here.

[0093] Using a pre-defined mapping relationship, at least one initial image in the initial image set is mapped to obtain a sample image set. Specifically, this is done by mapping at least one initial image in the initial image set using the pre-defined mapping relationship to obtain historical feature images. These historical feature images and the initial images without mapping are then used as the sample image set. Furthermore, using the pre-defined mapping relationship, at least one initial source image in the initial image set is mapped to obtain historical feature target images. Then, using the pre-defined mapping relationship, at least one initial target image in the initial image set is mapped to obtain historical feature source images. These historical feature source images, historical feature target images, and the initial images without mapping are then used as the sample image set. Historical feature images are mapped images that represent the image features of historical images, including mapped images representing the image features of historical source images and mapped images representing the image features of historical target images. It should be noted that the represented image features are a holistic representation of the image features of all historical training tasks, not the image features of a specific historical image. For example, a historical target image is a denoised historical image, and the historical target image features are the overall image features of the denoised image.

[0094] The image processing model is trained using a sample image set. Specifically, supervised training is performed on the image processing model using source and target images from the sample image set. Further, the source images are input into the image processing model to obtain predicted images. Based on the predicted images and the target images, a loss value is determined. The model parameters are then adjusted based on the loss value. Once the preset training termination conditions are met, the trained image processing model is obtained.

[0095] It should be noted that after training the image processing model, the sample image set is determined as the historical image set, the mapping relationship between the sample images is extracted, and it is added to the preset mapping relationship set for storage, for subsequent iterative updates.

[0096] For example, 100,000 initial weather images (X) are obtained from an open-source weather image library. i+1 Y i+1 An initial weather image set is constructed using pre-extracted preset mapping relationships. For the initial weather image set of 20,000 initial source weather images X i+1 Image mapping was performed to obtain 20,000 historical feature target weather images Y. i ', using preset mapping relationships For the initial weather image set of 20,000 initial target weather images Y i+1Image mapping was performed to obtain 20,000 historical feature source weather images X. i ', X' 20,000 historical feature source weather images i '20,000 historical feature target weather images Y' i 'and 60,000 initial weather images (X) without image mapping i+1 Y i+1 The sample weather image set is determined. Using the source and target weather images from this set, the image inpainting model is trained under supervision to obtain the completed model.

[0097] In the embodiments of this specification, based on the pre-determined mapping relationship between historical source images and historical target images, a large number of historical images are avoided from being stored in advance. During the training of the image processing model, the pre-determined mapping relationship is used to map at least one initial image in the initial image set, restoring the features of the historical images and obtaining a sample image set containing current knowledge and historical knowledge. The image processing model is then trained to obtain a more comprehensive and accurate image processing model, reducing storage costs and improving training efficiency.

[0098] In one optional embodiment of this specification, before performing image mapping on at least one initial image in the initial image set using a preset mapping relationship to obtain a sample image set, the following specific steps are further included:

[0099] Obtain a historical image set, which includes historical source images and historical target images;

[0100] Extract the mapping relationship between historical source images and historical target images to obtain the preset mapping relationship.

[0101] The historical image set is a collection of historical sample images used for the historical training task. It includes multiple historical images, which are used as historical sample images for the historical training task. Based on image function, historical images are divided into historical source images and historical target images. Historical source images are historical training sample images used in the historical training task. Historical target images are historical label sample images used in the historical training task.

[0102] To obtain a historical image set, we need to extract historical images from the historical sample image library used for model training to complete the historical training task and construct the historical image set.

[0103] The mapping relationship between historical source images and historical target images is extracted to obtain a preset mapping relationship. Specifically, a mapping relationship extraction model is used to extract the feature mapping relationship between the image features of the historical source images and the image features of the historical target images, thus obtaining the preset mapping relationship. The mapping relationship extraction model is a neural network model capable of extracting the feature mapping relationship between two image features, quantifying the abstract mapping relationship between two images into a feature mapping relationship between image features. It should be noted that the mapping relationship extraction model corresponds to the historical training task. For example, if the historical training task in the first stage is to train the snow restoration function of the image processing model, then the mapping relationship extraction model in the first stage corresponds to extracting the mapping relationship between snow images and snow-restored images in the historical images.

[0104] It should be noted that the extraction of the preset mapping relationship in the embodiments of this specification is unrelated to the model training task in terms of execution logic. That is, the preset mapping relationship can be extracted after the historical training task is completed, without needing to be executed in the current training task.

[0105] Optionally, after extracting the mapping relationship between the historical source image and the historical target image to obtain the preset mapping relationship, the following specific steps are also included: storing the preset mapping relationship.

[0106] The preset mapping relationships can be stored on persistent storage media such as hard drives, or on dynamic storage media such as memory or cache. The storage size of the preset mapping relationships is much smaller than that of historical images, thus reducing storage costs and improving training efficiency.

[0107] For example, from the historical sample image library used for model training, 10,000 historical weather images that have completed i (i∈[1,i]) historical training tasks are obtained to construct a historical weather image set. Using i mapping relationships, the historical source weather image X is extracted. i Image features (Feature_X) i and historical target weather images Y i Image features_Y i Feature mapping relationship between Get i preset mapping relationships

[0108] In the embodiments of this specification, the mapping relationship between historical source images and historical target images in the historical image set is extracted to obtain a preset mapping relationship, which avoids storing a large number of historical images in advance, reduces storage costs, and improves training efficiency.

[0109] In one optional embodiment of this specification, the initial image set includes multiple initial image subsets, and each initial image subset includes multiple initial images;

[0110] Using historical image mapping, at least one initial image in the initial image set is mapped to obtain a sample image set, including the following specific steps:

[0111] From the extracted preset mapping relationships, the first preset mapping relationship is extracted;

[0112] Using a first preset mapping relationship, at least one initial image in the first initial image subset is image mapped to obtain a first sample image subset, wherein the first initial image subset is any one of multiple initial image subsets;

[0113] Return to the step of extracting the first preset mapping relationship from the extracted preset mapping relationship, and construct a sample image set based on each sample image subset.

[0114] Training tasks are typically divided into multiple training subtasks and iterated through multiple batches of training. This requires dividing the sample image set into multiple sample image subsets and using one sample image subset to complete one training subtask.

[0115] The sample image subset is a subset of the sample image set corresponding to a training subtask in the current training task.

[0116] From the extracted preset mapping relationships, a first preset mapping relationship is obtained. Specifically, this is done by randomly selecting the first preset mapping relationship from the extracted preset mapping relationships. Specifically, a mapping selector is used to achieve random selection. The mapping selector can balance the extracted preset mapping relationships. For example, if the first extraction selects the preset mapping relationship corresponding to the third historical training task, the second extraction will not select the preset mapping relationship corresponding to the third historical training task, thus ensuring data balance in the sample image set.

[0117] Using a first preset mapping relationship, at least one initial image in the first initial image subset is image mapped to obtain a first sample image subset. Specifically, this is done by: using the first preset mapping relationship to perform image mapping on at least one initial image in the first initial image subset to obtain historical feature images; and then defining the historical feature images and the initial images that have not been image mapped as the first sample image subset. Furthermore, using the first preset mapping relationship, at least one first initial source image in the first initial image subset is image mapped to obtain a first historical feature target image; and using the first preset mapping relationship, at least one first initial target image in the first initial image subset is image mapped to obtain a first historical feature source image; and then defining the first historical feature source image, the first historical feature target image, and the first initial images that have not been image mapped as the first sample image subset.

[0118] A sample image set is constructed based on each subset of sample images. Specifically, the subsets are selected based on their information content to form the final sample image set. By selecting the subsets, a larger, richer subset of sample images is selected to participate in the current training task, thus enabling more effective training of the model.

[0119] For example, from the extracted i preset mapping relationships In the process, the first preset mapping relationship is randomly obtained. Using the first preset mapping relationship, the 20 initial source weather images X in the first initial weather image subset are processed. random Image mapping was performed on the images to obtain 20 weather images Y representing the first historical features of the targets. random Using the first preset mapping relationship, the 20 initial target weather images Y in the first initial weather image subset are mapped. random Image mapping was performed to obtain 20 weather images X representing the primary historical feature sources. random The first sample weather image subset is determined by identifying 20 first historical feature source weather images, 20 first historical feature target weather images, and 60 first initial weather images that have not undergone image mapping. The process then returns to execute the extracted i preset mapping relationships. In the process, the first preset mapping relationship is randomly obtained. The steps involve filtering the 1000 sample weather image subsets based on their information content to construct a sample weather image set.

[0120] In the embodiments of this specification, by iteratively extracting the first preset mapping relationship, image mapping is performed on each initial image subset to obtain the corresponding sample image subset, which is used to construct the sample image set. This ensures that a more comprehensive and accurate sample image set is obtained, and ensures that a more comprehensive and accurate image processing model is trained.

[0121] In one optional embodiment of this specification, a sample image set is constructed based on each subset of sample images, including the following specific steps:

[0122] Calculate the information content of each sample image subset;

[0123] If the information content of a subset of sample images reaches a preset threshold, the subset of sample images is added to the sample image set to obtain the sample image set.

[0124] The information content of a sample image subset represents the degree of difference in the image distribution of the sample images within that subset. The higher the degree of difference between the sample images within a subset, the higher the information content. For example, one sample image subset might contain current knowledge and 10 types of historical knowledge (historical feature images obtained through image mapping based on preset mapping relationships corresponding to 10 historical training tasks), while another sample image subset contains current knowledge and 2 types of historical knowledge (historical feature images obtained through image mapping based on preset mapping relationships corresponding to 2 historical training tasks). The former has a higher information content than the latter. Higher information content indicates that the sample image subset contains richer sample images, leading to better training results for the image processing model. Information content can be calculated using information entropy. P i The probability of a sample image of any type can also be calculated using maximum likelihood estimation (LLD = f). D (x1, x2, ..., x n |θ), where θ is the distribution parameter, x1, x2, ..., x n f is the sampled value. D ( ) is the assumed probability density function, which is not restricted here.

[0125] The preset threshold is a pre-defined threshold for judging the difference in image distribution. It can be a statistical value of the information content of each sample image subset, including but not limited to: mean, standard deviation, and variance.

[0126] The information content of each sample image subset is calculated as follows: based on the image distribution of each sample image subset, the information content of each sample image subset is calculated.

[0127] Optionally, if the amount of information in a subset of sample images does not reach a preset threshold, the subset of sample images is filtered out.

[0128] For example, the image distribution (P) based on a subset of 1000 sample weather images. i (i∈[1,1000]), calculate the information content (HM) of each sample weather image subset. i (i∈[1,1000]). If the information content of all sample weather image subsets reaches the mean information content E(HM) and the standard deviation of information content σ, the sample weather image subsets are added to the sample weather image set. If the information content of the sample weather image subsets does not reach the preset threshold, the sample weather image subsets are filtered out to obtain the sample weather image set.

[0129] In the embodiments of this specification, the information content of each sample sub-image is calculated, and the information content of each sample image subset is compared with a preset threshold. If the threshold is met, the subset is added to the sample image set, thereby constructing a more accurate sample image set. This results in a sample image set containing richer sample images, improving the comprehensiveness and accuracy of the sample image set, enhancing the training effect of the image processing model, and obtaining a more comprehensive and accurate image processing model.

[0130] In one optional embodiment of this specification, calculating the information content of each sample image subset includes the following specific steps:

[0131] Calculate the first information content of the initial image subset corresponding to each sample image subset;

[0132] Calculate the second information content of each sample image subset;

[0133] Based on the first and second information values, the information value of each sample image subset is calculated.

[0134] The sample image subset contains current and historical knowledge. It is necessary to consider the amount of information it contains before image mapping and the amount of information it contains after image mapping, and then calculate the information content of the sample image subset.

[0135] The first information content of the initial image subset corresponding to each sample image subset is the difference in image distribution of the initial image subset before image mapping. The specific calculation formula is shown in Formula 1:

[0136]

[0137] in, Let be the first information content of the initial image subset corresponding to the sample image subset in the i-th training task. Given memory, the memory is iteratively updated based on the reconstruction of a subset of sample images, and the specific formula for the update is shown in Formula 2. These are the initial source images in the initial image subset for the i-th training task. Let be the initial target image in the initial image subset of the i-th training task. The sample source images in memory during the i-th training task. Let be the sample target image in memory during the i-th training task.

[0138]

[0139] Where, θ i Let θ be the model parameters of the image processing model trained on the i-th training task. i-1 These are the model parameters of the image processing model trained on the (i-1)th training task. For a training task, ψ represents a subset of multiple sample images. -1 This is a preset mapping relationship.

[0140] The data distribution itself will gradually generalize concepts to new domains (the initial images that complete the image mapping), without forgetting the image distribution of the initial images in the past at any time. This allows the image processing model to continuously learn updated current and historical knowledge.

[0141] The second information content of the sample image subset is the degree of difference in the image distribution of the sample image subset after image mapping. The specific calculation formula is shown in Formula 3:

[0142]

[0143] in, H represents the second information content of the subset of sample images in the i-th training task. i For the image processing model trained on the i-th training task, ψ -1 This is a preset mapping relationship.

[0144] Based on the first and second information values, the information value of each sample image subset is calculated. Specifically, the first and second information values ​​are weighted and calculated to obtain the information value of each sample image subset. The specific calculation formula is as follows:

[0145] As shown in Equation 4:

[0146]

[0147] in, Let be the information content of the subset of sample images in the i-th training task, be the information entropy, and η represent the proportion of the initial image in the subset of sample images for image mapping. This weight parameter weighs the image distribution between the initial image and the historical feature images. Furthermore, based on Jensen's inequality, Equation 4 can be rewritten as Equation 5:

[0148]

[0149] in, Characterized by the mathematical expectation. θ i Let H be the image processing model trained on the i-th training task. i The parameters are used to map a portion of the initial images in the current training task to obtain historical feature images, thus avoiding the preservation of the features of the initial images themselves.

[0150] For example, using Formula 1, the first information content (L1) of the initial weather image subset corresponding to the 1000 sample weather image subsets is calculated. i (i∈[1,1000]), using Formula 3, calculate the second information content (L2) of a subset of 1000 sample weather images. i (i∈[1,1000]), using Formula 5, the first information quantity L1 and the second information quantity L2 are weighted and calculated to obtain the information quantity (HM) of each sample weather image subset. i (i∈[1,1000]).

[0151] In the embodiments of this specification, the first information quantity and the second information quantity are used to fully consider the image distribution of the initial image and the initial image after image mapping (historical feature image), and a more accurate information quantity is calculated, which ensures the comprehensiveness and accuracy of the subsequently determined sample image set.

[0152] In one optional embodiment of this specification, the image processing model is trained using a sample image set to obtain a trained image processing model, including the following specific steps:

[0153] The sample image set is divided into multiple sample image groups;

[0154] A first sample image group is extracted from multiple sample image groups, wherein the first sample image group is any one of the multiple sample image groups, and the first sample image group includes a first source image and a first target image;

[0155] The first source image is input into the image processing model to obtain the first predicted image;

[0156] The loss value is calculated based on the first target image and the first predicted image;

[0157] Based on the loss value, the model parameters of the image processing model are adjusted, and the process returns to extract the first sample image group from multiple sample image groups. If the preset training termination condition is met, the trained image processing model is obtained.

[0158] The current training task includes multiple training subtasks, which are executed iteratively to train the image processing model. A sample image set is the training sample for one training subtask within the current training task, and includes at least one subset of sample images.

[0159] The first source image is the training sample image of the sample image group in the current training sub-task. The first target image is the label sample image of the sample image group in the current training sub-task. The first prediction image is the prediction image corresponding to the first source image obtained by the image processing model in the current training sub-task.

[0160] The preset training termination condition is a pre-defined condition for determining the end of training, including but not limited to: preset number of iterations, preset loss threshold, and preset model convergence condition.

[0161] Based on the loss value, the model parameters of the image processing model are adjusted. Specifically, the model parameters of the image processing model are adjusted using gradient descent based on the loss value.

[0162] For example, the sample weather image set is divided into i sample weather image groups, and a first sample weather image group is extracted from the i sample weather image groups. The first sample weather image group includes a first source weather image and a first target weather image. The first source weather image is input into the image restoration model to obtain a first predicted weather image. Based on the first target weather image and the first predicted weather image, a loss value is calculated. Based on the loss value, the model parameters of the image restoration model are adjusted using the gradient descent method. The process of extracting the first sample weather image group from multiple sample weather image groups is returned. Under the condition that the preset model convergence condition is met, the trained image restoration model is obtained.

[0163] In the embodiments of this specification, the image processing model is trained by using multiple batches of sample images, thereby improving the training effect of the model.

[0164] In one optional embodiment of this specification, adjusting the model parameters of the image processing model based on the loss value includes the following specific steps:

[0165] Based on the image distribution of historical images in the historical image set, the initial model parameters of the image processing model are determined;

[0166] Based on the loss value, the initial model parameters are adjusted to obtain the image processing model parameters, and the first sample image group is added to the historical image set.

[0167] In the embodiments of this specification, after each training subtask is completed, historical images can be updated. To avoid storing and processing historical images, a parameter isolation mechanism is used to change the data transmission effect of synapses (any training subtask), maximizing the preservation of historical knowledge, i.e., preserving the image distribution of historical images. That is, the parameters of the image processing model are restored through the current image distribution between each training subtask, relying only on the parameters of the image processing model corresponding to the historical training subtask, without requiring additional historical images. This allows the image processing model to learn historical knowledge while learning current knowledge.

[0168] The loss value can be considered as a way to find the optimal model parameters based on all known images. Based on this, Bayes' theorem can be used to transform the calculation of the loss value, as shown in Equation 6:

[0169]

[0170] in, For all source and target images (including initial, historical, and sample images), θ represents the model parameters of the image processing model. θ represents the image distribution probabilities of the source and target images, and logp(θ) is the prior probability of the model parameters that can be matched for all training tasks.

[0171] The sample image set is decomposed into n sample image groups: The loss value for the i-th training task is derived as shown in Formula 7:

[0172]

[0173] Among them, This represents the image distribution for the i-th training task. This represents the image distribution of the previous i-1 training tasks. The probability of the image distribution for the i-th task. It is a constant, given the model parameters for the i-th task. The log probability of obtaining the image distribution for the i-th task is equal to the initial loss value.

[0174] in addition, It is a posterior probability distribution term that contains information from all images in the previous training task, making it computationally difficult. It can be approximated as a Gaussian distribution using the diagonal Laplacian approximation. The mean is... The parameters, and the variance, are the Fisher information matrix corresponding to the model parameters. The diagonal reciprocal of . With this approximation, the optimization problem of reducing plasticity can be defined as: to approximate, synaptic transmission from the historical image distribution. Change to the current image distribution Represented as Formula 8:

[0175]

[0176] in, yes The probability density function, i.e., p(θ|D past ).

[0177] For example, the initial model parameters θ of the image processing model are determined based on the sample weather image distribution of historical weather images in a historical weather image set. i-1 Based on the loss value, the initial model parameters θ i-1 Adjustments are made to obtain the image processing model parameters θ. i Add the first sample weather image group to the historical weather image set.

[0178] In the embodiments of this specification, the associative learning method, which uses mapping relationship restoration and parameter isolation as a medium, enables the image processing model to associate historical training tasks with the current training task. It can restore the experience knowledge of historical tasks based on the current sample images without relying on historical images, adaptively and continuously learn the current task while retaining the knowledge of historical tasks, thereby improving the accuracy of the loss value and further enhancing the comprehensiveness and accuracy of the trained image processing model.

[0179] In one optional embodiment of this specification, adding the first sample image group to the historical image set includes the following specific steps:

[0180] Compare each sample image in the first sample image group with each historical image in the historical image set, and remove redundant sample images in the first sample image group;

[0181] Add the first sample image group to the history image set.

[0182] The image sets for different training tasks are not completely independent. To address the conflict in image distribution within the mixed sample image sets, given prior information about the mixed samples... In this case, the Monte Carlo method is used to distribute the distribution. Approximately For formula 7 Rewrite it as shown in Formula 9:

[0183]

[0184] in, and These represent the initial source image in the current training task, the initial image in the current training task, and the historical feature source image and historical feature target image, respectively. Image distribution. Image distribution representing the sample image set The Monte Carlo method is used to characterize the true image distribution of the sample image set learned by the image processing model. Combined with parameter isolation in the above embodiments, the new weighting scheme remains balanced.

[0185] For example, by comparing each sample weather image in the first sample weather image group with each historical weather image in the historical weather image set, redundant sample weather images in the first sample weather image group are removed, and the first sample weather image group is added to the historical weather image set.

[0186] In the embodiments described in this specification, redundant sample images are removed to avoid conflicts in image distribution, thereby improving the accuracy of the loss value and enhancing the training effect of the image processing model.

[0187] See Figure 2 , Figure 2 A flowchart of another image processing method according to an embodiment of this specification is shown. The method is applied to a server and includes the following specific steps:

[0188] Step 202: Receive an image processing request sent by the client, wherein the image processing request carries the image to be processed.

[0189] Step 204: Input the image to be processed into the image processing model to obtain the target image. The image processing model is trained using a sample image set. The sample image set is obtained by mapping at least one initial image in the initial image set using a preset mapping relationship. The preset mapping relationship is the mapping relationship between historical source images and historical target images. The historical source images and historical target images are used to train the image processing model in advance.

[0190] Step 206: Feed back the target image to the client so that the client can render the target image.

[0191] This specification describes embodiments applicable to server-side applications, web pages, or mini-programs with image processing capabilities. These servers deploy image processing models to implement image processing functions. These functions include, but are not limited to, image inpainting, image super-resolution, image denoising, and image style transfer. A network connection exists between the client and server, establishing a data link for data transmission. The server is equipped with larger storage space and higher computing power.

[0192] The specific implementation of image rendering is achieved using a renderer.

[0193] Step 204 in the embodiments of this specification Figure 1 Step 104 in the embodiments is the same and has been described in detail, so it will not be repeated here.

[0194] In this embodiment, an image processing request sent by a client is received. The request carries an image to be processed. The image to be processed is input into an image processing model to obtain a target image. The image processing model is trained using a sample image set. The sample image set is obtained by mapping at least one initial image in the initial image set using a preset mapping relationship. The preset mapping relationship is a mapping relationship between historical source images and historical target images. The historical source images and historical target images are used to pre-train the image processing model. The target image is then fed back to the client so that the client can render the target image. Based on the pre-determined preset mapping relationship between historical source images and historical target images, a large number of historical images are avoided from being stored in advance. During the training of the image processing model, the features of historical images are restored using the preset mapping relationship, resulting in a sample image set containing current and historical knowledge. This allows for training of the image processing model, resulting in a more comprehensive and accurate image processing model. While ensuring the comprehensiveness and accuracy of image processing, storage costs are reduced and processing efficiency is improved. Image processing is implemented on the server side and rendered on the client side, reducing the client's computing and storage costs and improving the user experience.

[0195] See Figure 3 , Figure 3 A flowchart of a weather image restoration method according to an embodiment of this specification is shown, including the following specific steps:

[0196] Step 302: Obtain the weather image to be repaired.

[0197] Step 304: Input the weather image to be repaired into the image repair model to obtain the target weather image. The image repair model is trained using a sample weather image set. The sample weather image set is obtained by mapping at least one initial sample weather image in the initial weather image set using a preset mapping relationship. The preset mapping relationship is the mapping relationship between historical source weather images and historical target weather images. The historical source weather images and historical target weather images are used to train the image repair model in advance.

[0198] The weather image to be repaired is a visual image of the weather conditions, such as a road video during severe weather. The target weather image is the visual image of the weather obtained by performing image inpainting on the weather image to be repaired.

[0199] Image restoration models are neural network models with image restoration capabilities. Based on model structure, they include, but are not limited to, CNN models, VAE models, GAN models, and Diffusion models. Image restoration models are trained using the current training task.

[0200] The embodiments in this specification are... Figure 1The embodiments are based on the same inventive concept, and steps 302 and 304 are referenced. Figure 1 Steps 102 and 104 in the embodiments have been described in detail and will not be repeated here.

[0201] In the embodiments of this specification, a weather image to be repaired is acquired, and then input into an image repair model to obtain a target weather image. The image repair model is trained using a set of sample weather images. This set of sample weather images is obtained by mapping at least one initial sample weather image from the initial weather image set using a preset mapping relationship. This preset mapping relationship is a mapping relationship between historical source weather images and historical target weather images. The historical source weather images and historical target weather images are used to pre-train the image repair model. Based on the pre-determined preset mapping relationship between historical source weather images and historical target weather images, a large number of historical weather images are avoided from being stored in advance. During the training of the image repair model, the features of historical weather images are restored using the preset mapping relationship, resulting in a set of sample weather images containing both current and historical knowledge. This allows for the training of the image repair model, leading to a more comprehensive and accurate image repair model. This approach ensures the comprehensiveness and accuracy of weather image repair while reducing storage costs and improving the efficiency of weather image repair.

[0202] See Figure 4 , Figure 4 A flowchart of an image data processing method according to an embodiment of this specification is shown. The method is applied to a cloud-side device and includes the following specific steps:

[0203] Step 402: Obtain an initial image set, wherein the initial image set includes multiple initial images;

[0204] Step 404: Using a preset mapping relationship, perform image mapping on at least one initial image in the initial image set to obtain a sample image set, wherein the initial image is either an initial source image or an initial target image;

[0205] Step 406: Use the sample image set to train the image processing model to obtain the trained image processing model;

[0206] Step 408: Send the model parameters of the trained image processing model to the edge device.

[0207] The embodiments in this specification apply to cloud-side devices with model training capabilities. A cloud-side device is a network cloud device that provides model training capabilities; it is a virtual device. An edge device is a terminal device that provides image processing capabilities; it is a physical device. The edge device and the cloud-side device are connected by a network, establishing a data link to achieve data transmission. The computing power of the cloud-side device is higher than that of the edge device.

[0208] In the embodiments of this specification, steps 402 to 406 have been described above. Figure 1 The embodiments are described in detail and will not be repeated here.

[0209] In this embodiment, an initial image set is obtained, comprising multiple initial images. Using a preset mapping relationship, at least one initial image in the initial image set is mapped to obtain a sample image set. Each initial image can be either an initial source image or an initial target image. The image processing model is trained using the sample image set to obtain a trained image processing model. The model parameters of the trained image processing model are then sent to the edge device. Based on a pre-determined preset mapping relationship between historical source images and historical target images, this avoids pre-storing a large number of historical images. During the training of the image processing model, at least one initial image in the initial image set is mapped using the preset mapping relationship, restoring the features of the historical images and obtaining a sample image set containing both current and historical knowledge. Training the image processing model with this sample image set yields a more comprehensive and accurate model, reducing storage costs and improving training efficiency. Furthermore, model training is implemented through cloud-side devices, reducing client-side computing costs and improving model training efficiency.

[0210] Figure 5 A schematic flowchart of an image processing method according to one embodiment of this specification is shown. Figure 5 As shown:

[0211] During the continuous learning process of the image processing model, data streams are continuously acquired to iterate the training task. In historical training task i, the corresponding preset mapping relationship i is extracted and jointly stored for use in the current training task i+1. Specifically, in the current training task i+1, preset mapping relationships are extracted from the historical source images and historical target images in the historical image set: the historical image set of historical training task 1 includes historical source image 1 and historical target image 1; using the mapping relationship extraction model of stage 1, preset mapping relationship 1 is extracted, and so on. The historical image set of historical training task i includes historical source image i and historical target image i; using the mapping relationship extraction model of stage i, preset mapping relationship i is extracted. The extracted preset mapping relationships are written into a memory buffer. Obtain the initial image set for the current training task i+1, which includes the initial source image i+1 and the initial target image i+1. Use a mapping selector to randomly select a preset mapping relationship from the preset mapping relationships, and perform image mapping on the initial source image i+1 and the initial target image i+1 to obtain the sample image set for the current training task i+1, which includes the sample source image i+1 and the sample target image i+1. Use the sample image set for the current training task i+1 to train the image processing model.

[0212] Figure 6 This specification illustrates a schematic diagram of a front-end interface for an image processing method according to an embodiment of this specification, such as... Figure 6 As shown:

[0213] The front-end interface of the image processing application includes a first interface, a second interface, and a third interface, which are switched interactively. The application features weather image restoration, image super-resolution, and filter addition functions. These functions are implemented through multiple training tasks on the image processing model pre-configured within the application. The first interface includes a "Load Weather Image" control, an image preview area, a "Weather Image Restoration" control, an "Image Super-Resolution" control, and an "Add Filter" control. Users select the "Load Weather Image" control to load the weather image to be restored, and the image preview area renders the image. Users then select the "Weather Image Restoration" control again to enter the second interface. The second interface features controls for various weather restoration types: "Rainy Day," "Snowy Day," "Sandstorm," "Foggy Day," "Sunny Day," and "Back." These controls are implemented based on multiple weather image restoration functions within an image processing model. These functions are achieved through multiple training tasks on the image processing model configured in the application. Users select the "Rainy Day" control to perform rainy weather image restoration on the image to be restored, leading to the third interface. The third interface includes an image preview area, "Download," "Copy," and "Back" controls. The image preview area renders the restored target weather image, and users can activate the corresponding function by selecting different controls.

[0214] The following is in conjunction with the appendix Figure 7 Taking the application of the image processing method provided in this specification for road image restoration under severe weather conditions as an example, the image processing method will be further explained. Among other things, Figure 7 The present specification illustrates a process flowchart of an image processing method for road image restoration under severe weather conditions, according to an embodiment of this specification, including the following specific steps:

[0215] Step 702: Obtain a set of historical road images, extract and store the mapping relationship between the source images of historical roads and the target images of historical roads, and obtain a preset mapping relationship.

[0216] Step 704: Obtain the initial road image set.

[0217] Step 706: Extract the first preset mapping relationship from the extracted and stored preset mapping relationships.

[0218] Step 708: Using the first preset mapping relationship, perform image mapping on at least one initial road image in the first initial road image subset to obtain the first sample road image subset.

[0219] Step 710: Return to the step of extracting the first preset mapping relationship from the preset mapping relationships that have been extracted and stored.

[0220] Step 712: Calculate the first information content of the initial road image subset corresponding to each sample road image subset, calculate the second information content of each sample road image subset, and calculate the information content of each sample road image subset based on the first and second information contents.

[0221] Step 714: When the amount of information in the sample road image subset reaches a preset threshold, add the sample road image subset to the sample road image set to obtain the sample road image set.

[0222] Step 716: Divide the sample road image set into multiple sample road image groups, extract the first sample road image group from the multiple sample road image groups, input the first source image into the image restoration model to obtain the first predicted image, input the first source image into the image restoration model to obtain the first predicted image, determine the initial model parameters of the image restoration model based on the sample image distribution of historical road images in the historical road image set, adjust the initial model parameters based on the loss value to obtain the image restoration model parameters, add the first sample image group to the historical road image set, return to execute the extraction of the first sample road image group from the multiple sample road image groups, and obtain the trained image restoration model when the preset training termination condition is met.

[0223] Step 718: Obtain images of the road to be repaired under severe weather conditions.

[0224] Step 720: Input the image of the road to be repaired into the image restoration model to obtain the repaired road image.

[0225] This specification proposes a novel storage management scheme, associative memory, in its embodiments to sustainably reconstruct sample image sets. It establishes connections between different image distributions, preserving historical knowledge from historical images. In the current training task, historical knowledge is learned through image mapping, ensuring that the sample image set contains both current and historical knowledge. This allows for the training of the image processing model, resulting in a more comprehensive and accurate model. Furthermore, a parameter isolation strategy is used between historical and current training tasks to reinforce historical knowledge within the sample image set. This approach reduces storage costs and improves processing efficiency while maintaining the comprehensiveness and accuracy of image processing.

[0226] Corresponding to the above method embodiments, this specification also provides embodiments of an image processing apparatus. Figure 8 A schematic diagram of the structure of an image processing apparatus provided in one embodiment of this specification is shown. Figure 3 As shown, the device includes:

[0227] The first acquisition module 802 is configured to acquire the image to be processed;

[0228] The first processing module 804 is configured to input the image to be processed into the image processing model to obtain the target image. The image processing model is trained using a sample image set. The sample image set is obtained by mapping at least one initial image in the initial image set using a preset mapping relationship. The preset mapping relationship is the mapping relationship between historical source images and historical target images. The historical source images and historical target images are used to train the image processing model in advance.

[0229] Optionally, the device further includes:

[0230] The training module is configured to acquire an initial image set, which includes multiple initial images; use a preset mapping relationship to perform image mapping on at least one initial image in the initial image set to obtain a sample image set, wherein the initial images are initial source images or initial target images; and use the sample image set to train the image processing model to obtain the trained image processing model.

[0231] Optionally, the device further includes:

[0232] The extraction module is configured to acquire a historical image set, which includes historical source images and historical target images; and to extract the mapping relationship between the historical source images and the historical target images to obtain a preset mapping relationship.

[0233] Optionally, the initial image set includes multiple initial image subsets, and each initial image subset includes multiple initial images;

[0234] Correspondingly, the training module is further configured as follows:

[0235] From the extracted preset mapping relationships, a first preset mapping relationship is extracted; using the first preset mapping relationship, at least one initial image in the first initial image subset is image mapped to obtain a first sample image subset, wherein the first initial image subset is any one of multiple initial image subsets; return to execute the step of extracting the first preset mapping relationship from the extracted preset mapping relationships, and construct a sample image set based on each sample image subset.

[0236] Optionally, the training module is further configured as follows:

[0237] Calculate the information content of each sample image subset; when the information content of a sample image subset reaches a preset threshold, add the sample image subset to the sample image set to obtain the sample image set.

[0238] Optionally, the training module is further configured as follows:

[0239] Calculate the first information content of the initial image subset corresponding to each sample image subset; calculate the second information content of each sample image subset; and calculate the information content of each sample image subset based on the first and second information contents.

[0240] Optionally, the training module is further configured as follows:

[0241] The sample image set is divided into multiple sample image groups; a first sample image group is extracted from the multiple sample image groups, wherein the first sample image group is any one of the multiple sample image groups, and the first sample image group includes a first source image and a first target image; the first source image is input into the image processing model to obtain a first predicted image; based on the first target image and the first predicted image, the loss value is calculated; based on the loss value, the model parameters of the image processing model are adjusted, and the process of extracting the first sample image group from the multiple sample image groups is returned. Under the condition of reaching the preset training termination condition, the trained image processing model is obtained.

[0242] Optionally, the training module is further configured as follows:

[0243] Based on the image distribution of historical images in the historical image set, the initial model parameters of the image processing model are determined; based on the loss value, the initial model parameters are adjusted to obtain the image processing model parameters, and the first sample image group is added to the historical image set.

[0244] Optionally, the training module is further configured as follows:

[0245] Compare each sample image in the first sample image group with each historical image in the historical image set, remove redundant sample images in the first sample image group, and add the first sample image group to the historical image set.

[0246] In the embodiments of this specification, an image to be processed is acquired, input into an image processing model, and a target image is obtained. The image processing model is trained using a sample image set, which is obtained by mapping at least one initial image in the initial image set using a preset mapping relationship. This preset mapping relationship is a mapping relationship between historical source images and historical target images, used to pre-train the image processing model. Based on the pre-determined preset mapping relationship between historical source images and historical target images, a large number of historical images are avoided from being stored in advance. During the training of the image processing model, the features of historical images are restored using the preset mapping relationship, resulting in a sample image set containing both current and historical knowledge. This allows for training of the image processing model, leading to a more comprehensive and accurate image processing model. This approach reduces storage costs and improves processing efficiency while ensuring the comprehensiveness and accuracy of image processing.

[0247] The above is an illustrative scheme of an image processing apparatus according to this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the image processing apparatus, please refer to the description of the technical solution of the image processing method described above.

[0248] Corresponding to the above method embodiments, this specification also provides embodiments of an image processing apparatus. Figure 9 A schematic diagram of another image processing apparatus provided in one embodiment of this specification is shown. Figure 9 As shown, this device is used on the server side, and the device includes:

[0249] The receiving module 902 is configured to receive an image processing request sent by a client, wherein the image processing request carries an image to be processed;

[0250] The second processing module 904 is configured to input the image to be processed into the image processing model to obtain the target image. The image processing model is trained using a sample image set. The sample image set is obtained by mapping at least one initial image in the initial image set using a preset mapping relationship. The preset mapping relationship is the mapping relationship between historical source images and historical target images. The historical source images and historical target images are used to train the image processing model in advance.

[0251] Feedback module 906 is configured to feed back the target image to the client so that the client can render the target image.

[0252] In this embodiment, an image processing request sent by a client is received. The request carries an image to be processed. The image to be processed is input into an image processing model to obtain a target image. The image processing model is trained using a sample image set. The sample image set is obtained by mapping at least one initial image in the initial image set using a preset mapping relationship. The preset mapping relationship is a mapping relationship between historical source images and historical target images. The historical source images and historical target images are used to pre-train the image processing model. The target image is then fed back to the client so that the client can render the target image. Based on the pre-determined preset mapping relationship between historical source images and historical target images, a large number of historical images are avoided from being stored in advance. During the training of the image processing model, the features of historical images are restored using the preset mapping relationship, resulting in a sample image set containing current and historical knowledge. This allows for training of the image processing model, resulting in a more comprehensive and accurate image processing model. While ensuring the comprehensiveness and accuracy of image processing, storage costs are reduced and processing efficiency is improved. Image processing is implemented on the server side and rendered on the client side, reducing the client's computing and storage costs and improving the user experience.

[0253] The above is an illustrative scheme of an image processing apparatus according to this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the image processing apparatus, please refer to the description of the technical solution of the image processing method described above.

[0254] Corresponding to the above method embodiments, this specification also provides embodiments of a weather image restoration device. Figure 10 A schematic diagram of a weather image restoration apparatus according to one embodiment of this specification is shown. Figure 10 As shown, the device includes:

[0255] The second acquisition module 1002 is configured to acquire the weather image to be repaired;

[0256] The repair module 1004 is configured to input the weather image to be repaired into the image repair model to obtain the target weather image. The image repair model is trained using a sample weather image set. The sample weather image set is obtained by mapping at least one initial sample weather image in the initial weather image set using a preset mapping relationship. The preset mapping relationship is the mapping relationship between historical source weather images and historical target weather images. The historical source weather images and historical target weather images are used to pre-train the image repair model.

[0257] In the embodiments of this specification, a weather image to be repaired is acquired, and then input into an image repair model to obtain a target weather image. The image repair model is trained using a set of sample weather images. This set of sample weather images is obtained by mapping at least one initial sample weather image from the initial weather image set using a preset mapping relationship. This preset mapping relationship is a mapping relationship between historical source weather images and historical target weather images. The historical source weather images and historical target weather images are used to pre-train the image repair model. Based on the pre-determined preset mapping relationship between historical source weather images and historical target weather images, a large number of historical weather images are avoided from being stored in advance. During the training of the image repair model, the features of historical weather images are restored using the preset mapping relationship, resulting in a set of sample weather images containing both current and historical knowledge. This allows for the training of the image repair model, leading to a more comprehensive and accurate image repair model. This approach ensures the comprehensiveness and accuracy of weather image repair while reducing storage costs and improving the efficiency of weather image repair.

[0258] The above is a schematic scheme of a weather image restoration device according to this embodiment. It should be noted that the technical solution of this weather image restoration device and the technical solution of the weather image restoration method described above belong to the same concept. For details not described in detail in the technical solution of the weather image restoration device, please refer to the description of the technical solution of the weather image restoration method described above.

[0259] Corresponding to the above method embodiments, this specification also provides embodiments of image data processing apparatus. Figure 11 A schematic diagram of an image data processing apparatus according to one embodiment of this specification is shown. Figure 11 As shown, this device is applied to cloud-side equipment. The device includes:

[0260] The third acquisition module 1102 is configured to acquire an initial image set, wherein the initial image set includes multiple initial images;

[0261] The mapping module 1104 is configured to perform image mapping on at least one initial image in the initial image set using a preset mapping relationship to obtain a sample image set, wherein the initial image is an initial source image or an initial target image;

[0262] Training module 1106 is configured to train the image processing model using a sample image set to obtain the trained image processing model;

[0263] The sending module 1108 is configured to send the model parameters of the trained image processing model to the edge device.

[0264] In this embodiment, an initial image set is obtained, comprising multiple initial images. Using a preset mapping relationship, at least one initial image in the initial image set is mapped to obtain a sample image set. Each initial image can be either an initial source image or an initial target image. The image processing model is trained using the sample image set to obtain a trained image processing model. The model parameters of the trained image processing model are then sent to the edge device. Based on a pre-determined preset mapping relationship between historical source images and historical target images, this avoids pre-storing a large number of historical images. During the training of the image processing model, at least one initial image in the initial image set is mapped using the preset mapping relationship, restoring the features of the historical images and obtaining a sample image set containing both current and historical knowledge. Training the image processing model with this sample image set yields a more comprehensive and accurate model, reducing storage costs and improving training efficiency. Furthermore, model training is implemented through cloud-side devices, reducing client-side computing costs and improving model training efficiency.

[0265] The above is an illustrative scheme of an image data processing apparatus according to this embodiment. It should be noted that the technical solution of this image data processing apparatus and the technical solution of the image data processing method described above belong to the same concept. For details not described in detail in the technical solution of the image data processing apparatus, please refer to the description of the technical solution of the image data processing method described above.

[0266] Figure 12 A structural block diagram of a computing device according to one embodiment of this specification is shown. The components of the computing device 1200 include, but are not limited to, a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and a database 1250 is used to store data.

[0267] The computing device 1200 also includes an access device 1240, which enables the computing device 1200 to communicate via one or more networks 1260. Examples of such networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1240 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0268] In one embodiment of this specification, the aforementioned components of the computing device 1200 and Figure 12 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 12 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0269] The computing device 1200 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1200 can also be a mobile or stationary server.

[0270] The processor 1220 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned image processing method, weather image restoration method, or image data processing method.

[0271] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solutions of the image processing method, weather image restoration method, and image data processing method described above. For details not described in detail in the technical solution of the computing device, please refer to the descriptions of the technical solutions of the image processing method, weather image restoration method, or image data processing method described above.

[0272] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described image processing method, weather image restoration method, or image data processing method.

[0273] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the image processing method, weather image restoration method, and image data processing method described above. For details not described in detail in the technical solution of the storage medium, please refer to the descriptions of the technical solutions of the image processing method, weather image restoration method, or image data processing method described above.

[0274] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described image processing method, weather image restoration method, or image data processing method.

[0275] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program belongs to the same concept as the technical solutions of the image processing method, weather image restoration method, and image data processing method described above. Details not described in detail in the technical solution of the computer program can be found in the descriptions of the technical solutions of the image processing method, weather image restoration method, or image data processing method described above.

[0276] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0277] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0278] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0279] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0280] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An image processing method, comprising: Obtain the image to be processed; The image to be processed is input into the image processing model to obtain the target image. The image processing model is trained using a sample image set. The sample image set is obtained by mapping at least one initial image in the initial image set using a preset mapping relationship. The preset mapping relationship is the mapping relationship between historical source images and historical target images. The historical source images and the historical target images are used to pre-train the image processing model. The sample image set is a reconstructed set of sample images for the current training task.

2. The method according to claim 1, further comprising, before inputting the image to be processed into the image processing model to obtain the target image: Obtain an initial image set, wherein the initial image set includes multiple initial images; Using a preset mapping relationship, at least one initial image in the initial image set is image mapped to obtain a sample image set, wherein the initial image is an initial source image or an initial target image; The image processing model is trained using the sample image set to obtain the trained image processing model.

3. The method according to claim 2, before performing image mapping on at least one initial image in the initial image set using a preset mapping relationship to obtain a sample image set, further comprising: Obtain a historical image set, wherein the historical image set includes historical source images and historical target images; Extract the mapping relationship between the historical source image and the historical target image to obtain a preset mapping relationship.

4. The method according to claim 3, wherein the initial image set comprises a plurality of initial image subsets, and each initial image subset comprises a plurality of initial images; The step of using a preset mapping relationship to perform image mapping on at least one initial image in the initial image set to obtain a sample image set includes: From the extracted preset mapping relationships, a first preset mapping relationship is obtained; Using the first preset mapping relationship, at least one initial image in the first initial image subset is image mapped to obtain a first sample image subset, wherein the first initial image subset is any one of the plurality of initial image subsets; Return to the step of extracting the first preset mapping relationship from the extracted preset mapping relationship, and construct a sample image set based on each sample image subset.

5. The method according to claim 4, wherein constructing the sample image set based on each subset of sample images includes: Calculate the information content of each sample image subset; When the information content of a subset of sample images reaches a preset threshold, the subset of sample images is added to the sample image set to obtain the sample image set.

6. The method according to claim 5, wherein calculating the information content of each sample image subset includes: Calculate the first information content of the initial image subset corresponding to each sample image subset; Calculate the second information content of each sample image subset; Based on the first information content and the second information content, the information content of each sample image subset is calculated.

7. The method according to any one of claims 3 to 6, wherein training the image processing model using the sample image set to obtain the trained image processing model comprises: The sample image set is divided into multiple sample image groups; A first sample image group is extracted from the plurality of sample image groups, wherein the first sample image group is any one of the plurality of sample image groups, and the first sample image group includes a first source image and a first target image; The first source image is input into the image processing model to obtain the first predicted image; The loss value is calculated based on the first target image and the first predicted image; Based on the loss value, the model parameters of the image processing model are adjusted, and the process of extracting the first sample image group from the multiple sample image groups is returned. When the preset training termination condition is met, the trained image processing model is obtained.

8. The method according to claim 7, wherein adjusting the model parameters of the image processing model based on the loss value includes: Based on the image distribution of historical images in the historical image set, the initial model parameters of the image processing model are determined; Based on the loss value, the initial model parameters are adjusted to obtain image processing model parameters, and the first sample image group is added to the historical image set.

9. The method according to claim 8, wherein adding the first sample image group to the historical image set comprises: Compare each sample image in the first sample image group with each historical image in the historical image set, and remove redundant sample images in the first sample image group; Add the first sample image group to the historical image set.

10. An image processing method, applied on a server, comprising: Receive an image processing request sent by a client, wherein the image processing request carries an image to be processed; The image to be processed is input into the image processing model to obtain the target image. The image processing model is trained using a sample image set. The sample image set is obtained by mapping at least one initial image in the initial image set using a preset mapping relationship. The preset mapping relationship is the mapping relationship between historical source images and historical target images. The historical source images and the historical target images are used to pre-train the image processing model. The sample image set is a reconstructed set of sample images for the current training task. The target image is fed back to the client so that the client can render the target image.

11. A weather image restoration method, comprising: Acquire the weather image to be repaired; The weather image to be repaired is input into the image restoration model to obtain the target weather image. The image restoration model is trained using a sample weather image set. The sample weather image set is obtained by mapping at least one initial sample weather image in the initial weather image set using a preset mapping relationship. The preset mapping relationship is the mapping relationship between historical source weather images and historical target weather images. The historical source weather images and the historical target weather images are used to pre-train the image restoration model. The sample weather image set is a reconstructed set of sample images used for the current training task.

12. An image data processing method, applied to a cloud-side device, comprising: Obtain an initial image set, wherein the initial image set includes multiple initial images; Using a preset mapping relationship, at least one initial image in the initial image set is mapped to obtain a sample image set. The initial image is either an initial source image or an initial target image. The preset mapping relationship is a mapping relationship between historical source images and historical target images. The historical source images and the historical target images are used to pre-train the image processing model. The sample image set is a reconstructed collection of sample images for the current training task. The image processing model is trained using the sample image set to obtain the trained image processing model. The model parameters of the trained image processing model are sent to the edge device.

13. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 12.

14. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

Images