Weather migration model training method, weather migration method and device

Abstract

The invention provides a training method of a weather migration model and a weather migration method and device, and relates to the technical field of data migration, and the training method of the weather migration model comprises the steps: obtaining a plurality of to-be-recovered images and weather keywords corresponding to the plurality of to-be-recovered images, the geographic position information and / or weather information among the plurality of to-be-restored images are different; according to the plurality of weather keywords, performing image restoration on the plurality of to-be-restored images to obtain a plurality of target restored images; iteratively training the initial weather migration model according to the plurality of target recovery images and the plurality of weather texts to be migrated to obtain a target weather migration model, and inputting the target weather migration text and the reference image into the target weather migration model in the weather migration method to obtain a target migration image. The accuracy of the image obtained through weather migration is improved.

Description

Technical Field

[0001] This application relates to the technical field of data migration, and particularly relates to a training method for a weather migration model, a weather migration method, and an apparatus. Background Art

[0002] Weather migration is to migrate or integrate various different weather scene data (such as rain, snow, etc.) into different real environments (for example, migrating snow data into an image with location information such as teaching buildings, office buildings, roads, plants, etc.). Existing weather migration solutions lack large-scale training data sets. At the same time, there is a lack of sufficient real image references that conform to physical laws for the performance changes caused by different weathers. For example, snow accumulation in snowy days and water accumulation in rainy days cannot be realized in the vast majority of weather migration solutions. For the task of weather migration, the best data form is paired images of the same location corresponding to different weathers, learning the style migration of different weathers. However, it is very costly to collect paired data, so it is difficult to achieve.

[0003] To solve the problem of high cost of collecting paired data, existing mainstream weather migration solutions use unpaired images (that is, the weather information and location information corresponding to the two images are both different). During the learning process, content and weather style need to be decoupled, but there are often image detail defects caused by incomplete decoupling, resulting in low accuracy of the images after weather migration.

[0004] Regarding the problem of low image accuracy in existing weather migration solutions, no effective technical solution has been proposed yet. Summary of the Invention

[0005] Embodiments of this application provide a method to at least solve the problem of low image accuracy in existing weather migration solutions.

[0006] According to one aspect of the embodiments of this application, a training method for a weather migration model is provided, including: obtaining multiple images to be restored, and weather keywords corresponding to each of the multiple images to be restored, where there are differences in geographical location information and / or weather information among the multiple images to be restored; performing image restoration on the multiple images to be restored according to the multiple weather keywords to obtain multiple target restored images, each target restored image being an image containing geographical location information and preset weather information, and the weather information being different from the weather indicated by the preset weather information; and iteratively training an initial weather migration model according to the multiple target restored images and multiple weather texts to be migrated to obtain a target weather migration model.

[0007] According to another aspect of the embodiments of the present application, a weather migration method is further provided, including: obtaining a target weather migration text and a reference image, where the reference image is an image containing geographical location information and preset weather information; inputting the target weather migration text and the reference image into a target weather migration model to obtain a target migration image, and the target weather migration model is a model trained by a training method of the weather migration model.

[0008] According to another aspect of the embodiments of the present application, a training device for a weather migration model is further provided, including: a training acquisition unit, configured to obtain multiple images to be restored, and weather keywords corresponding to each of the multiple images to be restored, where there are differences in geographical location information and / or weather information among the multiple images to be restored; a restoration unit, configured to perform image restoration on the multiple images to be restored according to the multiple weather keywords to obtain multiple target restored images, and each target restored image is an image containing geographical location information and preset weather information, and the weather information and the preset weather information indicate different weathers; a training unit, configured to perform iterative training on an initial weather migration model according to the multiple target restored images and multiple weather texts to be migrated to obtain a target weather migration model.

[0009] Optionally, the above-mentioned restoration unit includes a first restoration subunit, configured to perform a first restoration operation on each of the multiple images to be restored by using an image restoration model according to the multiple weather keywords to obtain multiple first restored images; an acquisition subunit, configured to obtain first click information of a target account for each of the first restored images; a second restoration subunit, configured to perform a second restoration operation on each of the first restored images according to the first click information of each of the first restored images by using a segmentation model to obtain multiple target restored images.

[0010] Optionally, the above-mentioned first restoration subunit includes a first determination module, configured to determine any unrecovered image among the multiple images to be restored as the current image to be restored, and determine the weather keyword corresponding to the current image to be restored as the current weather keyword; a token module, configured to perform tokenization on the current image to be restored to obtain a token result set; a diffusion restoration module, configured to perform a diffusion restoration operation on the token result set by using a first image restoration sub-model according to the current weather keyword to obtain a diffusion restored image, and the diffusion restoration operation is used to remove floating objects corresponding to the current image to be restored; an image repair module, configured to perform an image repair operation on the diffusion restored image by using a second image restoration sub-model to obtain a first restored image, and the image repair operation is used to repair unreasonable information in the diffusion restored image, and the unreasonable information is formed by the diffusion restoration operation.

[0011] Optionally, the second recovery subunit described above includes a generation module, configured to determine the position information corresponding to the first click operation according to the first click information, and generate a heatmap area according to the position information, where the heatmap area is used to indicate the relevance around the position information; a second determination module, configured to determine the segmentation type corresponding to the first click operation according to the heatmap area; a third determination module, configured to determine the information to be segmented in the first recovery image according to the heatmap area and the segmentation type; and a segmentation module, configured to segment the first recovery image according to the information to be segmented to obtain multiple target recovery images.

[0012] Optionally, the training unit described above includes a training acquisition subunit, configured to acquire multiple weather texts to be migrated, and respectively extract keywords from the multiple weather texts to be migrated to obtain multiple sample weather keywords; an iterative training subunit, configured to input the multiple sample weather keywords and the multiple target recovery images into a first migration sub-model and a second migration sub-model in an initial weather migration model respectively, and perform iterative training on the first migration sub-model and the second migration sub-model in a manner of model weight fusion to obtain a target weather migration model.

[0013] Optionally, the iterative training subunit described above includes a first iterative training module, configured to perform iterative training on the second migration sub-model by using the multiple sample weather keywords and the multiple target recovery images, and determine multiple second model weights corresponding to the second migration sub-model according to the iterative training process of the second migration sub-model when the iterative training of the second migration sub-model is completed; a fusion update module, configured to perform weight fusion update on multiple first model weights in the first migration sub-model by using the multiple second model weights to obtain an updated first migration sub-model; and a second iterative training module, configured to perform iterative training on the updated first migration sub-model by using the multiple sample weather keywords and the multiple target recovery images.

[0014] According to another aspect of the embodiments of the present application, there is also provided a weather migration device, including: a migration acquisition unit, configured to acquire a target weather migration text and a reference image, where the reference image is an image including geographical location information and preset weather information; and a migration unit, configured to input the target weather migration text and the reference image into a target weather migration model to obtain a target migration image, and the target weather migration model is a model trained by the training method of the weather migration model.

[0015] According to another aspect of the embodiments of the present application, there is also provided a training system for a weather migration model for implementing a training method of a weather migration model, including: a multi-weather image restoration module, configured to obtain multiple images to be restored, as well as weather keywords corresponding to each of the multiple images to be restored, and respectively perform a first restoration operation on the multiple images to be restored by using an image restoration model according to the multiple weather keywords, to obtain multiple first restored images; there are differences in geographical location information and / or weather information among the multiple images to be restored; an artificial correction module, configured to obtain first click information of a target account on each first restored image, and respectively perform a second restoration operation on each first restored image by using a segmentation model according to the first click information of each first restored image, to obtain multiple target restored images; a weather condition generation module, configured to iteratively train an initial weather migration model according to the multiple target restored images and multiple weather texts to be migrated, to obtain a target weather migration model.

[0016] According to another aspect of the embodiments of the present application, there is provided a computer-readable storage medium storing computer instructions for causing a computer to execute the training method of the weather migration model as described above, or the weather migration method.

[0017] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to cause the at least one processor to execute the training method of the weather migration model as described above, or the weather migration method.

[0018] Compared with the prior art, the technical solutions provided by the embodiments of the present application may include the following beneficial effects:

[0019] Through the training method of the weather migration model and the above-mentioned weather migration method, the accuracy from model training to model application is effectively achieved, thereby improving the accuracy of the images obtained by weather migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0021] Figure 1 is a schematic diagram of an optional training system for a weather migration model according to an embodiment of the present invention;

[0022] Figure 2 It is a flowchart of an optional method for training a weather migration model according to an embodiment of the present invention;

[0023] Figure 3 It is a schematic diagram of an optional weather migration method according to an embodiment of the present invention;

[0024] Figure 4 It is a schematic structural diagram of an optional device for training a weather migration model according to an embodiment of the present invention;

[0025] Figure 5 It is a schematic structural diagram of an optional weather migration device according to an embodiment of the present invention;

[0026] Figure 6 It is a schematic structural diagram of an optional electronic device according to an embodiment of the present invention. Detailed implementation manners

[0027] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those clearly listed steps or units, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0029] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0030] To solve the problem of low accuracy in existing weather migration solutions, the embodiments of the present application provide a method for training a weather migration model, a weather migration method, and a device. As an alternative implementation, the above method for training a weather migration model can be applied but is not limited to a training system 100 of a weather migration model as shown in Figure 1 . The present application disassembles the training system of the weather migration model and provides a sub-module capable of obtaining multi-weather paired data (such as the multi-weather image restoration module shown in Figure 1 ), an interactive manual correction sub-module (such as the manual correction module shown in Figure 1 ), and a weather condition generation sub-module based on a diffusion model (such as the weather condition generation module shown in Figure 1 ). The overall framework diagram of the training system of the weather migration model is shown in Figure 1 . The training system 100 of the weather migration model includes a multi-weather image restoration module, a manual correction module, a data center, and a weather condition generation module. The weather image restoration module, the manual correction module, the data center, and the weather condition generation module are connected through a network. The above network can include but is not limited to: wired networks, wireless networks. Among them, the wired network includes: local area networks, metropolitan area networks, and wide area networks. The wireless network includes: Bluetooth, WIFI, and other networks that implement wireless communication.

[0031] According to one aspect of the embodiments of the present invention, the above training system of the weather migration model can also perform the following steps: The multi-weather image restoration module is used to obtain multiple images to be restored and the weather keywords corresponding to each of the multiple images to be restored, and respectively perform a first restoration operation on the multiple images to be restored using an image restoration model according to the multiple weather keywords to obtain multiple first restored images. There are differences in geographical location information and / or weather information among the multiple images to be restored; the manual correction module is used to obtain the first click information of the target account for each first restored image, and respectively perform a second restoration operation on each first restored image using a segmentation model according to the first click information of each first restored image to obtain multiple target restored images; the weather condition generation module is used to iteratively train the initial weather migration model according to the multiple target restored images and multiple weather texts to be migrated to obtain a target weather migration model.

[0032] In the above embodiments of the present invention, by using the above method for training a weather migration model, the difficulty of obtaining training data is reduced. At the same time, the problem of low image accuracy in existing weather migration solutions is solved, and the accuracy of the images obtained by weather migration is improved.

[0033] The above is only an example, and no limitation is made thereto in this embodiment.

[0034] As an alternative implementation, please refer toFigure 2 , which shows a flowchart of a method for training a weather migration model provided by an embodiment of the present application. The execution subject of each step of this method can be a terminal device or a server. In the following method embodiments, for ease of description, only the execution subject of each step is introduced as a "computer device". This method may include at least one of the following steps (S202 to S206):

[0035] S202, obtain multiple images to be restored, and weather keywords corresponding to each of the multiple images to be restored, and there are differences in geographical location information and / or weather information among the multiple images to be restored;

[0036] S204, perform image restoration on the multiple images to be restored according to the multiple weather keywords to obtain multiple target restored images, and each target restored image is an image containing geographical location information and preset weather information, and the weather information is different from the weather indicated by the preset weather information;

[0037] S206, perform iterative training on the initial weather migration model according to the multiple target restored images and multiple weather texts to be migrated to obtain the target weather migration model.

[0038] It should be noted that the image information corresponding to each of the multiple images to be restored in S202 includes geographical location information and weather information, and there are at least differences in the geographical location information or weather information corresponding to the images to be restored among the multiple images to be restored. The weather information corresponding to each image to be restored is different from the following preset weather information, and the preset weather information can be understood, but not limited to, information that does not affect the geographical location information corresponding to the image to be restored. For example, when the preset weather information is included in the image information corresponding to the image, there will be no floating objects such as rain and snow affecting the visual perception of the image, nor will there be an impact on the light perception of the image caused by sunlight, etc.

[0039] The way to obtain the weather keywords can be the weather keywords directly input by the user, or the weather keywords obtained by extracting keywords from the weather description text input by the user, or the weather keywords obtained by identifying the weather information of each image to be restored and extracting keywords from the recognition result. The present application does not limit the way to obtain the weather keywords.

[0040] The image restoration in S204 is used to remove the corresponding weather information in multiple images to be restored, so as to obtain target restored images containing preset weather information and geographical location information. For example, floating objects such as rain and snow in the images to be restored are removed, and obstacles (such as snow accumulation, puddles, etc.) caused by weather information in the corresponding geographical location information in the images to be restored are removed. The weather text to be migrated in S206 can be a description text input by the target user for describing the weather to be migrated. When performing iterative training, the conditions for terminating the iteration can be the number of iterative training times, or the function convergence. It can also be that when the weather information corresponding to the weather text to be migrated is the same as the weather information in the images to be restored corresponding to the target restored images, the image similarity between the iterative migration images output each time during the iterative training process of the initial weather migration model and the images to be restored corresponding to the target restored images is judged. When the image similarity is greater than or equal to a preset image similarity set in advance, the iterative training is terminated. The specific conditions for terminating the iterative training are not limited in this application.

[0041] Through the above implementation manners of this application, by using the above training method of the weather migration model, the accuracy from model training to model application is effectively achieved, thereby improving the accuracy of the images obtained by weather migration.

[0042] As an optional implementation manner, the above-mentioned image restoration of multiple images to be restored according to multiple weather keywords to obtain multiple target restored images includes:

[0043] S1. According to multiple weather keywords, use an image restoration model to perform a first restoration operation on each of the multiple images to be restored respectively to obtain multiple first restored images;

[0044] S2. Obtain the first click information of the target account for each of the first restored images;

[0045] S3. According to the first click information of each of the first restored images, use a segmentation model to perform a second restoration operation on each of the first restored images to obtain multiple target restored images.

[0046] It should be noted that the first restoration operation is used to remove the sky floating objects corresponding to the weather information in the image to be restored according to the weather keyword corresponding to each image to be restored. The operation in S1 can be, but is not limited to, understood as an operation performed by the multi-weather image restoration module included in the training system 100 of the weather migration model as shown in Figure 1 . Thus, according to multiple images to be restored (such as the image to be restored 1, the image to be restored 2... the image to be restored N shown in Figure 1 ) and the weather keyword corresponding to each image to be restored (such as Figure 1The weather keywords shown (weather keyword 1, weather keyword 2... weather keyword N) are used to restore multiple images to be restored into multiple first restored images (such as Figure 1 the first restored image 1, the first restored image 2... the first restored image N shown). The specific operations of S1 include: S1-1, determining any unrecovered image among the multiple images to be restored as the current image to be restored, and determining the weather keyword corresponding to the current image to be restored as the current weather keyword; S1-2, tokenizing the current image to be restored to obtain a token result set; S1-3, according to the current weather keyword, using a first image restoration sub-model to perform a diffusion restoration operation on the token result set to obtain a diffusion restored image, and the diffusion restoration operation is used to remove the floating objects corresponding to the current image to be restored; S1-4, using a second image restoration sub-model to perform an image repair operation on the diffusion restored image to obtain a first restored image, and the image repair operation is used to repair the unreasonable information in the diffusion restored image, and the unreasonable information is formed by the diffusion restoration operation.

[0047] The operations in S1-1 above can be understood, but are not limited to, selecting any unrecovered image from the multiple images to be restored as the current image to be restored, and determining the current weather keyword corresponding to the current image to be restored from multiple weather keywords. The tokenization in S1-2 is to decompose the current image to be restored into a series of discrete symbols or units (tokens), and these tokens can be effectively processed and analyzed by a machine learning model. The diffusion restoration operation in S1-3 can be understood, but is not limited to, removing the sky floating object data corresponding to the current image to be restored included in the token result set; the image repair operation in S1-4 can be understood, but is not limited to, being used to eliminate the abnormal (or unreasonable) data formed by the diffusion restoration operation on the current image to be restored. It can be understood that both the first image restoration sub-model and the second image restoration sub-model are models pre-trained with corresponding sample data. The first image restoration sub-model is used to perform diffusion restoration on the token result set according to the weather keyword, so as to remove the sky floating object data in the token result set indicating the current image to be restored, so as to obtain a diffusion restored image (that is, the image after removing the sky floating objects in the current image to be restored). The second image restoration sub-model is used to perform image rationalization recognition on the diffusion restored image, and in the case of identifying unreasonable regions in the diffusion restored image, perform image repair on the unreasonable regions in the diffusion restored image to repair the unreasonable regions in the diffusion restored image into reasonable regions.

[0048] The operations in S1 above can be, but are not limited to, understood as restoring various abnormal weather images to normal weather images (without floating objects in the air). Specifically, it includes: tokenizing the image, and then restoring the normal weather image through a diffusion model (such as Diffusiontransformer) (i.e., the first image restoration sub-model), and repairing the defects in the restored image. It can be understood that in this application, the task of obtaining paired data is transformed into the task of abnormal weather restoration, and the defects in the restored image are repaired, which reduces the cost of collecting paired data while ensuring the accuracy of the paired data.

[0049] The above image repair operation (repair) can be, but is not limited to, understood as that the diffusion restoration operation may not completely repair the image, because the entire restoration process of the image to be restored (i.e., the process corresponding to the first restoration operation) requires a guidance, but this guidance may cause some interference, and the image repair operation performed by the second image restoration sub-model is to specifically solve this problem existing after the diffusion restoration operation by the first image restoration sub-model. Specific image repair operations are as follows: for example, if the current image to be restored is a snowing image, then for the restoration of the snowing image, some parts where there are snowflakes are restored into a black hole (i.e., some unreasonable areas) during the diffusion restoration process, and the image repair operation is to repair this hole.

[0050] For example: after using the first image restoration sub-model to perform the diffusion restoration operation to remove the floating objects in the sky in the image to be restored, the second restoration sub-model identifies a black hole (which can be, but is not limited to, understood as a pure black area in the diffusion restoration image) in the diffusion restoration image caused by the diffusion restoration operation of the first restoration sub-model, then the second restoration sub-model will restore the black hole to a reasonable image.

[0051] For another example, assume that the geographical location information included in the current image to be restored is sky, office building, road, etc., and the weather information included in the current image to be restored is rainy. Then, the first image restoration operation will remove the floating objects in the sky (i.e., "raindrops") in the current image to be restored. During the removal process, assume that the operation of removing the floating object area in the sky removes the floating objects in the sky with the office building as the background, forming a pure black area, so that the original background behind the floating objects in the sky cannot be correctly displayed. Then, the second image restoration operation is to restore the pure black area to the original office building area. That is, the image restoration operation performed by the second image restoration sub-model is to restore the unreasonable area in the diffusion restored image to the actual background of the corresponding area in the current image to be restored. The specific image restoration operation performed inside the first image restoration sub-model can be that the first image restoration sub-model performs background recognition on the current image to be restored, so as to determine the reasonable background area corresponding to the unreasonable area in the current image to be restored, and then restore the unreasonable area in the diffusion restored image to the reasonable background area. The image restoration operation can also be that the first restoration sub-model automatically defines the reasonable area corresponding to the abnormal area in the diffusion restored image according to the self-learning ability of the model during the model training process (which can but is not limited to being understood as that there are a large number of rationalized images in the training samples when training the first restoration sub-model), combined with other non-abnormal areas in the image to be restored, and then performs image restoration on the abnormal area in the diffusion restored image according to the defined reasonable area.

[0052] Through the above implementation manners of the present application, by adopting the first restoration operation, not only can the corresponding floating objects in the sky in the image to be restored be removed, but also the consistency of the geographical location information included in the image after removing the floating objects in the sky and the geographical location information included in the image to be restored can be ensured, thereby ensuring the accuracy of the first restored image obtained by performing the first restoration operation. Moreover, the first restoration operation performed on the image to be restored in the present application only uses one multi-weather image restoration module, rather than using different modules for restoration for different weathers. In this way, there is no need for too many weights, and the execution process of different image restorations is simplified, achieving the effect of simplifying the training system of the weather migration model through the multi-weather image restoration module in the present application.

[0053] As an optional implementation manner, the above-mentioned second restoration operation is performed on each first restored image by using the segmentation model according to the first click information of each first restored image to obtain multiple target restored images, including:

[0054] S1, determining the position information corresponding to the first click operation according to the first click information, and generating a heat map area according to the position information, where the heat map area is used to indicate the correlation around the position information;

[0055] S2, determining the segmentation type corresponding to the first click operation according to the heat map area;

[0056] S3, determining the information to be segmented in the first restored image according to the heat map area and the segmentation type;

[0057] S4, segmenting the first restored image according to the information to be segmented to obtain multiple target restored images.

[0058] The operations in S1 to S4 above can be understood as, but not limited to, Figure 1 The artificial correction module shown is executed. The image restored from snowy days, rainy days and other weather conditions may contain residual snow, puddles and other areas of varying sizes. The artificial correction module automatically segments and removes the residual objects by clicking on them, or manually selects the area. This step is to divide the weather categories into detailed categories (for example, snowy days are divided into snow after snowing, snow during snowing, no snow during snowing, and snowflakes in the air during snowing. Snow accumulation means adding a layer of snow to objects with roofs, which increases the richness of weather condition generation). The first click information in the above S1 can be understood as, but is not limited to, the target account's operation on the first restored image. Figure 1 The click information corresponding to the first click operation is shown. The location information is the location information of the mouse click in the first restored image corresponding to the first click operation. The heat map area can be understood, but is not limited to, as an area with the same features as the location information. For example, when the target user clicks on snow or a puddle in the first restored image, the entire snow or puddle area corresponding to the click location (i.e., the aforementioned heat map area) can be determined based on the target user's click.

[0059] The segmentation type in S2 can be understood as, but not limited to, the type corresponding to the heat map area, such as snow, puddles, and other types of objects to be removed. The information to be segmented can be determined based on the heat map area and the segmentation type. The information to be segmented can be understood as, but not limited to, the type corresponding to the heat map area. Figure 1 The segmentation guidance information of the segmentation model shown is then used to segment all snow or puddles in the first restored image based on the segmentation information, thereby obtaining an image without snow or puddles (the final target restored image). The information to be segmented in S3 includes information such as size, shape, and color. The first click operation is used to indicate the segmentation guidance of the segmentation model.

[0060] At the same time, if Figure 1The training system 100 of the weather migration model shown also integrates a data center. After obtaining multiple target restored images, the multiple target restored images and weather keywords (and / or weather description texts input by the target user) are stored in the data center. When iteratively training the initial weather migration model in the weather condition generation module, the target restored images stored in the data center are used, so as to facilitate comparing the images generated in each iteration with the images to be restored corresponding to the target restored images during the iterative training process. That is, when training the initial weather migration model in the present application, paired images are used (that is, the geographical location information included in the images to be restored and the target restored images is exactly the same, while the weather information is different).

[0061] The operations in S1 to S4 above can be, but are not limited to, understood as manually clicking on snow, puddles, etc. in the first restored image. However, these cannot be repaired and solved by the second image restoration sub-model. Therefore, in the present application, it is removed by the way of manual click guidance. However, the way of manual click guidance does not require manually selecting a region in the image, but only needs to click on a certain point. For example, if the point is on a snow accumulation, then the features in the first restored image that are the same as the click features such as color and shape at the clicked position will be segmented and removed.

[0062] Through the above implementation manner of the present application, the second restoration operation can further remove the information to be removed corresponding to the weather information included in the image to be restored, further improving the accuracy of the target restored image. At the same time, combining the first restoration operation and the second restoration operation can completely and effectively remove all the weather information included in the image to be restored without affecting the geographical location information included in the image to be restored, effectively realizing the complete decoupling of the geographical location information and the weather information included in the image to be restored, and reducing the influence of the weather information on the geographical location information. The problem of the accuracy of the training samples corresponding to the initial weather migration model caused by the influence of the weather information on the geographical location information is eliminated, thereby avoiding the problem of low training accuracy of the initial weather migration model caused by the influence of the weather information on the geographical location information. At the same time, it also avoids the defects in image details caused by the complete decoupling of the geographical location information and the weather information.

[0063] As an optional implementation manner, the iterative training of the initial weather migration model according to multiple target restored images and the weather text to be migrated to obtain the target weather migration model includes:

[0064] S1, obtaining multiple weather texts to be migrated, and respectively extracting keywords from the multiple weather texts to be migrated to obtain multiple sample weather keywords;

[0065] S2. Input multiple sample weather keywords and multiple target restored images into the first migration sub-model and the second migration sub-model in the initial weather migration model respectively, and iteratively train the first migration sub-model and the second migration sub-model by means of model weight fusion to obtain the target weather migration model.

[0066] With a large amount of paired data in the data center, the distribution of a certain weather in the real world under different scenarios can be obtained, the concept of a certain weather can be learned, and then the migration of the weather can be learned. Input the source image and the description of the weather to be migrated (for example, snow days are divided into having snow accumulation after snowing, having snow accumulation during snowing, having no snow accumulation during snowing, having snowflakes in the air during snowing, and having snow accumulation means adding a layer of snow to objects with a roof), and output the image through an image encoder, a text encoder, a diffusive model, and an image decoder.

[0067] It should be noted that the above first migration sub-model can be but is not limited to diffusive gengration, and the second migration sub-model can be but is not limited to diffusive controlnet. The above multiple weather texts to be migrated can be new texts input by the target user for describing the weather to be migrated, or weather keywords stored in the data center as shown in Figure 1 , or weather description texts stored in the data center. After obtaining multiple weather texts to be migrated, when the weather text to be migrated is a weather keyword stored in the data center, the weather keyword is determined as the sample weather keyword. When the weather text to be migrated is a weather description text stored in the data center, the weather keyword corresponding to the weather description text stored in the data center is determined as the sample weather keyword, thus avoiding repeated keyword extraction operations. When the weather text to be migrated is a new text input by the target user for describing the weather to be migrated, the operations in S1 above are performed to obtain multiple sample weather keywords.

[0068] It should be noted that after obtaining multiple weather texts to be migrated, the text encoder in the weather condition generation module is also used to perform text encoding on the multiple weather texts to be migrated. After obtaining multiple target restored images, the image encoder is used to perform image encoding on the multiple target restored images. Then, the first migration sub-model and the second migration sub-model are iteratively trained according to the text encoding results and the image encoding results respectively.

[0069] The model weight fusion in S2 above can be understood, but is not limited to, fusing multiple varying weights (each layer in the second transfer sub-model corresponds to one weight) obtained in each iteration during the iterative training of the second transfer sub-model with multiple weights in the first transfer sub-model. The operations in S2 specifically include: S2-1, iteratively training the second transfer sub-model using multiple sample weather keywords and multiple target restoration images, and when the iterative training of the second transfer sub-model is completed, determining multiple second model weights corresponding to the second transfer sub-model according to the iterative training process of the second transfer sub-model; S2-2, performing weight fusion update on multiple first model weights in the first transfer sub-model using multiple second model weights to obtain an updated first transfer sub-model; S2-3, iteratively training the updated first transfer sub-model using multiple sample weather keywords and multiple target restoration images.

[0070] After the iterative training of the second transfer sub-model in S2-1 above is completed, multiple varying weights corresponding to the second transfer sub-model (i.e., multiple second model weights) will be obtained. The second model weights are used to indicate the changes that occur to the preset weights corresponding to the second transfer sub-model during the iterative training when the second transfer sub-model is iteratively trained. The multiple first model weights in S2-2 can be understood, but are not limited to, the first initial weights of the first transfer sub-model obtained by training the initial sub-model corresponding to the first transfer sub-model using other training tasks and training samples (tasks and samples different from multiple sample weather keywords, multiple target restoration images, and the weather transfer training task). The ways of weight fusion update include layer-by-layer replacement or weighted average. The layer-by-layer replacement way can be understood, but is not limited to, replacing the second model weights of the corresponding layer in the second transfer sub-model with the first model weights of the corresponding layer in the first transfer sub-model; the weighted average way is to perform weighted average on the weights of the same layer of the first transfer sub-model and the second transfer sub-model, that is, performing weighted average on the second model weights of the corresponding layer in the second transfer sub-model and the first model weights of the corresponding layer in the first transfer sub-model, and taking the weighted average obtained weight as the second initial weight of the first transfer sub-model before iterative training.

[0071] It should be noted that after the first transfer sub-model outputs the transfer result, it is necessary to use an image decoder to decode the transfer result to obtain the final weather transfer image, and then complete the training of the first transfer sub-model through iterative termination conditions such as restricting the number of iterations of the first transfer sub-model or the judgment condition of whether the weather transfer image is qualified. After training the first transfer sub-model, the target weather transfer model corresponding to the weather condition generation module can be obtained, including a text encoder, an image encoder, a first transfer sub-model, a second transfer sub-model, and an image decoder.

[0072] It can be understood thatFigure 1 The images and texts in the weather condition generation module shown are arbitrarily input, and the data in the data center can be used to train the initial weather migration model. During the training process, the input image of the initial weather migration model is the corrected image (such as Figure 1 multiple target restoration images in the weather condition generation module shown), and the output is the original image input in the multi-weather image restoration module (such as Figure 1 multiple images to be restored in the multi-weather image restoration module shown). Then, it can be determined whether the initial weather migration model is trained completed according to the consistency between the image output by the weather condition generation module and the original image input in the multi-weather image restoration module.

[0073] During the training process of the initial weather migration model, the second migration sub-model can be, but is not limited to, understood as a black box. It can output a change weight at each layer, and then give the change of the weight of each layer of the second migration sub-model (the weight of the second model) to the first migration sub-model. The first migration sub-model and the second migration sub-model can be, but is not limited to, understood as an overall model. The structures of the two sub-models are the same, only the weights are different. The first migration sub-model is pre-trained through other tasks, and the weight of the second migration sub-model will change preferentially. Then, during the iterative training process, when backpropagating, the changed part of the second migration sub-model will be fused into the unchanged weight of the first migration sub-model, and the second migration sub-model is the main part that needs to train and modify the weight.

[0074] It can be understood that the text encoder and image encoder in this application also need to be iteratively trained, but the specific training methods of the encoder and decoder can adopt existing training methods, which are not limited in this application.

[0075] Through the above implementation manners of this application, model training is carried out by means of model weight fusion. Not only can the model better adapt to the changes and non-stationary characteristics of the data by fusing the change weights, improving the response ability to new samples and new tasks, but also the way of weight fusion can help smooth the gradient in the loss function, making the training process of the model more stable, thereby reducing the problems of oscillation and low convergence speed, and can also reduce the overall number of parameters to be optimized, achieving more efficient model training.

[0076] As Figure 3 shown, this application also provides a weather migration method, including:

[0077] S302, obtaining a target weather migration text and a reference image, where the reference image is an image containing geographical location information and preset weather information;

[0078] S304. Input the target weather migration text and the reference image into the target weather migration model to obtain the target migration image. The target weather migration model is a model trained by the training method of the weather migration model.

[0079] The target weather migration text is, for example, a detailed description text of the weather to be migrated such as snow days being divided into having snow accumulation after snowing, having snow accumulation during snowing, having no snow accumulation during snowing, and having snowflakes in the air during snowing. Having snow accumulation means adding a layer of snow to objects with a top. After training the target weather migration model through the training method of the weather migration model, when actually performing weather migration, the obtained target weather migration text and reference image can be directly input into the target weather migration model to obtain the target migration image. Similarly, the target weather migration text can be the text input by the target user for describing the weather information to be migrated, and the reference image can be the target restored image stored in the data center, or an image uploaded by the target user that is different from the multiple target restored images stored in the data center but contains geographical location information and preset weather information. The preset weather information is an image that will not cause visual and light perception effects on the geographical location information included in the reference image.

[0080] Through the above implementation manner of the present application, the target migration image obtained by using the training method of the weather migration model not only conforms to the actual physical laws, but also does not require collecting too much training data. Moreover, paired images are used in the training method of the weather migration model, which makes the training of the weather migration model more accurate, further ensuring the accuracy of the target migration image when using the target weather migration model to generate the target migration image.

[0081] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0082] According to another aspect of the embodiments of the present invention, there is also provided a training device for the weather migration model for implementing the above training method of the weather migration model, as Figure 4 shown. The device includes:

[0083] A training and acquisition unit 402, configured to acquire multiple images to be restored, and weather keywords corresponding to each of the multiple images to be restored. There are differences in geographical location information and / or weather information among the multiple images to be restored;

[0084] A restoration unit 404, configured to perform image restoration on multiple images to be restored according to multiple weather keywords, so as to obtain multiple target restored images, where each target restored image is an image including geographical location information and preset weather information, and the weather information is different from the weather indicated by the preset weather information;

[0085] A training unit 406, configured to iteratively train an initial weather migration model according to multiple target restored images and multiple weather texts to be migrated, so as to obtain a target weather migration model.

[0086] According to another aspect of the embodiments of the present invention, there is also provided a weather migration device for implementing the above weather migration method, as Figure 5 shown. The device includes:

[0087] A migration acquisition unit 502, configured to acquire a target weather migration text and a reference image, where the reference image is an image including geographical location information and preset weather information;

[0088] A migration unit 504, configured to input the target weather migration text and the reference image into the target weather migration model to obtain a target migration image, where the target weather migration model is a model trained by the training method of the weather migration model.

[0089] The specific manners of the operations executed by the units in the above device embodiments have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0090] According to yet another aspect of the embodiments of the present invention, there is also provided an electronic device for implementing the above training method of the weather migration model or the weather migration method. The electronic device may be a Figure 6 terminal device or a server as shown. In this embodiment, the electronic device is taken as an example of a terminal device for illustration. As Figure 6 shown, the electronic device includes: at least one processor 604; and a memory 602 communicatively connected to the at least one processor 604; where the memory 602 stores a computer program executable by the at least one processor 604, and the computer program is executed by the at least one processor 604, so that the at least one processor 604 executes the steps in any one of the above method embodiments.

[0091] Optionally, in this embodiment, the above electronic device may be at least one network device among multiple network devices in a computer network.

[0092] Optionally, in this embodiment, the above processor may be configured to execute each step in the above training method of the weather migration model or the weather migration method through a computer program.

[0093] Optionally, those of ordinary skill in the art can understand thatFigure 6 The structure shown is only schematic. The electronic device can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a personal digital assistant, and terminal devices such as Mobile Internet Devices (MID), PAD, etc. Figure 6 It does not limit the structure of the above-mentioned electronic devices. For example, the electronic device may further include more or fewer components (such as a network interface, etc.) than those shown Figure 6 in the figure, or have a different configuration from that shown Figure 6 in the figure.

[0094] Among them, the memory 602 can be used to store software programs and modules, such as the program instructions / modules corresponding to the training method of the weather migration model, the weather migration method and the device in the embodiments of the present invention. The processor 604 executes various functional applications and data processing by running the software programs and modules stored in the memory 602, that is, implements the above-mentioned training method of the weather migration model or the weather migration method. The memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 602 may further include a memory remotely provided with respect to the processor 604, and these remote memories can be connected to the terminal through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof. Among them, the memory 602 can specifically but not limitedly be used to store various data information involved in the present application. As an example, as Figure 6 shown, the above-mentioned memory 602 may but is not limited to include the training acquisition unit 402, the recovery unit 404, and the training unit 406 in the training device of the above-mentioned weather migration model, or the migration acquisition unit 502 and the migration unit 504 in the above-mentioned weather migration device. In addition, it may also include but is not limited to other module units in the training device of the above-mentioned weather migration model or the weather migration device, which will not be elaborated in this example.

[0095] Optionally, the above-mentioned transmission device 606 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired network and a wireless network. In one instance, the transmission device 606 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router through a network cable so as to communicate with the Internet or a local area network. In one instance, the transmission device 606 is a Radio Frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0096] In addition, the above electronic device further includes: a display 608, and a connection bus 610 for connecting each module component in the above electronic device.

[0097] In other embodiments, the above terminal device or server may be a node in a distributed system. Among them, the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes in a form of network communication. Among them, the nodes may form a peer-to-peer (P2P) network, and any form of computing device, such as an electronic device like a server or a terminal, can become a node in the blockchain system by joining the peer-to-peer network.

[0098] According to one aspect of the present application, there is provided a computer program product, which includes computer programs / instructions, and the computer programs / instructions contain program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit, it executes various functions provided by the embodiments of the present application.

[0099] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0100] According to one aspect of the present application, there is provided a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above training method of the weather migration model or the weather migration method. In this embodiment, the above computer-readable storage medium may be set to store a computer program for executing each step in the above training method of the weather migration model or the weather migration method.

[0101] Those skilled in the art can understand that to implement all or part of the processes in the above method embodiments, it can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium, and when the program is executed, it may include the processes of the above method embodiments. Among them, the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FM), a hard disk drive (HDD), or a solid-state drive (SSD), etc.; the storage medium may also include a combination of the above types of memories.

[0102] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage media. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing one or more computer devices (which can be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the above methods in various embodiments of the present invention.

[0103] In the above embodiments of the present invention, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0104] In the several embodiments provided by the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the above division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of units or modules can be in electrical or other forms.

[0105] The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0106] In addition, the functional units in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0107] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A training method for a weather migration model, characterized in that, Including: Obtain multiple images to be restored, as well as weather keywords corresponding to each of the multiple images to be restored. There are differences in geographical location information and / or weather information among the multiple images to be restored. Perform image restoration on the multiple images to be restored according to the multiple weather keywords to obtain multiple target restored images. Each target restored image is an image containing geographical location information and preset weather information, and the weather information is different from the weather indicated by the preset weather information. Iteratively train the initial weather migration model according to the multiple target restored images and multiple weather texts to be migrated to obtain the target weather migration model.

2. The method according to claim 1, wherein Perform image restoration on the multiple images to be restored according to the multiple weather keywords to obtain multiple target restored images, including: Perform a first restoration operation on the multiple images to be restored respectively according to the multiple weather keywords using an image restoration model to obtain multiple first restored images. Obtain the first click information of the target account for each of the first restored images. Perform a second restoration operation on each of the first restored images according to the first click information of each first restored image using a segmentation model to obtain the multiple target restored images.

3. The method according to claim 2, wherein Perform a first restoration operation on the multiple images to be restored respectively according to the multiple weather keywords using an image restoration model to obtain multiple first restored images, including: Determine any unrecovered image among the multiple images to be restored as the current image to be restored, and determine the weather keyword corresponding to the current image to be restored as the current weather keyword. Perform tokenization on the current image to be restored to obtain a token result set. Perform a diffusion restoration operation on the token result set according to the current weather keyword using a first image restoration sub-model to obtain a diffusion restored image. The diffusion restoration operation is used to remove floating objects corresponding to the current image to be restored. Perform an image repair operation on the diffusion restored image using a second image restoration sub-model to obtain the first restored image. The image repair operation is used to repair unreasonable information in the diffusion restored image, and the unreasonable information is formed by the diffusion restoration operation.

4. The method according to claim 2, wherein Perform a second restoration operation on each of the first restored images according to the first click information of each first restored image using a segmentation model to obtain the multiple target restored images, including: Determine the position information corresponding to the first click operation according to the first click information, and generate a heat map area according to the position information, where the heat map area is used to indicate the relevance around the position information. Determine the segmentation type corresponding to the first click operation according to the heat map area. Determine the information to be segmented in the first restored image according to the heat map area and the segmentation type. Segment the first restored image according to the information to be segmented to obtain the multiple target restored images.

5. The method according to claim 1, wherein Iteratively train the initial weather migration model according to the multiple target restored images and weather texts to be migrated to obtain the target weather migration model, including: Obtain multiple pieces of the weather text to be migrated, and respectively extract keywords from the multiple pieces of the weather text to be migrated to obtain multiple sample weather keywords; Input the multiple sample weather keywords and the multiple target restored images into the first migration sub-model and the second migration sub-model in the initial weather migration model respectively, and iteratively train the first migration sub-model and the second migration sub-model by means of model weight fusion to obtain the target weather migration model.

6. The method according to claim 5, wherein Iteratively training the first migration sub-model and the second migration sub-model by means of model weight fusion includes: Iteratively train the second migration sub-model with the multiple sample weather keywords and the multiple target restored images, and when the iterative training of the second migration sub-model is completed, determine multiple second model weights corresponding to the second migration sub-model according to the iterative training process of the second migration sub-model; Use the multiple second model weights to perform weight fusion update on the multiple first model weights in the first migration sub-model to obtain an updated first migration sub-model; Iteratively train the updated first migration sub-model with the multiple sample weather keywords and the multiple target restored images.

7. A weather migration method, characterized in that, including: Obtain a target weather migration text and a reference image, where the reference image is an image containing geographical location information and preset weather information; Input the target weather migration text and the reference image into the target weather migration model to obtain a target migration image, and the target weather migration model is a model trained by the training method according to any one of claims 1 to 6.

8. A training device for a weather migration model, characterized in that, including: A training acquisition unit for acquiring multiple images to be restored, and weather keywords corresponding to the multiple images to be restored respectively, and there are differences in geographical location information and / or weather information among the multiple images to be restored; A restoration unit for restoring the multiple images to be restored according to the multiple weather keywords to obtain multiple target restored images, and each target restored image is an image containing geographical location information and preset weather information, and the weather indicated by the weather information and the preset weather information is different; A training unit for iteratively training the initial weather migration model according to the multiple target restored images and multiple weather texts to be migrated to obtain a target weather migration model.

9. A weather migration device, characterized in that, including: A migration acquisition unit for acquiring a target weather migration text and a reference image, where the reference image is an image containing geographical location information and preset weather information; A migration unit for inputting the target weather migration text and the reference image into the target weather migration model to obtain a target migration image, and the target weather migration model is a model trained by the training method according to any one of claims 1 to 6.

10. A training system for a weather migration model for implementing the training method of the weather migration model according to any one of claims 1 to 6, characterized in that, including: A multi-weather image restoration module, which is used to obtain multiple images to be restored, as well as weather keywords corresponding to each of the multiple images to be restored, and respectively perform a first restoration operation on the multiple images to be restored by using an image restoration model according to the multiple weather keywords, and there are differences in geographical location information and / or weather information among the multiple images to be restored; An artificial correction module, which is used to obtain the first click information of the target account on each of the first restored images, and respectively perform a second restoration operation on each of the first restored images by using a segmentation model according to the first click information of each of the first restored images, so as to obtain multiple target restored images; A weather condition generation module, which is used to iteratively train an initial weather migration model according to the multiple target restored images and multiple weather texts to be migrated, so as to obtain a target weather migration model.

Images