Image data processing method, device and storage medium

Abstract

Embodiments of the present application provide an image data processing method, device and storage medium, the image conversion method comprising: acquiring a sequence of collected images corresponding to a first time period, and inputting the sequence of collected images into a target generation network to output a first sequence of predicted images corresponding to a second time period from the target generation network. Finally, according to the first sequence of predicted images generated by the target generation network, a prediction result corresponding to the second time period is determined. Wherein, for the training of the initial generation network, the difference between the discrimination result corresponding to the second sequence of predicted images generated by the initial generation network and the preset reference result can reflect the authenticity of the whole sequence of predicted images, and the similarity between the respective feature maps of the second sequence of predicted images and the reference sequence of images can reflect the authenticity of the details of the image sequence. The initial generation network is trained, thereby ensuring the image generation capability of the target generation network, and further improving the accuracy of the prediction result.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to an image data processing method, apparatus, and storage medium. Background Technology

[0002] In various scenarios, predictions are often required, with weather forecasting being the most common example in daily life. A common forecasting method involves first generating weather images for multiple consecutive moments based on collected meteorological images, and then using these images to make a weather forecast. These meteorological images can include radar echo composite images or satellite cloud images, among others. In the field of autonomous driving, for instance, images of the vehicle taken a few minutes earlier can be used to generate images for the next few minutes, thus predicting the vehicle's driving status.

[0003] However, regardless of the scenario, in reality, the prediction results are often inaccurate. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an image data processing method, apparatus, and storage medium to improve the accuracy of prediction results.

[0005] In a first aspect, embodiments of the present invention provide an image data processing method, comprising:

[0006] Obtain the image sequence corresponding to the first time period;

[0007] The acquired image sequence is input into the target generation network, so that the target generation network outputs a first predicted image sequence corresponding to the second time period, wherein the first time period is earlier than the second time period;

[0008] Based on the first predicted image sequence, determine the prediction result corresponding to the second time period;

[0009] Specifically, the initial generation network is trained based on the difference between the discrimination result corresponding to the second predicted image sequence output by the initial generation network and the preset reference result, as well as the similarity between the feature maps of the reference image sequence and the second predicted image sequence, to obtain the target generation network.

[0010] Secondly, embodiments of the present invention provide an image data processing method, including:

[0011] Obtain the sequence of meteorological images corresponding to the first time period;

[0012] The acquired meteorological image sequence is input into the target generation network, so that the target generation network outputs a first predicted meteorological image sequence corresponding to the second time period, wherein the first time period is earlier than the second time period;

[0013] Based on the first predicted meteorological image sequence, determine the weather forecast result corresponding to the second time period;

[0014] Specifically, the initial generation network is trained based on the difference between the discrimination result corresponding to the second predicted meteorological image sequence output by the initial generation network and the preset reference result, as well as the similarity between the feature maps of the reference meteorological image sequence and the second meteorological predicted image sequence, to obtain the target generation network.

[0015] Thirdly, embodiments of the present invention provide an image data processing method, including:

[0016] Obtain the sequence of meteorological images corresponding to the first time period;

[0017] The acquired meteorological image sequence is input into the target generation network, so that the target generation network outputs a first predicted meteorological image sequence corresponding to the second time period, wherein the first time period is earlier than the second time period;

[0018] Based on the first predicted meteorological image sequence, determine the weather forecast result corresponding to the second time period;

[0019] Based on the weather forecast results, reminder information is generated to guide the planting of agricultural crops;

[0020] Specifically, the initial generation network is trained based on the difference between the discrimination result corresponding to the second predicted meteorological image sequence output by the initial generation network and the preset reference result, as well as the similarity between the feature maps of the reference meteorological image sequence and the second meteorological predicted image sequence, to obtain the target generation network.

[0021] Fourthly, embodiments of the present invention provide an electronic device, including a processor and a memory, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the image data processing methods described in the first to third aspects. The electronic device may also include a communication interface for communicating with other devices or communication networks.

[0022] Fifthly, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, enables the processor to at least implement the image data processing method as described in the first to third aspects.

[0023] The image data processing method provided in this embodiment of the invention acquires a sequence of images corresponding to a first time period, and inputs this sequence of images into a target generation network in an adversarial generative network. The target generation network then outputs a first predicted image sequence corresponding to a second time period, where the first time period is earlier than the second time period. Finally, based on the first predicted image sequence generated by the target generation network, the prediction result corresponding to the second time period is determined. Specifically, the initial generation network can be trained based on the difference between the discrimination result corresponding to the second predicted image sequence generated by the initial generation network and a preset reference result, as well as the similarity between the feature maps of the second predicted image sequence and the reference image sequence, to obtain the aforementioned target generation network.

[0024] Since the difference between the discrimination result and the reference result reflects the overall realism of the predicted image sequence, and the similarity between feature maps reflects the realism of the predicted image sequence from the perspectives of texture and semantics, the target generation network trained in the above manner considers both the overall image sequence and the details of each image in the sequence during the generation of the predicted image sequence. This ensures that the image sequence generated by the target generation network possesses a certain degree of realism, both overall and in terms of texture and semantics of each individual image, thus guaranteeing the image generation capability of the target generation network. Since the quality of image generation capability is directly reflected in the clarity of the generated image, the target generation network trained in the above manner can generate clear image sequences, resulting in better visualization effects and a higher level of detail. Furthermore, the relatively clear image sequences generated by the target generation network further improve the accuracy of the prediction results. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart of an image data processing method provided in an embodiment of the present invention;

[0027] Figure 2 for Figure 1 The illustrated embodiment provides a display style for a prediction result corresponding to the image data processing method.

[0028] Figure 3 for Figure 1 The illustrated embodiment provides an alternative display style for the prediction result corresponding to the image data processing method.

[0029] Figure 4 A flowchart of another image data processing method provided in an embodiment of the present invention;

[0030] Figure 5 A flowchart illustrating a training method for a generative adversarial network provided in an embodiment of the present invention;

[0031] Figure 6 This is a schematic diagram illustrating the training process of an adversarial generative network according to an embodiment of the present invention.

[0032] Figure 7 A schematic diagram illustrating another adversarial generative network training process provided in an embodiment of the present invention;

[0033] Figure 8 A flowchart illustrating another method for training an adversarial generative network provided in an embodiment of the present invention;

[0034] Figure 9 A schematic diagram illustrating another adversarial generative network training process provided in an embodiment of the present invention;

[0035] Figure 10 A schematic diagram illustrating another adversarial generative network training process provided in an embodiment of the present invention;

[0036] Figure 11 A flowchart illustrating the training process of another adversarial generative network provided in an embodiment of the present invention;

[0037] Figure 12 A schematic diagram illustrating another training process for an adversarial generative network provided in an embodiment of the present invention;

[0038] Figure 13 This is a schematic diagram of the structure of an image data processing device provided in an embodiment of the present invention;

[0039] Figure 14 To and Figure 13 A schematic diagram of the electronic device corresponding to the image data processing device provided in the embodiment shown;

[0040] Figure 15 This is a schematic diagram of another image data processing device provided in an embodiment of the present invention;

[0041] Figure 16 To and Figure 15 The illustrated embodiment provides a schematic diagram of the electronic device corresponding to the image data processing apparatus. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one.

[0044] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0045] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to identification.” Similarly, depending on the context, the phrases “if determination” or “if identification (of the condition or event of the statement)” can be interpreted as “when determination” or “in response to determination” or “when identification (of the condition or event of the statement)” or “in response to identification (of the condition or event of the statement).”

[0046] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.

[0047] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Where there is no conflict between the embodiments, the following embodiments and features can be combined with each other. Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0048] Figure 1This is a flowchart illustrating an image data processing method provided in an embodiment of the present invention. This image data processing method can be executed by a processing device. It is understood that the processing device can be implemented as software, or a combination of software and hardware. In this embodiment, the processing device can specifically be a server. Figure 1 As shown, the method includes the following steps:

[0049] S101, acquire the image sequence corresponding to the first time period.

[0050] S102, the acquired image sequence is input into the target generation network in the adversarial generative network, so that the target generation network outputs a first predicted image sequence corresponding to the second time period, the first time period being earlier than the second time period.

[0051] S103, Based on the first predicted image sequence, determine the prediction result corresponding to the second time period.

[0052] First, the processing device acquires the sequence of images collected within the first time period. Optionally, the processing device can directly obtain the image sequence from the image acquisition device. Furthermore, the specific form of the acquired image sequence varies in different scenarios. For example, in a weather forecast scenario, the acquired image sequence can specifically represent meteorological images for a certain time period. These meteorological images can further include radar echo sequences or satellite cloud image sequences. As another example, in an autonomous driving scenario, the acquired image sequence can include images of the vehicle driving within the first M minutes.

[0053] Next, the processing device can input the acquired image sequence into the target generation network in the adversarial generative network, so that the target generation network can output a first predicted image sequence, which corresponds to a second time period earlier than the first time period. Here, the target generation network is obtained by training the initial generation network, and at this time the target generation network has a better image generation capability, that is, the target generation network can generate image sequences with good visualization effect and clarity.

[0054] Finally, the processing device can determine the prediction result corresponding to the second time period based on the output first predicted image sequence. Optionally, the first predicted image sequence can be input into a prediction network, so that the prediction network outputs the prediction result corresponding to the second time period. The prediction network can be a network independent of the generative adversarial network, but both can be deployed in the processing device.

[0055] Optionally, the confidence level of the prediction result can be obtained simultaneously with the prediction result. Optionally, the display style of the prediction result and its confidence level can also be determined based on the second time period corresponding to the first predicted image sequence and the prediction result. The prediction result and its confidence level can be directly displayed on the processing device.

[0056] As described above, the quality of the first predicted image sequence generated by the target generation network directly affects the accuracy of the prediction result. Therefore, to improve the image generation capability of the target generation network, the initial generation network can be trained using the discriminative network in the adversarial generation network to obtain the target generation network.

[0057] During training, a real-world reference image sequence and a second predicted image sequence generated by the initial generator network are first obtained. The reference image sequence and the second predicted image sequence correspond to the same time period, which is earlier than the first time period. Then, based on the difference between the discrimination result of the discrimination network output for the second predicted image sequence and the preset reference result, as well as the similarity between the feature maps of the reference image sequence and the second predicted image sequence, the network parameters of the initial generator network are adjusted to finally obtain the target generator network.

[0058] In practice, the initial generator network often needs to be trained multiple times, meaning that the network parameters need to be adjusted multiple times to achieve convergence, which is to obtain the target generator network. It should be noted that before the generator network converges, each time the network parameters are adjusted, the resulting generator network can be considered as the initial generator network in the various embodiments of this invention.

[0059] In this embodiment, the processing device acquires an image sequence corresponding to a first time period and inputs this image sequence into the target generation network in the adversarial generative network, so that the target generation network outputs a first predicted image sequence corresponding to a second time period. Finally, based on the first predicted image sequence generated by the target generation network, the prediction result corresponding to the second time period is determined. Specifically, the initial generation network can be trained based on the difference between the discrimination result corresponding to the second predicted image sequence generated by the initial generation network and a preset reference result, as well as the similarity between the feature maps of the second predicted image sequence and the reference image sequence, to obtain the aforementioned target generation network.

[0060] Since the difference between the discrimination result and the reference result reflects the overall realism of the predicted image sequence, and the similarity between feature maps reflects the realism of the predicted image sequence from the perspective of texture and semantics of each image in the sequence, the target generation network trained in the above manner considers both the overall image sequence and the details of each image in the sequence when generating the predicted image sequence. Therefore, the image sequence generated by the target generation network has a certain degree of realism, both overall and in terms of texture and semantics of each image, ensuring the image generation capability of the target generation network and enabling it to generate clear image sequences. Based on these clear image sequences, more accurate prediction results can then be obtained.

[0061] Optionally, the captured image sequence can also be stored on the user's terminal device. In this case, a remote processing device can obtain the captured image sequence uploaded by the user through the terminal device via a communication connection. Then, in Figure 1 Similar to the illustrated embodiment, the processing device can obtain a first predicted image sequence and further obtain the corresponding prediction result and its confidence level. At this time, the processing device can also feed back the prediction result and confidence level to the user's terminal device. The display style of the prediction result and its confidence level can also be found in the relevant description in the above embodiments. This part can be combined with... Figure 2 Understood. Furthermore, the image generation capabilities of the trained target generation network can also be made available to other developers through an interface.

[0062] In practice, Figure 1 The image data processing method provided in the illustrated embodiment can be specifically used for weather forecasting, and the implementation process may include the following steps:

[0063] First, acquire the sequence of meteorological images corresponding to the first time period.

[0064] Second, the collected meteorological image sequence is input into the target generation network, so that the target generation network outputs a first predicted meteorological image sequence corresponding to the second time period, the first time period being earlier than the second time period.

[0065] Third, based on the first predicted meteorological image sequence, determine the weather forecast results corresponding to the second time period.

[0066] Specifically, the initial generation network is trained based on the difference between the discrimination result corresponding to the second predicted meteorological image sequence output by the initial generation network and the preset reference result, as well as the similarity between the feature maps of the reference meteorological image sequence and the second meteorological predicted image sequence, to obtain the target generation network. The reference image sequence and the second predicted image sequence correspond to the same time period earlier than the first time period.

[0067] Specifically, a sequence of meteorological images collected in the first time period is acquired. This sequence is then input into a target generation network, which outputs a first predicted meteorological image sequence corresponding to the second time period. Finally, the weather forecast for the second time period is determined based on the output first predicted meteorological image sequence.

[0068] To facilitate understanding, the specific implementation process of the image data processing method is illustrated by example:

[0069] The processing equipment first acquires a sequence of meteorological images collected between 8:00 AM and 12:00 PM (the first time period), then inputs it into a target generation network. The network outputs a predicted meteorological image sequence (the first predicted image sequence) for the period between 2:00 PM and 4:00 PM (the second time period). Next, based on the predicted meteorological image sequence for the 2:00 PM to 4:00 PM period, the weather forecast for that time period is predicted, yielding the forecast result for the second time period, thus achieving short-term weather forecasting. In practice, weather forecasts within 0 to 3 hours can be referred to as short-term weather forecasts.

[0070] Optionally, the weather forecast result can be displayed on the processing device. When the weather forecast result obtained from the first predicted image sequence is light rain, and the confidence level is 80%, the terminal device displays a light rain pattern and also indicates an 80% probability of precipitation. Since the probability of precipitation is greater than a preset threshold, the display size can be increased. Simultaneously, since the second period is daytime, the background of the weather forecast displayed on the processing device can be white, i.e., daytime mode. Conversely, if the second period is nighttime, the background can be set to gray, i.e., nighttime mode. This part can be combined with... Figure 3 understand.

[0071] In this weather forecasting process, the first predicted image sequence generated not only has good visualization effects and high image detail, but also meets the meteorological TS score.

[0072] For details not described in this embodiment, please refer to [link / reference]. Figure 1 The description in the illustrated embodiment.

[0073] Following the above description, when the acquired image sequence is meteorological imagery, weather forecasting can be achieved. Furthermore, the obtained weather forecast results can be applied to various fields. When weather forecast results are applied to agricultural scenarios, Figure 4 A flowchart illustrating another image data processing method provided in an embodiment of the present invention. Figure 4 As shown, the method may include the following steps:

[0074] S201, Obtain the sequence of meteorological images corresponding to the first time period.

[0075] S202, the collected meteorological image sequence is input into the target generation network in the adversarial generative network, so that the target generation network outputs the first predicted meteorological image sequence corresponding to the second time period, the first time period being earlier than the second time period.

[0076] S203, Based on the first predicted meteorological image sequence, determine the weather forecast result corresponding to the second time period;

[0077] S204 generates reminder information based on weather forecast results to guide the planting of agricultural crops.

[0078] Specifically, a sequence of meteorological images collected in the first time period is acquired. This sequence is then input into a target generation network, which outputs a first predicted meteorological image sequence corresponding to the second time period. The weather forecast for the second time period is determined based on the output first predicted meteorological image sequence. Finally, a reminder message can be generated based on the weather forecast results to guide crop planting.

[0079] In agriculture, the first and second time periods can also be relatively long, such as a week. In this case, the weather forecast results can be used as a basis for crop planting guidance. For example, reminders can be sent to farmers based on the predicted weather forecast, indicating the timing of fertilization and irrigation, thus providing guidance for crop planting. Optionally, the weather forecast results and corresponding reminder messages can be displayed on the processing device or sent to the user's terminal device.

[0080] In addition, the processes and technical effects not described in detail in this embodiment can be found in [reference needed]. Figures 1-3 The relevant descriptions in the illustrated embodiments.

[0081] Similar to the agricultural scenario described above, in the field of water conservancy, after obtaining weather forecast results from the first predicted meteorological image sequence output by the target generation network, these results can optionally be used as a basis for water level prediction, and flood disasters can be monitored based on water level levels, thereby achieving intelligent flood control. Optionally, compared to the field of short-term weather forecasting, in the field of water conservancy, the first time period corresponding to the acquired image sequence and the second time period corresponding to the first predicted image sequence can also be a slightly longer period, such as one day.

[0082] Based on the above embodiments, the training process of the initial generator network can be further described in detail. Figure 5 The training method for generative adversarial networks provided in this embodiment of the invention is used for training... Figures 1-4 The initial generator network in the illustrated embodiment is trained to obtain the target generator network. The execution entity of this method can be a training device, which can be the processing device in the above embodiments or a separate device. Figure 5 As shown, the method includes the following steps:

[0083] S301, acquire the first reference image sequence corresponding to the third time period and the second reference image sequence corresponding to the fourth time period, wherein the third time period, the fourth time period and the first time period are sequentially increased.

[0084] A first reference image sequence corresponding to the third time period and a second reference image sequence corresponding to the fourth time period are obtained, and these two reference image sequences are real historical data. The length relationship between the first and second reference image sequences is not strictly limited. Generally, the number of images in the first reference image sequence is greater than or equal to the number of images in the second reference image sequence.

[0085] and Figure 1 Similarly, in the illustrated embodiments, the specific representation of the reference images varies depending on the application scenario and field. For example, in a weather forecast scenario, the first reference image sequence may include meteorological images from the first N hours of the day, and the second reference image sequence may include meteorological images from the next N hours. In an autonomous driving scenario, the first reference image sequence may include driving images of the vehicle within the first T minutes, and the second reference image sequence may include driving images of the vehicle from minute T+1 to minute T+N. Wherein, combined with... Figure 1 In the illustrated embodiment, the first and second time periods, the third and fourth time periods, and the first and second time periods are in an ascending order. That is, the third time period is the earliest, and the second time period is the latest.

[0086] S302, the first reference image sequence is input into the initial generation network so that the initial generation network outputs the second predicted image sequence.

[0087] S303, the second predicted image sequence is input into the discrimination network so that the discrimination network outputs the discrimination result corresponding to the second predicted image sequence.

[0088] Then, the first reference image sequence obtained in step 301 is input into the initial generator network in the generative adversarial network, so that the initial generator network generates a second predicted image sequence based on the first reference image sequence. Optionally, the initial generator network can be a U2-Net model, or other network models based on convolutional neural networks (CNNs), etc.

[0089] The similarity between the second predicted image sequence generated here and the second reference image sequence obtained in step 301 is that both the predicted image sequence and the second reference image sequence correspond to the second time period, that is, they contain the same number of images; the difference is that the second reference sequence is the actual collected real image, while the second predicted image sequence is a fake image sequence generated by the generator network based on the first reference image sequence.

[0090] Next, the generated second predicted image sequence is used as the whole input to the discriminant network in the generative adversarial network, so that the discriminant network can determine the authenticity of the entire predicted image sequence. That is, the discriminant network outputs a judgment result that can be either "true" or "false".

[0091] In practice, the stronger the image generation capability of the generative network, the closer the second predicted image sequence generated by the generative network is to the second reference image sequence, the higher the realism of the images in the second predicted image sequence, and the easier it is for the adversarial network to identify the predicted image sequence as "real".

[0092] S304, based on the difference between the discrimination result corresponding to the second predicted image sequence and the preset reference result, and the similarity between the feature maps of the second reference image sequence and the second predicted image sequence, train the initial generation network.

[0093] Finally, using the loss function, a first loss value can be calculated based on the difference between the discrimination result output by the discriminator network and the preset reference result. For clarity in subsequent descriptions, this preset reference result can be referred to as the first preset reference result. The first preset reference result is a known value. Since the training objective of the generator network is to make the predicted image sequence generated by the generator network more realistic, preventing the discriminator network from accurately distinguishing between real and fake predicted image sequences, the aforementioned first preset reference result can be set to "true".

[0094] Furthermore, since the aforementioned first loss value is obtained using the discrimination result output by the discriminant network, it reflects the adversarial relationship between the discriminant and generator networks during network training. Therefore, the first loss value can also be called the adversarial loss value. The magnitude of the first loss value can reflect the overall realism of the predicted image sequence.

[0095] While calculating the first loss value, a second loss value can also be calculated using the loss function, based on the similarity between feature maps at different levels of the second reference image sequence and the second predicted image sequence. Specifically, since the second reference image sequence and the second predicted image sequence each contain the same number of images, the second loss value can be calculated based on the similarity between feature maps at the same level of corresponding images in the two image sequences. The magnitude of the second loss value reflects the similarity in image content between the two image sequences; therefore, it can also be called the perceptual loss value corresponding to the feature map.

[0096] Optionally, feature maps of the second reference image sequence and the second predicted image sequence can be extracted using convolutional computation. The number of feature maps for any image in the image sequence is the same as the number of convolutions. Furthermore, the extracted feature maps can include both low-level and high-level feature maps.

[0097] The similarity between low-level feature maps of two image sequences reflects their textural similarity, while the similarity between high-level feature maps reflects their semantic similarity. Therefore, compared to the first loss value, the second loss value can reflect the authenticity of the predicted image sequence in more detail from the perspectives of image texture and semantics.

[0098] Alternatively, the second loss value L can be calculated using the following loss function. FM (y,y'):

[0099]

[0100] Where y and y' represent the second reference image sequence and the second predicted image sequence, respectively, and F l (y) represents the feature map of each image in the second reference image sequence, F l (y') represents the feature map of each image in the second prediction reference image sequence, and L is the number of times each image in the second reference image sequence and the second prediction image sequence is convolved.

[0101] Finally, the initial generator network can be trained by combining the first loss value (adversarial loss value) and the second loss value (perceptual loss value) obtained above. In other words, the network parameters of the initial generator network are adjusted according to the loss values ​​to obtain the target generator network.

[0102] It should be noted that when adjusting the network parameters of the initially generated network, the network parameters of the discrimination network need to be fixed.

[0103] In this embodiment, a first reference image sequence and a second reference image sequence are acquired sequentially. Then, the generator network in the generative adversarial network (GAN) generates a second predicted image sequence based on the first reference image sequence. The discriminator network in the GAN then discriminates against the second predicted image sequence. Finally, an initial generator network is trained based on the difference between the discriminator's output and a preset reference result, and on the similarity between the feature maps of the images in the second reference image sequence and the second predicted image sequence corresponding to the same acquisition time period. The difference between the discriminator and the reference result reflects the overall realism of the predicted image sequence; the similarity between the feature maps reflects the realism of the predicted image sequence from the perspectives of texture and semantics.

[0104] As can be seen, the training process described above considers different dimensions of different image sequences when training the initial generator network. This means it considers both the overall image sequence and the details of each image within it. This ensures that the image sequences generated by the trained target generator network possess a certain degree of realism, both overall and in terms of texture and semantics of each individual image, guaranteeing the target generator network has good image generation capabilities. Furthermore, the quality of image generation capability is directly reflected in the clarity of the generated images. Therefore, the generator network trained in this way can generate clear image sequences, thus optimizing the visualization effect of the image sequences and improving their refinement.

[0105] When calculating the perceptual loss value, feature maps at different levels of the second reference image sequence and the second predicted image sequence are required. One alternative feature map extraction method is to input the second reference image sequence and the second predicted image sequence into a discriminator network, so that the discriminator network simultaneously obtains feature maps at different levels of the two image sequences during the process of discriminating the second predicted image sequence.

[0106] Based on this feature map extraction method, Figure 5 The training process in the illustrated embodiment can be combined with Figure 6 To understand the network structure of the Generative Adversarial Network shown.

[0107] Alternatively, a feature map extraction method can be used where the second reference image sequence and the second predicted image sequence are input into a feature extraction network, respectively, to extract feature maps. This network is primarily responsible for feature extraction during the entire training process of the Generative Adversarial Network (GAN). The feature extraction network does not participate in the training of the GAN; that is, the network parameters of the feature extraction network are not updated during the training of the GAN.

[0108] Optionally, the feature extraction network mentioned above can be any network with feature map extraction capabilities, such as a network, a CNN network, etc.

[0109] Optionally, after extracting features from the feature maps, the feature extraction network can also obtain the style matrix corresponding to each feature map. Optionally, a loss value can also be calculated based on the similarity between the style matrices; this loss value can also be considered as the perceptual loss value corresponding to the style matrix. Furthermore, the network parameters of the generator network can be adjusted simultaneously based on multiple perceptual loss values ​​and adversarial loss values.

[0110] Optionally, the style matrix mentioned above is specifically the representation gram matrix, and the loss value L corresponding to the style matrix can be calculated using the following loss function. STYLE (y,y'):

[0111]

[0112] Where y and y' represent the second reference image sequence and the second predicted image sequence, respectively, and F l (y) represents the feature map of each image in the second reference image sequence, F l (y') represents the feature map of each image in the second prediction reference image sequence, G(F) l (y) represents the feature map F l The Gram matrix corresponding to (y), G(F) l (y')) represents the feature map F l (y') is the Gram matrix corresponding to the second reference image sequence and the second predicted image sequence, where L is the number of convolutions of each image in the second reference image sequence and the second predicted image sequence.

[0113] Based on this feature map extraction method, Figure 5 The training process in the illustrated embodiment can be combined with Figure 7 To understand the network structure of the Generative Adversarial Network shown.

[0114] Optionally, in practice, either of the two feature map extraction methods mentioned above can be used.

[0115] In summary, based on Figure 6 The network structure shown considers both the overall image sequence and the details of each image within it during the initial training process. These details can include the texture and semantics of each image in the sequence. Based on... Figure 7 The training process shown also considers both the overall image sequence and the details of each image in the sequence, except that the details of the images also include image style.

[0116] Based on the above embodiments, optionally, pixel-level similarity between images in the image sequence can also be calculated. Specifically, this involves calculating the similarity between pixels of corresponding images in a second prediction reference image sequence of equal length and a second reference image sequence. This yields a pixel loss value, which is then used to train the initial generator network. In other words, pixel-level image details are taken into account during the training of the initial generator network.

[0117] Optionally, the similarity between pixels can be characterized by any of the following metrics: Structural Similarity (SSIM), Peak Signal to Noise Ratio (PSNR), or LPIPS (Learned Perceptual Image Patch Similarity).

[0118] In practice, the entire training process of a generative adversarial network typically involves alternating training of the generator network and the discriminator network. Therefore, after completing one round of training for the initial generator network according to the methods provided in the above embodiments, another round of training for the discriminator network can be initiated.

[0119] Optionally, Figure 8 A flowchart illustrating another method for training an adversarial generative network provided in an embodiment of the present invention. For example... Figure 8 As shown, the method may include the following steps:

[0120] S401, acquire the first reference image sequence corresponding to the third time period and the second reference image sequence corresponding to the fourth time period, wherein the third time period, the fourth time period and the first time period are sequentially increased.

[0121] S402, the first reference image sequence is input into the initial generation network so that the initial generation network outputs the second predicted image sequence.

[0122] S403, the second predicted image sequence is input into the discrimination network so that the discrimination network outputs the discrimination result corresponding to the second predicted image sequence.

[0123] S404, based on the difference between the discrimination result corresponding to the second predicted image sequence and the first preset reference result, and the similarity between the feature maps of the second reference image sequence and the second predicted image sequence, train the initial generation network.

[0124] The execution process of steps S401 to S404 above can be found in the following example. Figure 5 The relevant descriptions in the illustrated embodiments will not be repeated here. It should be noted that the first preset reference result in step 404 is... Figure 5 The preset reference result in the illustrated embodiment.

[0125] S405, based on the difference between the discrimination result corresponding to the second predicted image sequence and the second preset reference result, a discrimination network is trained, wherein the first preset reference result is different from the second preset reference result.

[0126] During the training of the generator network, the network parameters of the discriminator network need to be fixed. After completing steps 401-404, the initial training round of the generator network is complete. Afterwards, the network parameters of the generator network can be fixed, and the training round of the discriminator network can begin. Similar to the generator network, the discriminator network also requires multiple training iterations to converge. Therefore, any discriminator network that fails to converge during training can be considered the initial discriminator network. The discriminator network trained in the various embodiments provided by this invention is the initial discriminator network.

[0127] Specifically, the discrimination network is trained based on the difference between the discrimination result corresponding to the second predicted image sequence output by the discrimination network and the second preset reference result during the initial generation network training.

[0128] The second preset reference result differs from the first preset reference result used in step 404. The relationship between the two preset reference results can be understood as follows: the purpose of training the generator network is to make the fake image sequences generated by the generator network more realistic; that is, if a fake image sequence generated by the generator network is input into the discriminator network, the discriminator network can classify it as "real." Conversely, the purpose of training the discriminator network is to enable the discriminator network to accurately distinguish whether the input image sequence is a real image sequence actually captured or a fake image sequence generated by the generator network.

[0129] To achieve the aforementioned training objective, the two preset reference results need to be set differently, or more specifically, opposite. When training the generator network using the predicted image sequence in step 402, the first preset reference result can be set to "true". When training the discriminator network again using the predicted image sequence in step 402 after completing one round of training of the generator network, the second preset reference result can be set to "false".

[0130] After performing the above steps, one round of training for the initial generator network and the adversarial network is completed. In practice, the initial generator network and the discriminator network can be trained alternately multiple times, as described in this embodiment, until a converged target generator network is obtained. It should be noted that the above embodiments are only illustrative examples of training the generator network first and then the adversarial network. In practice, the adversarial network can also be trained first, followed by the generator network. This invention does not limit the training order of the two networks.

[0131] based on Figure 6 The network structure shown can be combined with the training process of the generator network and the discriminator network in one round. Figure 9 Understanding. Based on Figure 7 The network structure shown can be combined with the training process of the generator network and the discriminator network in one round. Figure 10 understand.

[0132] In addition, based on the above training process, in order to further ensure the training effect of the discriminant network, the second reference image sequence can optionally be used as a training sample input to the discriminant network to adjust the network parameters of the discriminant network according to the discrimination result and the preset reference result. Here, the preset reference result can be set to "true".

[0133] In this embodiment, the network parameters of the discriminator network can be fixed first to train the generator network, and then the network parameters of the generator network can be fixed to train the discriminator network, thus achieving alternating training of the two. During this alternating training process, the generator network, trained in the previous round, already possesses a certain image generation capability, while the discriminator network, trained in the previous round, also possesses a certain discrimination capability. Therefore, in the next round of training for the generator network, the discrimination results output by the discriminator network trained in the previous round will have a stronger adversarial effect on the training of the generator network. This stronger adversarial effect will further improve the image generation capability of the generator network, that is, to enable the image sequences generated by the generator network to have better visualization effects and ensure the refinement of the image sequences.

[0134] Additionally, for parts not described in detail in this embodiment, please refer to the description of the relevant documentation. Figures 5 to 7 The relevant descriptions of the illustrated embodiments are provided below. For the execution process and technical effects of this technical solution, please refer to [link / reference]. Figures 5 to 7 The descriptions in the illustrated embodiments will not be repeated here.

[0135] Figure 11 This is a flowchart illustrating another method for training an adversarial generative network, as provided in an embodiment of the present invention. Figure 11 As shown, the method may include the following steps:

[0136] S501, acquire the first reference image sequence corresponding to the third time period and the second reference image sequence corresponding to the fourth time period, wherein the third time period, the fourth time period and the first time period are sequentially increased.

[0137] S502, the first reference image sequence is input into the initial generation network so that the initial generation network outputs the second predicted image sequence.

[0138] S503, the second predicted image sequence is input into the discrimination network so that the discrimination network outputs the discrimination result corresponding to the second predicted image sequence.

[0139] S504, based on the difference between the discrimination result corresponding to the second predicted image sequence and the first preset reference result, and the similarity between the feature maps of the second reference image sequence and the second predicted image sequence, train the initial generation network.

[0140] The execution process of steps S501 to S504 above can be found in the following example. Figure 5 The relevant descriptions in the illustrated embodiments will not be repeated here. It should be noted that the first preset reference result in step 504 is... Figure 5 The preset reference result in the illustrated embodiment. S505, the first stitching result of the first reference image sequence and the second predicted reference image sequence is input into the discrimination network, so that the discrimination network outputs the discrimination result corresponding to the first stitching result.

[0141] After obtaining the second predicted image sequence in step 502, the first reference image sequence and the second predicted reference image sequence can be concatenated to obtain a first concatenation result. Then, this first concatenation result is input into a discrimination network, which outputs a discrimination result corresponding to the first concatenation result.

[0142] Since the first stitching result also includes the predicted image sequence, the discrimination result corresponding to the first stitching result output by the discriminator network can also reflect the authenticity of the second predicted image sequence generated by the initial generator network. Furthermore, since the first reference image sequence contains multiple images continuously acquired within the first time period, meaning that the images in the first reference image sequence have a certain degree of continuity in content, the discriminator network will also consider the continuity between the images in the second predicted image sequence when using the first stitching result to determine the authenticity of the second predicted image sequence.

[0143] S506, train the initial generator network based on the difference between the discrimination result corresponding to the first concatenation result and the third preset reference result. Similar to step 504, the adversarial loss value used to train the initial generator network can also be obtained based on the difference between the discrimination result corresponding to the first concatenation result and the third preset reference result. The third preset reference result can be the same as the first preset reference result, both set to "true".

[0144] It should be noted that in practical applications, there are no strict timing restrictions between steps 503-504 and steps 505-506. Furthermore, steps 503-504 and steps 505-506 can be executed selectively.

[0145] based on Figure 6 The network structure shown in this embodiment, and the training process of the generative network provided, can be combined with... Figure 12 understand.

[0146] Since the similarity between feature maps can reflect the authenticity of an image sequence by showing greater detail, the discrimination result corresponding to the predicted image sequence can reflect the authenticity of the image sequence as a whole, and the discrimination result corresponding to the first stitching result can reflect the authenticity of the image sequence from the perspective of the coherence of the image content, this embodiment uses the similarity between feature maps of the image sequence, as well as the discrimination results corresponding to the predicted image sequence and the first stitching result, to adjust the network parameters. This ensures that the generator network has good image generation capabilities from different dimensions, so that the image sequence generated by the trained generator network not only has good visualization effects and refinement, but also has consistency in the temporal dimension, that is, the image content of each image in the image sequence has a certain continuity.

[0147] exist Figure 8 Based on the illustrated embodiment, for training the generator network, optionally, the first reference image sequence and the second reference image sequence can be concatenated to obtain a second concatenation result. Then, the first and second concatenation results are respectively input into a discriminator network, which outputs feature maps of each of the two concatenation results. Furthermore, the network parameters of the generator network are adjusted based on the similarity between the feature maps of the first and second concatenation results.

[0148] In accordance with Figure 9 After completing one round of training of the initial generated network in the manner shown in the embodiment, the network parameters of the initial generated network can be further fixed in order to start training of the discrimination network.

[0149] Optionally, the network parameters of the discrimination network can be adjusted based on the difference between the discrimination result corresponding to the first splicing result and the fourth preset reference result. The fourth preset reference result is the opposite of the third preset reference result, inheriting... Figure 9 As illustrated in the embodiments, this fourth preset reference result can be the same as the second preset reference result, both set to "false".

[0150] Alternatively, the second splicing result can be directly used as a training sample to adjust the network parameters of the discriminant network based on the difference between the discriminant result of the second splicing result output by the discriminant network and the preset reference result "true", which is to achieve the training of the discriminant network.

[0151] To facilitate understanding, the specific implementation process of the training method for the Generative Adversarial Network (GAN) provided above will be illustrated using the following weather forecast scenario as an example.

[0152] The meteorological department can collect meteorological image sequence 1 (i.e., the first reference image sequence in the above embodiment) from 8:00 to 10:00 on April 20, 2021, and can also collect meteorological image sequence 2 (i.e., the second reference image sequence in the above embodiment) from 10:00 to 12:00 on April 20, 2021. Meteorological image sequence 1 includes images a1 to an, and meteorological image sequence 2 includes images b1 to bm. Furthermore, the meteorological image sequence is composed of multiple radar echo images.

[0153] Next, the meteorological image sequence 1 is input into the initial generator network of the adversarial generative network to be trained, so that the initial generator network generates meteorological image sequence 2' (i.e., the second predicted image sequence in the above embodiment) based on meteorological image sequence 1. Meteorological image sequence 2' includes images b1' to bm', and is a meteorological image sequence generated by the generator network based on meteorological image sequence 1 for predicting the weather from 10:00 to 12:00 on April 20th.

[0154] Next, the meteorological image sequence 2' is input into the discriminant network in the adversarial generative network to be trained. The discriminant network can determine that the meteorological image sequence 2' is "fake" as a whole. At this point, the adversarial loss value can be calculated based on the difference between the discrimination result "fake" output by the discriminant network and the first preset reference result "true".

[0155] While inputting meteorological image sequence 2' into the discriminant network, meteorological image sequence 2 can also be input into the discriminant network. This allows the discriminant network to obtain feature maps at different levels for both meteorological image sequence 2 and meteorological image sequence 2' during the process of outputting the aforementioned discrimination result "false". The perceptual loss value corresponding to the discriminant network is then obtained based on the similarity between the feature maps of the same level in corresponding meteorological images of the two meteorological image sequences. That is, the perceptual loss value can be obtained based on the similarity between the low-level feature maps of image bx in image sequence 2 and image bx' in meteorological image sequence 2', as well as the similarity between their high-level feature maps. Here, x is any value from 1 to m, and x' is any value from 1' to m'.

[0156] Optionally, after the initial generator network generates meteorological image sequence 2', meteorological image sequence 2 and meteorological image sequence 2' can be input into the feature extraction model, and the perceptual loss value corresponding to the feature extraction model can be calculated using the feature maps of the corresponding images extracted by the feature extraction model. While extracting feature maps using the feature extraction model, the style matrix of each image in meteorological image sequence 2 and meteorological image sequence 2' can also be obtained, and the perceptual loss value corresponding to the style matrix can be obtained based on the similarity between the style matrices.

[0157] In practice, the initial generator network can be trained based on the various perceptual loss values ​​and adversarial loss values ​​obtained above. This ensures that when generating images, the trained generator network will consider the overall authenticity of the image sequence as well as the authenticity of each image in terms of texture, semantics, and style, so as to guarantee the visualization effect and refinement of the generated images.

[0158] Optionally, the initial generator network can be trained from the pixel perspective of the images. That is, the similarity between each pixel in image bx in image sequence 2 and image bx' in meteorological image sequence 2' is calculated to obtain the pixel loss value. At this time, the network parameters of the generator network can be adjusted together based on the pixel loss value and the perceptual loss value and adversarial loss value obtained above to ensure the realism of each image in the image sequence generated by the generator network at the pixel level.

[0159] The initial training of the generator network can be achieved using the aforementioned loss values. At this point, the network parameters of the generator network can be fixed, and training of the discriminator network can begin. Specifically, the network parameters of the discriminator network are adjusted based on the discrimination result "false" of the meteorological image sequence 2' output by the discriminator network and the second preset reference result "false".

[0160] Alternatively, meteorological image sequence 2 can be input into the discrimination network, and the network parameters of the discrimination network can be adjusted based on the discrimination result "true" corresponding to meteorological image sequence 2 output by the discrimination network and the preset reference result "true".

[0161] Optionally, for the initial generation network training process, in addition to using meteorological image sequence 2 and meteorological image sequence 2' separately as described above, the first splicing result of meteorological image sequence 1 and meteorological image sequence 2' can also be used.

[0162] Specifically, the first stitching result is input into the discriminant network, which outputs a "false" discriminant result corresponding to the first stitching result. The adversarial loss value is then obtained based on the difference between the third preset reference result ("true") and the discriminant result ("false"), and this adversarial loss value is used to train the initial generator network. When using the generator network trained based on the stitching result to generate image sequences, the generated image sequences will have a certain continuity in image content, thereby improving the realism of the generated image sequences.

[0163] Optionally, the second stitching result of meteorological image sequence 2 and meteorological image sequence 2', as well as the first stitching result mentioned above, can be input into the discriminant network so that the discriminant network can calculate the perceptual loss value corresponding to the extracted feature map and train the generator network based on this perceptual loss value.

[0164] After generating the network once based on both individual image sequences and the stitched result, the discriminator network can be further trained. Specifically, an adversarial loss value can be obtained based on the difference between the discrimination result "false" corresponding to the first stitched result and the fourth preset reference result "false," and the network parameters of the discriminator network can be adjusted based on this adversarial loss value. Optionally, the discriminator network can also be trained based on the difference between the discrimination result "true" corresponding to the second stitched result and the preset reference result "true."

[0165] The following will describe in detail one or more embodiments of an image data processing apparatus according to the present invention. Those skilled in the art will understand that these image data processing apparatuses can all be configured using commercially available hardware components through the steps taught in this invention.

[0166] Figure 13 This is a schematic diagram of the structure of an image data processing device provided in an embodiment of the present invention, as shown below. Figure 13 As shown, the device includes:

[0167] The acquisition module 11 is used to acquire the image sequence corresponding to the first time period.

[0168] Input module 12 is used to input the acquired image sequence into the target generation network so that the target generation network outputs a first predicted image sequence corresponding to a second time period, the first time period being earlier than the second time period.

[0169] The result determination module 13 is used to determine the prediction result corresponding to the second time period based on the first predicted image sequence.

[0170] Specifically, the initial generation network is trained based on the difference between the discrimination result corresponding to the second predicted image sequence output by the initial generation network and the preset reference result, as well as the similarity between the feature maps of the reference image sequence and the second predicted image sequence, to obtain the target generation network.

[0171] Optionally, the device further includes a style determination module 21 and a display module 22.

[0172] The result determination module 13 is also used to determine the confidence level of the prediction result.

[0173] The style determination module 21 is used to determine the display style of the prediction result based on the second time period corresponding to the first prediction image sequence and the prediction result.

[0174] The display module 22 is used to display the prediction result and the confidence level of the prediction result according to the display style.

[0175] Optionally, the device further includes a transmitting module 23.

[0176] The acquisition module 12 is also used to receive the acquired image sequence sent by the terminal device.

[0177] The sending module is used to send the prediction result corresponding to the first predicted image sequence and the confidence level of the prediction result, so that the terminal device can receive and display the prediction result and the confidence level according to the display style.

[0178] Optionally, the device is further used for:

[0179] A first reference image sequence corresponding to the third time period and a second reference image sequence corresponding to the fourth time period are obtained, wherein the third time period, the fourth time period, and the first time period are sequentially increased;

[0180] The first reference image sequence is input into the initial generation network, so that the initial generation network outputs the second predicted image sequence;

[0181] The second predicted image sequence is input into the discrimination network, and the discrimination network outputs the discrimination result corresponding to the second predicted image sequence;

[0182] The initial generation network is trained based on the difference between the discrimination result corresponding to the second predicted image sequence and the preset reference result, as well as the similarity between the feature maps of the second reference image sequence and the second predicted image sequence.

[0183] Optionally, the apparatus is further configured to: train the initial generation network based on the similarity between the second predicted image sequence and corresponding images in the second reference image sequence.

[0184] Optionally, the apparatus is further configured to: input a first stitching result of the first reference image sequence and the second predicted reference image sequence into the discriminant network, so that the discriminant network outputs a discriminant result corresponding to the first stitching result;

[0185] The initial generation network is trained based on the difference between the discrimination result corresponding to the first splicing result and the preset reference result.

[0186] Optionally, the device is further configured to: input the first splicing result into the discriminant network, so that the discriminant network outputs a feature map of the first splicing result;

[0187] The second concatenation result of the first reference image sequence and the second reference image sequence is input into the discriminant network, so that the discriminant network outputs the feature map of the second concatenation result;

[0188] The initial generative network is trained based on the similarity between the feature maps of the first and second splicing results.

[0189] Figure 13 The device shown can perform Figures 1 to 3 as well as Figures 5 to 12 For the methods shown in the embodiments, the parts not described in detail in this embodiment can be referred to the following: Figures 1 to 3 as well as Figures 5 to 12 The relevant descriptions of the illustrated embodiments are provided below. For the execution process and technical effects of this technical solution, please refer to [link / reference]. Figures 1 to 3 as well as Figures 5 to 12 The descriptions in the illustrated embodiments will not be repeated here.

[0190] The above describes the internal functions and structure of the image data processing device. In one possible design, the image data processing device can be implemented as an electronic device, such as... Figure 14As shown, the electronic device may include a processor 31 and a memory 32. The memory 32 is used to store data supporting the electronic device in performing the above-described actions. Figures 1 to 3 as well as Figures 5 to 12 The image data processing method program provided in the illustrated embodiment is configured to execute the program stored in the memory 32.

[0191] The program includes one or more computer instructions, wherein when the one or more computer instructions are executed by the processor 31, they can perform the following steps:

[0192] Obtain the image sequence corresponding to the first time period;

[0193] The acquired image sequence is input into the target generation network, so that the target generation network outputs a first predicted image sequence corresponding to the second time period, the first time period being earlier than the second time period;

[0194] Based on the first predicted image sequence, determine the prediction result corresponding to the second time period;

[0195] Specifically, the initial generation network is trained based on the difference between the discrimination result corresponding to the second predicted image sequence output by the initial generation network and the preset reference result, as well as the similarity between the feature maps of the reference image sequence and the second predicted image sequence, to obtain the target generation network.

[0196] Optionally, the processor 31 is further configured to perform the aforementioned Figures 1 to 3 as well as Figures 5 to 12 All or part of the steps in the illustrated embodiments.

[0197] The structure of the electronic device may also include a communication interface 33 for the electronic device to communicate with other devices or communication networks.

[0198] In addition, embodiments of the present invention provide a computer storage medium for storing computer software instructions used by the aforementioned electronic device, which includes instructions for executing the above-mentioned... Figures 1 to 3 as well as Figures 5 to 12 The procedure involved in the image data processing method in the illustrated embodiment.

[0199] Figure 15 This is a schematic diagram of another image data processing device provided in an embodiment of the present invention, as shown below. Figure 15 As shown, the device includes:

[0200] The acquisition module 41 is used to acquire the sequence of meteorological images corresponding to the first time period.

[0201] Input module 42 is used to input the acquired meteorological image sequence into the target generation network so that the target generation network outputs a first predicted meteorological image sequence corresponding to a second time period, the first time period being earlier than the second time period.

[0202] The result determination module 43 is used to determine the weather forecast result corresponding to the second time period based on the first predicted meteorological image sequence.

[0203] The generation module 44 is used to generate reminder information to guide the planting of agricultural crops based on the weather forecast results.

[0204] Specifically, the initial generation network is trained based on the difference between the discrimination result corresponding to the second predicted meteorological image sequence output by the initial generation network and the preset reference result, as well as the similarity between the feature maps of the reference meteorological image sequence and the second meteorological predicted image sequence, to obtain the target generation network.

[0205] Optionally, the device further includes a style determination module 45 and a display module 46.

[0206] The result determination module 43 is also used to determine the confidence level of the weather forecast result.

[0207] The style determination module 45 is used to determine the display style of the weather forecast result based on the second time period corresponding to the first predicted meteorological image sequence and the weather forecast result.

[0208] The display module 46 is used to display the prediction result and the confidence level of the prediction result according to the display style.

[0209] Figure 15 The device shown can perform Figure 4 For the methods shown in the embodiments, the parts not described in detail in this embodiment can be referred to the following: Figure 4 The relevant descriptions of the illustrated embodiments are provided below. For the execution process and technical effects of this technical solution, please refer to [link / reference]. Figure 4 The descriptions in the illustrated embodiments will not be repeated here.

[0210] The above describes the internal functions and structure of the image data processing device. In one possible design, the image data processing device can be implemented as an electronic device, such as... Figure 16 As shown, the electronic device may include a processor 51 and a memory 52. ​​The memory 52 is used to store data supporting the electronic device in performing the above-described actions. Figure 4 The image data processing method program provided in the illustrated embodiment is configured to execute the program stored in the memory 52.

[0211] The program includes one or more computer instructions, wherein when the one or more computer instructions are executed by the processor 51, they can perform the following steps:

[0212] Obtain the sequence of meteorological images corresponding to the first time period;

[0213] The acquired meteorological image sequence is input into the target generation network, so that the target generation network outputs a first predicted meteorological image sequence corresponding to the second time period, wherein the first time period is earlier than the second time period;

[0214] Based on the first predicted meteorological image sequence, determine the weather forecast result corresponding to the second time period;

[0215] Based on the weather forecast results, reminder information is generated to guide the planting of agricultural crops;

[0216] Specifically, the initial generation network is trained based on the difference between the discrimination result corresponding to the second predicted meteorological image sequence output by the initial generation network and the preset reference result, as well as the similarity between the feature maps of the reference meteorological image sequence and the second meteorological predicted image sequence, to obtain the target generation network.

[0217] Optionally, the processor 51 is further configured to perform the aforementioned Figure 4 All or part of the steps in the illustrated embodiments.

[0218] The structure of the electronic device may also include a communication interface 53 for the electronic device to communicate with other devices or communication networks.

[0219] In addition, embodiments of the present invention provide a computer storage medium for storing computer software instructions used by the aforementioned electronic device, which includes instructions for executing the above-mentioned... Figure 4 The procedure involved in the image data processing method in the illustrated embodiment.

[0220] It should be noted that, as mentioned above Figure 15 and Figure 16 The provided devices and electronic equipment can also be applied to weather forecasting scenarios to obtain weather forecast results.

[0221] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image data processing method, characterized in that, include: Obtain the image sequence corresponding to the first time period; The acquired image sequence is input into the target generation network, so that the target generation network outputs a first predicted image sequence corresponding to the second time period, wherein the first time period is earlier than the second time period; Based on the first predicted image sequence, determine the prediction result corresponding to the second time period; Specifically, the initial generation network is trained based on the difference between the discrimination result and the preset reference result corresponding to the second predicted image sequence output by the initial generation network, the similarity between the semantic feature maps of the reference image sequence and the second predicted image sequence, the similarity between the style matrices corresponding to the feature maps of different levels of the reference image sequence and the second predicted image sequence, and the similarity between the pixels of the corresponding images in the reference image sequence and the second predicted image sequence, so as to obtain the target generation network. The second predicted image sequence is output by the initial generation network before the first predicted image sequence, and the discrimination result is output by the discrimination network. The difference between the discrimination result corresponding to the second predicted image sequence and the preset reference result is the difference between the discrimination result corresponding to the first stitching result and the preset reference result. The discrimination result corresponding to the first stitching result is obtained by processing the first stitching result of the first reference image sequence and the second predicted image sequence through the discrimination network. The first reference image sequence contains multiple images continuously acquired within the first time period.

2. The method according to claim 1, characterized in that, The method further includes: Determine the confidence level of the prediction result; Based on the second time period corresponding to the first predicted image sequence and the prediction result, determine the display style of the prediction result; The prediction result and its confidence level are displayed according to the specified display style.

3. The method according to claim 2, characterized in that, The acquisition of the image sequence corresponding to the first time period includes: The acquired image sequence is sent by the receiving terminal device; The method further includes: The prediction result corresponding to the first predicted image sequence and the confidence level of the prediction result are sent so that the terminal device can receive and display the prediction result and the confidence level according to the display style.

4. The method according to claim 1, characterized in that, The step of training the initial generation network based on the difference between the discrimination result corresponding to the second predicted image sequence output by the initial generation network and the preset reference result, and the similarity between the feature maps of the reference image sequence and the second predicted image sequence, includes: A first reference image sequence corresponding to the third time period and a second reference image sequence corresponding to the fourth time period are obtained, wherein the third time period, the fourth time period, and the first time period are sequentially increased; The first reference image sequence is input into the initial generation network, so that the initial generation network outputs the second predicted image sequence; The second predicted image sequence is input into the discrimination network, and the discrimination network outputs the discrimination result corresponding to the second predicted image sequence; The initial generation network is trained based on the difference between the discrimination result corresponding to the second predicted image sequence and the preset reference result, as well as the similarity between the feature maps of the second reference image sequence and the second predicted image sequence.

5. The method according to claim 4, characterized in that, The method further includes: The initial generation network is trained based on the similarity between the second predicted image sequence and the corresponding images in the second reference image sequence.

6. The method according to claim 4, characterized in that, The method further includes: The first stitching result of the first reference image sequence and the second predicted image sequence is input into the discrimination network, so that the discrimination network outputs the discrimination result corresponding to the first stitching result; The initial generation network is trained based on the difference between the discrimination result corresponding to the first splicing result and the preset reference result.

7. The method according to claim 6, characterized in that, The method further includes: The first splicing result is input into the discriminant network, so that the discriminant network outputs the feature map of the first splicing result; The second concatenation result of the first reference image sequence and the second reference image sequence is input into the discriminant network, so that the discriminant network outputs the feature map of the second concatenation result; The initial generative network is trained based on the similarity between the feature maps of the first and second splicing results.

8. An image data processing method, characterized in that, include: Obtain the sequence of meteorological images corresponding to the first time period; The acquired meteorological image sequence is input into the target generation network, so that the target generation network outputs a first predicted meteorological image sequence corresponding to the second time period, wherein the first time period is earlier than the second time period; Based on the first predicted meteorological image sequence, determine the weather forecast result corresponding to the second time period; Specifically, the initial generation network is trained based on the difference between the discrimination result and the preset reference result corresponding to the second predicted meteorological image sequence output by the initial generation network, the similarity between the semantic feature maps of the reference meteorological image sequence and the second predicted meteorological image sequence, the similarity between the style matrices corresponding to the feature maps of different levels of the reference meteorological image sequence and the second predicted meteorological image sequence, and the similarity between the pixels of the corresponding images in the reference meteorological image sequence and the second predicted meteorological image sequence, so as to obtain the target generation network. The second predicted meteorological image sequence is output by the initial generation network before the first predicted meteorological image sequence, and the discrimination result is output by the discrimination network. The difference between the discrimination result corresponding to the second predicted meteorological image sequence and the preset reference result is the difference between the discrimination result corresponding to the first stitching result and the preset reference result. The discrimination result corresponding to the first stitching result is obtained by processing the first stitching result of the first reference meteorological image sequence and the second predicted meteorological image sequence through the discrimination network. The first reference meteorological image sequence contains multiple images continuously collected within the first time period.

9. The method according to claim 8, characterized in that, The method further includes: Determine the confidence level of the weather forecast results; Based on the second time period corresponding to the first predicted meteorological image sequence and the weather forecast result, determine the display style of the weather forecast result; The weather forecast result and its confidence level are displayed according to the specified display style.

10. An image data processing method, characterized in that, include: Obtain the sequence of meteorological images corresponding to the first time period; The acquired meteorological image sequence is input into the target generation network, so that the target generation network outputs a first predicted meteorological image sequence corresponding to the second time period, wherein the first time period is earlier than the second time period; Based on the first predicted meteorological image sequence, determine the weather forecast result corresponding to the second time period; Based on the weather forecast results, reminder information is generated to guide the planting of agricultural crops; Specifically, the initial generation network is trained based on the difference between the discrimination result and the preset reference result corresponding to the second predicted meteorological image sequence output by the initial generation network, the similarity between the feature maps of the reference meteorological image sequence and the second predicted meteorological image sequence, the similarity between the style matrices corresponding to the feature maps of different levels of the reference meteorological image sequence and the second predicted meteorological image sequence, and the similarity between the pixels of the corresponding images in the reference meteorological image sequence and the second predicted meteorological image sequence, so as to obtain the target generation network. The second predicted meteorological image sequence is output by the initial generation network before the first predicted meteorological image sequence, and the discrimination result is output by the discrimination network. The difference between the discrimination result corresponding to the second predicted meteorological image sequence and the preset reference result is the difference between the discrimination result corresponding to the first stitching result and the preset reference result. The discrimination result corresponding to the first stitching result is obtained by processing the first stitching result of the first reference meteorological image sequence and the second predicted meteorological image sequence through the discrimination network. The first reference meteorological image sequence contains multiple images continuously collected within the first time period.

11. An electronic device, characterized in that, include: A memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor performs the image data processing method as described in any one of claims 1 to 10.

12. A non-transitory machine-readable storage medium, characterized in that, The non-transitory machine-readable storage medium stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the image data processing method as described in any one of claims 1 to 10.

Images