Image prediction method, image prediction model training method, device and equipment

Abstract

The present disclosure relates to the field of metallurgical technology, and discloses an image prediction method, a training method, a device and equipment for an image prediction model; the method includes: obtaining a foam image before operation and operation adjustment data in real time; processing the obtained data using a foam image prediction model to predict a foam image after operation; the prediction model is a first model or a second model; the first model includes a first encoder for processing the foam image before operation and the operation adjustment data to obtain a first latent code, and a first generator for processing the first latent code to obtain a foam image after operation; the second model includes a second encoder for processing the foam image before operation to obtain a second latent code, and a second generator for processing the second latent code and the operation adjustment data to obtain a foam image after operation. The model provided by the present disclosure can extract style information of the foam image, and the style information can be used to efficiently and accurately predict the foam image after operation adjustment.

Description

Technical Field

[0001] The present disclosure relates to the field of metallurgical technology, and in particular to an image prediction method, an image prediction model training method, a device and equipment. Background Art

[0002] As a mineral processing technology widely used in mining and metallurgy, froth flotation can use the hydrophobicity of target minerals to separate target minerals from non-target minerals such as gangue, thereby achieving the purpose of mineral processing. As an important visual representation of the flotation process state, the change of froth image can intuitively reflect the effect of flotation operation adjustment and the flotation behavior of minerals. Therefore, the prediction of froth image occupies a pivotal position in flotation optimization control.

[0003] In the related art, the patent application with the patent publication number CN113128126A discloses a modeling method for the flotation dosing process based on a generative adversarial network. Although this method can predict the foam image after the flotation operation, the training process of the model provided by this method has the problem of instability, resulting in poor prediction effect of the model, and even mode collapse, thereby affecting the quality of the generated foam image. Therefore, it is urgent to solve this technical problem. Summary of the invention

[0004] In view of the above situation, the embodiments of the present disclosure provide an image prediction method, a training method, an apparatus and a device for an image prediction model, aiming to solve the above problems or at least partially solve the above problems.

[0005] In a first aspect, an embodiment of the present disclosure provides an image prediction method, the method comprising:

[0006] Acquire in real time the foam image before operation and the operation adjustment data affecting the foam image before operation;

[0007] Using the trained foam image prediction model, the foam image before the operation and the operation adjustment data are processed to predict the foam image after the operation;

[0008] Among them, the trained foam image prediction model is the first model or the second model; the first model includes a first encoder and a first generator, the first encoder is used to process the foam image before operation and the operation adjustment data to obtain a first latent code in the latent space for representing the image style, and the first generator is used to process the first latent code to obtain the foam image after operation; the second model includes a second encoder and a second generator, the second encoder is used to process the foam image before operation to obtain a second latent code in the latent space for representing the image style, and the second generator is used to process the second latent code and the operation adjustment data to obtain the foam image after operation.

[0009] In a second aspect, the present disclosure also provides a method for training an image prediction model, the method comprising:

[0010] Acquire a first data set, wherein any first target sample in the first data set is composed of a first sample pre-operation image and first sample operation adjustment data, and a label of the first target sample is a corresponding first sample post-operation image;

[0011] Constructing a first foam image prediction model to be trained; wherein the first foam image prediction model to be trained is the first model to be trained or the second model to be trained; the first model to be trained includes a first encoder, a first generator and a first discriminator, the first encoder is used to process the first sample pre-operation image and the first sample operation adjustment data to obtain a first sample latent code in a latent space for characterizing an image style, and the first generator is used to process the first sample latent code to obtain a predicted foam image after operation; the second model to be trained includes a second encoder, a second generator and a second discriminator, the second encoder is used to process the first sample pre-operation image to obtain a second sample latent code in a latent space for characterizing an image style, and the second generator is used to process the second sample latent code and the first sample operation adjustment data to obtain a predicted foam image after operation;

[0012] The first foam image prediction model to be trained is trained using the first data set to obtain a trained first foam image prediction model; and a trained foam image prediction model is obtained according to an encoder and a generator in the trained first foam image prediction model.

[0013] In a third aspect, the present disclosure also provides an image prediction device, the device comprising:

[0014] A first acquisition module, used for acquiring in real time the foam image before operation and the operation adjustment data affecting the foam image before operation;

[0015] A prediction module, used to process the pre-operation foam image and the operation adjustment data using a trained foam image prediction model, and predict a post-operation foam image;

[0016] Among them, the trained foam image prediction model is the first model or the second model; the first model includes a first encoder and a first generator, the first encoder is used to process the foam image before operation and the operation adjustment data to obtain a first latent code in the latent space for representing the image style, and the first generator is used to process the first latent code to obtain the foam image after operation; the second model includes a second encoder and a second generator, the second encoder is used to process the foam image before operation to obtain a second latent code in the latent space for representing the image style, and the second generator is used to process the second latent code and the operation adjustment data to obtain the foam image after operation.

[0017] In a fourth aspect, the embodiments of the present disclosure further provide a training device for an image prediction model, the device comprising:

[0018] A second acquisition module is used to acquire a first data set, wherein any first target sample in the first data set is composed of a first sample pre-operation image and first sample operation adjustment data, and a label of the first target sample is a corresponding first sample post-operation image;

[0019] A construction module, used for constructing a first foam image prediction model to be trained; wherein the first foam image prediction model to be trained is the first model to be trained or the second model to be trained; the first model to be trained includes a first encoder, a first generator and a first discriminator, the first encoder is used for processing the first sample pre-operation image and the first sample operation adjustment data to obtain a first sample latent code in a latent space for characterizing an image style, and the first generator is used for processing the first sample latent code to obtain a predicted foam image after operation; the second model to be trained includes a second encoder, a second generator and a second discriminator, the second encoder is used for processing the first sample pre-operation image to obtain a second sample latent code in a latent space for characterizing an image style, and the second generator is used for processing the second sample latent code and the first sample operation adjustment data to obtain a predicted foam image after operation;

[0020] A training module is used to train the first foam image prediction model to be trained using the first data set to obtain a trained first foam image prediction model; and to obtain a trained foam image prediction model based on the encoder and generator in the trained first foam image prediction model.

[0021] In a fifth aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor executes the steps of the above-mentioned image prediction method / image prediction model training method.

[0022] In a sixth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple applications, the electronic device executes the steps of the above-mentioned image prediction method / image prediction model training method.

[0023] By means of the above technical solution, the image prediction method, the training method, the device and the equipment of the image prediction model provided by the embodiment of the present disclosure, the prediction method provides a style-based foam image prediction model, when implemented, the model structure of the model is a first model or a second model. The foam image before operation and the operation adjustment data can be processed by the first encoder in the first model, and the first latent code used to characterize the style of the foam image after the operation adjustment can be effectively extracted in the latent space, and the first latent code can be processed by the first generator to generate the foam image after operation. The foam image before operation can also be processed by the second encoder in the second model, and the second latent code used to characterize the style of the operation latent foam image can be effectively extracted in the latent space, and the second latent code and the operation adjustment data can be processed by the second generator to generate the foam image after operation. Among them, since the first latent code or the second latent code contains the style information of the foam image, this scheme can achieve more precise control of the foam image generation process after the operation, and also makes the prediction model have the advantages of stable training, insensitivity of model performance to the weight parameters of the loss function, and easy reproduction and popularization of application, thereby greatly enhancing the authenticity of the foam image after the operation, and achieving the purpose of accurate and fast real-time prediction of the foam image after the operation adjustment.

[0024] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the specific implementation methods of the present disclosure are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

[0026] Figure 1 A schematic diagram of a process flow of an image prediction method provided by an embodiment of the present disclosure is shown;

[0027] Figure 2 A schematic diagram of the structure of a trained foam image prediction model provided by an embodiment of the present disclosure is shown;

[0028] Figure 3A schematic diagram of the structure of a foam image prediction model provided by an embodiment of the present disclosure is shown;

[0029] Figure 4 A comparison diagram of the effects of the predicted foam image after operation and the real foam image after operation adjustment provided by the embodiment of the present disclosure is shown;

[0030] Figure 5 A schematic diagram of the structure of a conversion network and an initial generator provided by an embodiment of the present disclosure is shown;

[0031] Figure 6 A schematic diagram of a process of training a second foam image prediction model to be trained provided by an embodiment of the present disclosure is shown;

[0032] Figure 7 An example diagram of an image editing result based on the bubble semantic direction provided by an embodiment of the present disclosure is shown;

[0033] Figure 8 A line graph showing the average foam size change of an edited image edited at an amplitude of 0.2 along 50 semantic directions provided by an embodiment of the present disclosure relative to an edited image edited at an amplitude of -0.2 is shown;

[0034] Fig. 9 A line graph showing the change in the mean value of the V channel of an edited image obtained by editing at an amplitude of 0.2 along 50 semantic directions relative to an edited image obtained by editing at an amplitude of -0.2 provided by an embodiment of the present disclosure;

[0035] Fig.10 A diagram showing experimental results of mining the influence of operation adjustment on foam properties in an interpretable manner provided by an embodiment of the present disclosure;

[0036] Fig.11 A schematic diagram showing a flow chart of a method for training an image prediction model provided by an embodiment of the present disclosure;

[0037] Fig.12 A schematic diagram of the structure of an initial encoder provided by an embodiment of the present disclosure is shown;

[0038] Fig.13 A schematic diagram showing the structure of an image prediction device provided by an embodiment of the present disclosure is shown;

[0039] Fig.14 A schematic diagram of the structure of a training device for an image prediction model provided by an embodiment of the present disclosure is shown;

[0040] Fig.15 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure will be clearly and completely described below in combination with the specific embodiments of the present disclosure and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0042] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0043] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such usage is interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the term "including" and its variants are to be interpreted as open-ended terms meaning "including but not limited to".

[0044] As described above, in the related art, the patent application with patent publication number CN113128126A discloses a modeling method for flotation dosing process based on generative adversarial network. Although this method can predict the foam image after flotation operation, the training process of the model provided by this method has the problem of instability, resulting in poor prediction effect of the model, and even mode collapse, thereby affecting the quality of the generated foam image. Based on this, the present invention proposes an image prediction method, training method, device, equipment and medium of the image prediction model. The present disclosure is described in detail below through specific embodiments.

[0045] The execution subject of the image prediction method / image prediction model training method provided in the embodiments of the present disclosure is generally a computer device with certain computing capabilities, and the computer device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, etc. In some possible implementations, the image prediction method / image prediction model training method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0046] For ease of understanding, an image prediction method disclosed in an embodiment of the present disclosure is first introduced in detail. Figure 1A schematic diagram of the process of the image prediction method provided by the embodiment of the present disclosure is shown. Figure 1 It can be seen that the embodiment of the present disclosure at least includes steps S101-S102:

[0047] S101: acquiring in real time a foam image before operation and operation adjustment data affecting the foam image before operation.

[0048] In the actual flotation process, the flotation cell mainly contains ore pulp, flotation reagents and other substances. Bubbles and flotation reagents can be used to separate useful minerals from ores. Usually, a shooting device for real-time monitoring of the foam state, flow conditions and state changes after operation adjustment in the flotation cell is installed above the flotation cell. During implementation, one or more shooting devices can be used to capture the image of the foam in the flotation cell in real time to obtain the foam image before operation, and then the collected foam image before operation is transmitted to the execution subject of the embodiment of the present disclosure by wireless or wired means. Here, the shooting device includes an image acquisition device and a video acquisition device. If it is an image acquisition device, the foam image in the flotation cell is directly acquired. If it is a video acquisition device, the video of the flotation cell is acquired and then decoded to acquire the foam image therein. The operation adjustment data includes one or more of all controllable parameters that affect the flotation state, such as the amount of reagent added, the size of the air intake valve, the opening of the liquid level valve, etc., which are not limited in the embodiment of the present disclosure. During implementation, for example, the operation adjustment data can be directly obtained from the flotation monitoring system, and the operation adjustment data can also be manually input.

[0049] S102: using the trained foam image prediction model, processing the foam image before the operation and the operation adjustment data, and predicting the foam image after the operation.

[0050] Figure 2 FIG. 2 shows a schematic diagram of the structure of a trained foam image prediction model provided by an embodiment of the present disclosure. Figure 2 As shown, in this step, the trained foam image prediction model is the first model or the second model; the first model includes a first encoder and a first generator, the first encoder is used to process the foam image before the operation and the operation adjustment data to obtain a first latent code in the latent space for representing the image style, and the first generator is used to process the first latent code to obtain the foam image after the operation; the second model includes a second encoder and a second generator, the second encoder is used to process the foam image before the operation to obtain a second latent code in the latent space for representing the image style, and the second generator is used to process the second latent code and the operation adjustment data to obtain the foam image after the operation.

[0051] Here, the first encoder and the second encoder are encoders that can encode the input data into the w latent space and its expansion space, such as convolutional encoders, etc., which are not limited to the embodiments of the present disclosure. The first generator and the second generator can process the input latent code, constant and random noise to obtain the foam image after the operation. Here, the generator can adopt the generator structure in the StyleGAN architecture, and preferably, the generator structure in the StyleGAN2.0 architecture can be used. The logic of this step can be expressed by the following formula:

[0052]

[0053] in, represents the trained foam image prediction model, x represents the foam image before the input model operation, a represents the operation adjustment data, Post-operation foam image representing the model's predictions.

[0054] from Figure 1 As can be seen from the method shown, the embodiment of the present disclosure provides a style-based foam image prediction model. When implemented, the model structure of the model is a first model or a second model. The first encoder in the first model can be used to process the foam image before operation and the operation adjustment data, effectively extract the first latent code in the latent space for characterizing the style of the foam image after the operation adjustment, and then use the first generator to process the first latent code to generate the foam image after the operation. The second encoder in the second model can also be used to process the foam image before operation, effectively extract the second latent code in the latent space for characterizing the style of the latent foam image, and then use the second generator to process the second latent code and the operation adjustment data to generate the foam image after the operation. Among them, since the first latent code or the second latent code contains the style information of the foam image, this solution can achieve more precise control of the foam image generation process after the operation, and also make the prediction model have the advantages of stable training, insensitivity of the model performance to the weight parameters of the loss function, and easy to reproduce and popularize, thereby greatly enhancing the authenticity of the foam image after the operation, and achieving the purpose of accurate and fast real-time prediction of the foam image after the operation adjustment.

[0055] Further, in order to better illustrate the process of the above-mentioned image prediction method, as a refinement and extension of the above-mentioned embodiment, the embodiment of the present invention provides several optional embodiments, but is not limited thereto, which are specifically as follows:

[0056] In a possible implementation manner, before the step of using the trained foam image prediction model to process the pre-operation foam image and the operation adjustment data to predict the post-operation foam image, the method further includes:

[0057] Step A1: Acquire a first data set, wherein any first target sample in the first data set is composed of a first sample pre-operation image and first sample operation adjustment data, and a label of the first target sample is a corresponding first sample post-operation image.

[0058] Step A2: construct a first foam image prediction model to be trained; wherein the first foam image prediction model to be trained is the first model to be trained or the second model to be trained; the first model to be trained includes a first encoder, a first generator and a first discriminator, the first encoder is used to process the first sample image before operation and the first sample operation adjustment data to obtain a first sample latent code in the latent space for representing the image style, and the first generator is used to process the first sample latent code to obtain a predicted foam image after operation; the second model to be trained includes a second encoder, a second generator and a second discriminator, the second encoder is used to process the first sample image before operation to obtain a second sample latent code in the latent space for representing the image style, and the second generator is used to process the second sample latent code and the first sample operation adjustment data to obtain a predicted foam image after operation.

[0059] Step A3: using the first data set to train the first foam image prediction model to be trained to obtain a trained first foam image prediction model; according to the encoder and the generator in the trained first foam image prediction model, a trained foam image prediction model is obtained.

[0060] In this embodiment, the first data set can be obtained first. During implementation, for example, multiple groups of historical data can be selected from the flotation monitoring system as samples in the first data set, and each group of historical data includes a foam image before operation adjustment, operation adjustment data, and a foam image after operation adjustment. It is also possible to first perform center cropping and scaling on each foam image, and perform standardization on the operation adjustment data, and then use the foam image before operation, operation adjustment data, and foam image after operation that meet the requirements as a data group in the data set according to actual needs, thereby completing the construction of the first data set.

[0061] Reconstruct the first foam image prediction model to be trained. Here, the first encoder and the second encoder are encoders that can encode input data into w latent space and its expansion space, such as convolutional encoders, etc., which are not limited to the embodiments of the present disclosure. The first generator and the second generator can adopt the generator structure in the StyleGAN architecture, and the input data includes the input of the main branch and the input of the side branch. Among them, the main branch is composed of a series of convolutional layers and upsampling layers, the input of the main branch is a constant, and the input of the side branch is the first sample latent code or the second sample latent code, and random noise. Here, the first sample latent code or the second sample latent code can control the foam image generation process after the operation by modulating and demodulating the convolution weights, so that the generated image is more accurate. The discriminator is a network structure that can process images, such as a convolutional neural network structure, etc., which is not limited to the embodiments of the present disclosure.

[0062] Finally, the first foam image prediction model to be trained can be trained using the first data set, and the network parameters therein can be updated. After obtaining the first foam image prediction model that meets the training requirements, the network structure composed of the first encoder and the first generator in the trained first foam image prediction model that meets the training requirements can be determined as the above-mentioned trained foam image prediction model, or the network structure composed of the second encoder and the second generator in the trained first foam image prediction model that meets the training requirements can be determined as the above-mentioned trained foam image prediction model. Among them, meeting the training requirements can be: the function value of the preset loss function converges, or the training accuracy of the first foam image prediction model to be trained meets the preset requirements. The present disclosure does not make specific limitations on meeting the training requirements, and it is subject to what can be achieved.

[0063] In one possible implementation, the trained foam image prediction model is the second model; the second model also includes a mapping network and an editing module; the mapping network is used to process the operation adjustment data and generate a corresponding image editing amplitude vector; the editing module is used to edit the second latent code according to the image editing amplitude vector and a preset semantic direction set, generate an edited latent code, and input it into the second generator.

[0064] Figure 3 FIG. 2 shows a schematic diagram of the structure of the foam image prediction model provided by the embodiment of the present disclosure. Figure 3 As shown, in this embodiment, the trained foam image prediction model (second model) further includes a mapping network and an editing module. The outputs of the second encoder and the mapping network are used as inputs of the editing module, and the output of the editing module is used as input of the second generator.

[0065] The mapping network is used to convert the input operation adjustment data into a vector composed of edit amplitudes corresponding to different semantic attributes in the latent space. During implementation, the mapping network is a network structure that can convert the operation adjustment data into a vector, such as a fully connected neural network, which is not limited in the embodiments of the present disclosure. Exemplarily, if the total number of semantic attributes is 2, the mapping network can convert the operation adjustment data into a two-dimensional image edit amplitude vector.

[0066] The editing module can use the obtained image editing amplitude vector and the preset semantic direction set to edit the second latent code to obtain the edited latent code, and input it into the generator. After being processed by the generator, the foam image after the operation can be obtained. The specific editing process is to edit the second latent code along multiple semantic directions in the preset semantic direction set according to the editing amplitude indicated by the aforementioned image editing amplitude vector. When implemented, the edited latent code can be calculated according to the following formula:

[0067]

[0068] in, Indicates the latent code after editing, represents the second latent code, represents the i-th semantic direction, Represents the image editing magnitude corresponding to the i-th semantic direction.

[0069] Here, for the preset semantic direction set, for example, the principal component analysis method can be used to obtain it. Specifically, a series of foam images can be collected as a data set; high-dimensional feature vectors of the foam images can be extracted; then, the extracted feature vectors can be standardized to eliminate the dimensional differences between different features; the covariance matrix can be calculated (the covariance matrix can be calculated based on the standardized feature vectors); the eigenvalues ​​and eigenvectors can be calculated (the eigenvalue decomposition of the covariance matrix can be performed to obtain the eigenvalues ​​and corresponding eigenvectors); the principal components, i.e., the semantic directions, can be selected to form the preset semantic direction set (the first k principal components can be selected based on the size of the eigenvalues, and these principal components represent the directions with the largest changes in the data).

[0070] For example, Figure 4 A comparison diagram of the effects of the predicted foam image after operation and the real foam image after operation adjustment provided by an embodiment of the present disclosure is shown. Figure 4In the figure, from left to right, the image before operation is the foam image before operation, the true value image after operation is the real foam image after operation adjustment, and the generated image after operation is the generated foam image after operation. The horizontal axis of the area distribution graph is the foam area Bubble Area, and the vertical axis is the number Number. The horizontal axis of the V channel distribution graph is the V channel pixel value Value, and the vertical axis is the frequency Frequency of the V channel pixel value. In the two types of graphs, the gray column corresponds to the real foam image after operation adjustment, and the black column corresponds to the generated foam image after operation. Figure 4 It can be seen that, firstly, the changes in the foam size and color attributes in the generated foam image after operation are consistent with the operation adjustment, and the generated foam image after operation has good image quality. Secondly, from the foam area distribution and V channel distribution corresponding to the generated foam image after operation and the real foam image after operation adjustment, it can be seen that the generated foam image after operation and the real foam image after operation adjustment are similar. In addition, the average speed of predicting the image using the established foam image prediction model (second model) is 0.14 seconds. In summary, the performance of the established foam image prediction model in terms of generated image quality, simulation prediction accuracy, simulation prediction speed, etc. meets the application requirements of industrial sites.

[0071] The foam image prediction model in this embodiment also includes a mapping network and an editing module, wherein the mapping network can process the operation adjustment data to obtain the corresponding image editing amplitude vector, and the editing module can use the image editing amplitude vector and the preset semantic direction set to edit the second latent code to obtain the edited latent code and input it into the generator. It can be seen that the foam image prediction model provided by this embodiment adopts the real image editing method. Through the real image editing, the operation adjustment data in the data space can be converted into the editing amplitude of the foam semantic attribute in the latent space, so that the model can more accurately simulate the impact of the operation adjustment on the flotation state, thereby generating a higher quality predicted foam image.

[0072] In a possible implementation manner, the second model including the mapping network and the editing module is generated by training according to the following method:

[0073] Step B1: Acquire a second data set, a third data set and a fourth data set; wherein the second data set includes multiple real foam images, the third data set includes multiple foam images to be reconstructed, and any second target sample in the fourth data set is composed of a second sample pre-operation image and second sample operation adjustment data, and the label of the second target sample is the corresponding second sample post-operation image.

[0074] Step B2: construct an initial generative adversarial network, and use the second data set to train the initial generative adversarial network to obtain a trained generative adversarial network; wherein the initial generative adversarial network includes a conversion network, an initial generator and an initial discriminator, the conversion network is used to process the random latent vector to obtain a first target latent code, and the initial generator is used to process the first target latent code, a constant input and random noise to obtain a generated foam image.

[0075] Step B3: construct an initial image reconstruction model, which includes: an initial encoder and a generator in a trained generative adversarial network, the initial encoder is used to encode the foam image to be reconstructed to obtain a second target latent code, and the generator in the trained generative adversarial network is used to generate a reconstructed foam image based on the second target latent code, a constant input and random noise.

[0076] Step B4: Use the third data set to train the initial image reconstruction model, update the weights of the initial encoder, and obtain a trained image reconstruction model.

[0077] Step B5: construct a second foam image prediction model to be trained, the second foam image prediction model to be trained includes a mapping network, a discriminator, an editing module, a pre-trained encoder and a pre-trained generator, wherein the mapping network is used to process the second sample operation adjustment data to generate a sample image editing amplitude vector, the pre-trained encoder and the pre-trained generator are the encoder and the generator in the trained image reconstruction model, the pre-trained encoder is used to process the first sample pre-operation image to obtain the corresponding third target latent code, and the editing module is used to generate a sample edited latent code based on the sample image editing amplitude vector, a preset semantic direction set and the third target latent code.

[0078] Step B6: Use the fourth data set to train the second foam image prediction model to be trained, update the weights of the mapping network and the discriminator, and obtain the trained second foam image prediction model; obtain the second model according to the mapping network, editing module, encoder and generator in the trained second foam image prediction model.

[0079] In this embodiment, the second data set, the third data set and the fourth data set can be obtained first. In specific implementation, for example, the historical foam images captured by the camera in the flotation monitoring system can be used as samples in the second data set and the third data set. It is also possible to first perform center cropping and scaling on each historical foam image, and then use the foam image that meets the requirements as a sample in the data set according to actual needs, thereby completing the construction of the second data set and the third data set. For the fourth data set, the construction method of the first data set in the aforementioned step A1 can be specifically referred to, and no further details will be given here.

[0080] Then, an initial generative adversarial network can be constructed. The initial generative adversarial network includes a conversion network, an initial generator, and an initial discriminator for distinguishing generated foam images from real foam images. Exemplarily, Figure 5 A schematic diagram of the structure of a conversion network and an initial generator provided by an embodiment of the present disclosure is shown. Figure 5 In it, Z represents Z latent space, latent z represents random latent vector z, Normalize represents normalization, FC represents fully connected layer, w represents latent code, A represents affine transformation operation, Mod represents modulation, Demod represents demodulation, Conv represents convolution kernel, Upsample represents upsampling, Const represents constant, B represents broadcast operation, and Noise represents random noise. The conversion network can transform the received random latent vector to obtain the latent code in w latent space. The generator uses the input of the main branch and the side branch at the same time, where the input of the main branch is a constant, and the input of the side branch is random noise and latent code. Among them, the latent code controls the image generation process by modulating and demodulating the convolution weights, and the input of random noise can add random details to the generated image, thereby increasing the diversity of the generated graph, and finally obtaining the generated foam image. The data processing logic of the conversion network and the initial generator is as follows:

[0081]

[0082]

[0083] Among them, z represents the random latent vector, T represents the conversion network, G represents the generator, is the generated foam image. During implementation, the size of the input foam image may be, for example, 256×256.

[0084] After obtaining the second data set and the initial generative adversarial network, the second data set can be used to train the initial generative adversarial network to obtain a trained generative adversarial network. During implementation, for example, the second data set can be randomly divided into a training set, a validation set, and a test set in a ratio of 8:1:1 to iteratively update the network parameters in the initial generator and the initial discriminator in the initial generative adversarial network to obtain a trained generative adversarial network. The optimizer used for model training can be, for example, the Ranger optimizer, the initial learning rate can be set to 0.001, the number of training steps can be set to 800,000 steps, and the number of batch training can be set to 4. After the model is trained, the model parameters with the best performance can be saved.

[0085] After obtaining the trained generative adversarial network, an initial image reconstruction model can be constructed, which includes an initial encoder and a generator in the aforementioned trained generative adversarial network. Among them, the initial encoder is used to encode the foam image to be reconstructed to obtain a second target latent code, which is equivalent to replacing the aforementioned conversion network with the initial encoder. The generator can process the second target latent code, constant and random noise to obtain a reconstructed foam image. The structure of the initial encoder is an encoder that can encode the initial image reconstruction model into the w latent space and its extended space, such as a convolutional encoder, etc., which is not limited to this embodiment of the present disclosure. Then, the initial image reconstruction model is trained using a third data set, and the weight parameter data of the initial encoder in the model is updated to obtain a trained image reconstruction model. During implementation, the optimizer used for model training is, for example, a Ranger optimizer, the initial learning rate can be set to 0.001, the number of training times can be set to 200,000 steps, and the number of batch training can be set to 4. After the model is trained, the encoder model parameters with the best performance can be saved.

[0086] Next, a second foam image prediction model to be trained is constructed. Figure 6 FIG. 2 shows a schematic diagram of a process for training a second foam image prediction model to be trained provided by an embodiment of the present disclosure. Figure 6 As shown, the second foam image prediction model to be trained includes a mapping network R, a discriminator D, an editing module, an encoder E in a trained image reconstruction model, and a generator G. The second foam image prediction model to be trained can be trained using the fourth data set to update the weight parameters of the mapping network and the discriminator.

[0087] In a possible implementation manner, the second foam image prediction model to be trained is trained using the fourth data set according to any of the following methods, and the weights of the mapping network and the discriminator are updated to obtain a trained second foam image prediction model:

[0088] Method 1: The second sample pre-operation image and the second sample operation adjustment data of any third target sample in the fourth data set are respectively input into the pre-trained encoder and the mapping network to obtain the corresponding third target latent code and the sample image editing amplitude vector; along each semantic direction in the preset semantic direction set, the third target latent code is edited using the sample image editing amplitude vector corresponding to the third target sample to obtain the first sample edited latent code; the first sample edited latent code is processed using the pre-trained generator to obtain the first predicted image after operation; according to the preset loss function, the first predicted image after operation and the second sample foam image after operation of the third target sample, the weights of the mapping network and the discriminator are updated to finally obtain a trained second foam image prediction model.

[0089] Method 2: Input the second sample pre-operation image and the second sample operation adjustment data of any third target sample in the fourth data set into the pre-trained encoder and the mapping network respectively, and obtain the corresponding third target latent code and sample image editing amplitude vector respectively; select any unused semantic direction from the preset semantic direction set as the semantic direction for image editing; along the semantic direction for image editing, edit the third target latent code using the corresponding editing amplitude in the sample image editing amplitude vector of the third target sample to obtain the second sub-sample edited latent code; use the pre-trained generator to process the second sub-sample edited latent code to obtain the second sub-prediction operation image; update the mapping network, according to the preset loss function, the second sub-prediction operation image and the second sample operation foam image of the third target sample. The weight of the discriminator is obtained, and the step of selecting any unadopted semantic direction from the preset semantic direction set as the semantic direction for image editing is returned, until there is no unadopted semantic direction in the preset semantic direction set, and the intermediate second foam image prediction model is obtained; along each semantic direction in the preset semantic direction set, the third target latent code is edited using the sample image editing amplitude vector corresponding to the third target sample to obtain the edited latent code of the second sample; the edited latent code of the second sample is processed using the pre-trained generator to obtain the second predicted image after operation; according to the preset loss function, the second predicted image after operation and the foam image of the second sample after operation of the third target sample, the weights of the mapping network and the discriminator in the intermediate second foam image prediction model are updated, and finally the trained second foam image prediction model is obtained.

[0090] This embodiment provides two training methods for the training process of the second foam image prediction model to be trained. Figure 6 These two methods are explained.

[0091] For the first method, in the specific implementation, the second sample pre-operation image and the second sample operation adjustment data of any third target sample in the fourth data set can be respectively input into the pre-trained encoder (encoder E) and the mapping network R to obtain the third target latent code w and the sample image editing amplitude vector f; then, the editing module uses the sample image editing amplitude vector f to edit the third target latent code w along all semantic directions in the preset semantic direction set to obtain the first sample edited latent code; then, the pre-trained generator (generator G) processes the first sample edited latent code to obtain the first predicted image after operation (generated image y'), and calculates the preset loss function value (mapping network loss value and discriminator loss value) according to the first predicted image after operation and the true value image y (the second sample after operation foam image of the third target sample), and updates the weight parameters of the mapping network R and the discriminator according to the mapping network loss value and the discriminator loss value. By analogy, the second foam image prediction model to be trained can continue to be trained using other samples until the model meets the training requirements. The present disclosure does not specifically limit the satisfaction of the training requirements, which shall be subject to implementation. This training method focuses on the overall impact of the operation adjustment data on the pre-operation image, and edits the third target latent code simultaneously along all semantic directions. After iterative training, the final mapping network is obtained. The editing amplitude output by the mapping network is used to edit the pre-operation image to obtain the predicted post-operation image, which can greatly improve the accuracy of predicting the post-operation image.

[0092] For the second method, in the specific implementation, the second sample pre-operation image and the second sample operation adjustment data of any third target sample in the fourth data set can be respectively input into the pre-trained encoder (encoder E) and the mapping network R to obtain the third target latent code w and the sample image editing amplitude vector f; then, any unused semantic direction is selected from the preset semantic direction set as the semantic direction for image editing. For example, the preset semantic direction set includes 10 semantic directions, and one semantic direction can be selected from them as the semantic direction for image editing; then, along the semantic direction, the sample image of the third target sample is used to generate the image editing image. The third target latent code is edited according to the edit amplitude corresponding to the semantic direction in the edit amplitude vector to obtain the edited latent code of the second sub-sample; the pre-trained generator (generator G) is then used to process the edited latent code of the second sub-sample to obtain the second sub-prediction operation image (generated image y'); according to the second sub-prediction operation image and the true value image y (the second sample operation foam image of the third target sample), the preset loss function value (mapping network loss value and discriminator loss value) is calculated, and the weight parameters of the mapping network R and the discriminator are updated according to the mapping network loss value and the discriminator loss value. Then, return to the step of selecting any unused semantic direction from the preset semantic direction set as the semantic direction for image editing, select any semantic direction from the remaining 9 semantic directions as the semantic direction for image editing, and edit the third target latent code along the semantic direction, generate the corresponding second sub-prediction operation image, and update the weight parameters of the mapping network R and the discriminator until there is no unused semantic direction in the preset semantic direction set, and obtain the intermediate second foam image prediction model. Finally, the third target latent code w is edited along all semantic directions in the preset semantic direction set using the sample image editing amplitude vector f to obtain the second sample edited latent code; then, the pre-trained generator (generator G) processes the second sample edited latent code to obtain the second predicted image after operation (generated image y'), and the preset loss function value (mapping network loss value and discriminator loss value) is calculated based on the second predicted image after operation and the true value image y (the second sample operated foam image of the third target sample), and the weight parameters of the mapping network R and the discriminator in the intermediate second foam image prediction model are updated based on the mapping network loss value and the discriminator loss value. Similarly, other samples can be used to continue to train the second foam image prediction model to be trained until the model meets the training requirements. The present disclosure does not specifically limit the satisfaction of the training requirements, and it is subject to what can be achieved.This training method adds an image editing process for a single semantic direction on the basis of method one, so that this scheme can focus on the local impact of the operation adjustment data on the pre-operation image. After iterative training, it can accurately control and adjust the local image features in a single semantic direction, enhance the influence and controllability on specific areas of the pre-operation image, and make the entire scheme more flexible in image modification and generation and more goal-oriented.

[0093] During implementation, for example, the Ranger optimizer can be used to optimize and update the model parameters, the initial learning rate can be set to 0.001, the number of training times can be set to 200,000 steps, and the number of batch training can be set to 2. After obtaining the trained model, the model parameters with the best performance are saved as the trained second foam image prediction model. Finally, the network structure consisting of the mapping network, editing module, encoder and generator in the trained second foam image prediction model is determined as the second model. It can be understood that the encoder and generator in the trained second foam image prediction model correspond to the second encoder and the second generator in the second model, respectively.

[0094] In a possible implementation, the preset semantic direction set is generated according to the following method: extracting weight parameter data of each layer from the generator of the trained generative adversarial network to obtain a weight matrix; constructing a semantic direction optimization model, and solving the semantic direction optimization model to obtain a candidate semantic direction set; for any target candidate semantic direction in the candidate semantic direction set, along the target candidate semantic direction, respectively using a preset first editing amplitude and a preset second editing amplitude, the acquired target to-be-edited latent code is edited to obtain a corresponding first edited latent code and a second edited latent code; respectively inputting the first edited latent code and the second edited latent code into the generator of the trained generative adversarial network to obtain a first edited image and a second edited image; calculating similarity data between the first edited image and the second edited image; based on each similarity data, eliminating elements in the candidate semantic direction set to obtain a preset semantic direction set; wherein the semantic direction optimization model is:

[0095]

[0096] in, is the candidate semantic direction set, It is used to calculate the independent variable or independent variable set corresponding to the maximum value of the function. is a d×k real number matrix representing the set of semantic directions, represents the i-th semantic direction vector, for The transpose of is the number of semantic directions, d is the dimension of each semantic direction vector, and A is the weight matrix.

[0097] In this embodiment, the weight parameter data of each layer can be extracted from the generator of the trained generative adversarial network to obtain a weight matrix; the constructed semantic direction optimization model is solved to obtain a set of candidate semantic directions. The disclosed embodiment does not limit the solution method. For example, the semantic direction optimization model can be optimized and solved using the gradient ascent method, the projected gradient method, the Lagrange multiplier method, the global optimization algorithm, etc. The number of semantic directions can be set according to actual needs, and this embodiment does not limit this.

[0098] Then, the semantic directions in the candidate semantic direction set are screened to select directions with obvious semantic changes. In specific implementation, for any target semantic direction in the preset semantic direction set, the acquired target latent code to be edited can be edited along the target semantic direction using the preset first editing amplitude and the preset second editing amplitude, respectively, to obtain the corresponding first edited latent code and the second edited latent code; then, the first edited latent code and the second edited latent code are respectively input into the pre-training generator to obtain the first edited image and the second edited image. Here, the preset first editing amplitude and the preset second editing amplitude can be set as needed, and the embodiment of the present disclosure is not limited. Then, the similarity data between the first edited image and the second edited image is calculated. For example, the structural similarity index (SSIM), mean square error (MSE) and other methods can be used for calculation, and the embodiment of the present disclosure is not limited to this. Finally, based on each similarity data, the elements in the candidate semantic direction set are eliminated to obtain the preset semantic direction set. For example, each similarity data can be compared with a preset similarity threshold. If the similarity data is greater than or equal to the preset similarity threshold, the semantic direction corresponding to the similarity data is eliminated. If the similarity data is less than the preset similarity threshold, the semantic direction corresponding to the similarity data is retained, and finally a preset semantic direction set is obtained. Each similarity data can also be sorted in ascending order, and a preset number of semantic directions ranked at the back are eliminated to obtain a preset semantic direction set. Here, the preset similarity threshold and the preset number can be set as needed, and the embodiments of the present disclosure are not limited thereto.

[0099] This embodiment first pre-trains a generator. Since the data processed during the training of the generator is noise data randomly sampled from the latent space, the trained generator can generate more realistic and diverse images. Based on this, when subsequently training the image reconstruction network and the second foam image prediction model, the network parameters of the generators in the two networks are fixed to the network parameters of the pre-trained generator, so that the trained encoder and mapping network are consistent with the semantic direction characteristics in the latent space represented by the weight parameters of the trained generator, thereby ensuring that the trained second foam image prediction model can adjust the data according to the operation, and accurately edit the pre-operation image along the semantic direction, greatly improving the accuracy of the generated post-operation image.

[0100] In a possible implementation manner, if the trained second foam image prediction model is obtained by training according to the second method; the method further includes:

[0101] Step C1: Use the pre-trained encoder to process the image to be edited to obtain the latent code to be edited.

[0102] Step C2: along any target semantic direction in the preset semantic direction set, the preset third editing amplitude and the preset fourth editing amplitude are used to edit the latent code to obtain the corresponding third edited latent code and fourth edited latent code.

[0103] Step C3: respectively input the third edited latent code and the fourth edited latent code into the pre-trained generator to obtain the third edited image and the fourth edited image.

[0104] Step C4: For any first target attribute in the attribute set to be processed, determine the attribute change corresponding to each target semantic direction according to the attribute values ​​of each first edited image and each second edited image under the first target attribute, and the first target attribute includes at least one of the following: foam size, foam color and foam shape.

[0105] Step C5: Determine the semantic direction corresponding to the maximum attribute change as the semantic direction to be processed; and determine the first target attribute as the attribute label of the semantic direction to be processed.

[0106] Step C6: Acquire the target operation adjustment data set and the target image before operation.

[0107] Step C7: For any target operation adjustment data in the target operation adjustment data set, use the trained foam image prediction model to process the target operation adjustment data and the image before the target operation to obtain the target image editing amplitude vector and the image after the target operation corresponding to the target operation adjustment data.

[0108] Step C8: Calculate the first attribute value of the image before the target operation under the second target attribute, and calculate the second attribute value set of each image after the target operation under the second target attribute, wherein the attribute value of the second target attribute is affected by the target operation adjustment data.

[0109] Step C9: Generate the law of how the operation adjustment data affects the foam state according to each target operation adjustment data, the target image editing amplitude vector, the preset semantic direction set, the first attribute value, and the second attribute value set.

[0110] In this embodiment, during specific implementation, for example, the aforementioned trained generative adversarial network can be used to process a random latent vector in the latent space to obtain a foam image, and this image is used as the image to be edited. An image can also be selected from the historical real foam image captured by the camera as the image to be edited. In order to facilitate image editing, the acquired foam image to be edited can be first encoded using a pre-trained encoder to obtain a latent code to be edited. Then, along any target semantic direction in the preset semantic direction set, the preset third editing amplitude and the preset fourth editing amplitude are used to edit the latent code to be edited, respectively, to obtain the corresponding third edited latent code and fourth edited latent code; then, the third edited latent code and the fourth edited latent code are respectively input into the pre-trained generator to obtain the third edited image and the fourth edited image. Here, the preset third editing amplitude and the preset fourth editing amplitude can be set as needed, and the embodiment of the present disclosure is not limited.

[0111] For example, the preset third editing amplitude is 0.2, the preset fourth editing amplitude is -0.2, and the preset semantic direction set includes 2 semantic direction vectors and , we can first edit the latent code to be edited with an edit amplitude of 0.2 along the semantic direction of the semantic direction vector n1 to obtain a third edited latent code; and edit the latent code to be edited with an edit amplitude of -0.2 along the semantic direction to obtain a fourth edited latent code. Then, the third edited latent code and the fourth edited latent code are respectively input into the pre-trained generator to obtain the third edited image and the fourth edited image for the semantic direction vector n1. Similarly, the third edited image and the fourth edited image for the semantic direction vector n2 can be obtained. Figure 7 An example of image editing results based on the semantic direction of foam provided by an embodiment of the present disclosure is shown. The image to be edited is edited using editing amplitudes of -1.5, -0.75, 0, 0.75 and 1.5 respectively along the semantic direction corresponding to any attribute of the foam size, color and shape. The editing results are shown in FIG. Figure 7 As shown. Figure 7 This fully demonstrates the effectiveness of the bubble semantic direction identified in the latent space.

[0112] Then, for any first target attribute in the attribute set to be processed, the attribute change corresponding to each target semantic direction is determined according to the attribute value of each third edited image and each fourth edited image under the first target attribute. The first target attribute includes but is not limited to at least one of the following: foam size, foam color and foam shape.

[0113] Exemplarily, the attribute values ​​of the third edited image and the fourth edited image under the first target attribute corresponding to the semantic direction vector n1 can be calculated to obtain attribute values ​​M1 and M2, and the attribute values ​​of the third edited image and the fourth edited image under the first target attribute corresponding to the semantic direction vector n2 can be calculated to obtain attribute values ​​N1 and N2. Then, the difference between the attribute values ​​M1 and M2 is calculated. M, calculate the difference between attribute values ​​N1 and N2 N. M. N are the semantic direction vectors , semantic direction vector The corresponding attribute change of the first target attribute.

[0114] Next, the semantic direction corresponding to the largest attribute change is determined as the semantic direction to be processed, that is, in the semantic direction to be processed, the attribute change of the first target attribute is the most obvious, so the first target attribute is determined as the attribute label of the semantic direction to be processed.

[0115] Taking the foam size and foam color of the foam image as examples, for the foam size attribute, the watershed segmentation algorithm can be used to segment the foam image, and the average size of the foam in the foam image can be counted and obtained. Along all semantic directions, the average foam size changes of the edited image edited by the preset third editing amplitude and the edited image edited by the preset fourth editing amplitude are compared. The semantic direction corresponding to the edited image with the most obvious change in the average foam size has an attribute label of foam size. For example, Figure 8 A line graph showing the average foam size change of an edited image edited at an amplitude of 0.2 along 50 semantic directions relative to an edited image edited at an amplitude of -0.2 provided by an embodiment of the present disclosure. Figure 8 In , the horizontal axis is the direction, i.e. the semantic direction, and the vertical axis is the size, i.e. the average size of the foam. Figure 8It can be seen that among all semantic directions, in the second semantic direction, the average foam size change of the edited image edited by an amplitude of 0.2 is the largest relative to the edited image edited by an amplitude of -0.2. Therefore, the attribute label of the second semantic direction can be determined as the foam size. For the foam color attribute, since the brightness of the foam color in the foam image can reflect the quality, the V (brightness) channel mean in the HSV space of the foam image is extracted as the representation of the image color. Along all semantic directions, the V channel mean changes of the edited image edited by the preset third editing amplitude and the edited image edited by the preset fourth editing amplitude are compared. The semantic direction corresponding to the edited image with the most obvious V channel mean change has the attribute label of foam color. Fig. 9 A line graph showing the change in the mean value of the V channel of an edited image edited at an amplitude of 0.2 along 50 semantic directions relative to an edited image edited at an amplitude of -0.2 provided by an embodiment of the present disclosure is shown. Fig. 9 In , the horizontal axis is the direction, i.e. the semantic direction, and the vertical axis is the color value, i.e. the V channel color value. Fig. 9 It can be seen that among all semantic directions, in the fifth semantic direction, the V channel mean value of the edited image edited by an amplitude of 0.2 has the largest change relative to the edited image edited by an amplitude of -0.2. Therefore, the attribute label of the fifth semantic direction can be determined as foam color.

[0116] After labeling each semantic direction with attribute labels, the target operation adjustment data set and the image before the target operation can be obtained. Then, for any target operation adjustment data in the target operation adjustment data set, any of the aforementioned trained foam image prediction models can be used to process the target operation adjustment data and the image before the target operation to obtain the target image editing amplitude vector and the image after the target operation corresponding to the target operation adjustment data. Next, the first attribute value of the image before the target operation under the second target attribute is calculated, and the second attribute value set of each image after the target operation under the second target attribute is calculated. Among them, the attribute value of the second target attribute is affected by the target operation adjustment data. Exemplarily, the influence of the agent adjustment on the flotation foam is mainly reflected in the change of the foam size. Therefore, if the data in the target operation adjustment data set is an adjustment to the agent-related data, the corresponding second target attribute is the foam size attribute.

[0117] Finally, according to each target operation adjustment data, the target image editing amplitude vector, the preset semantic direction set, the first attribute value, and the second attribute value set, the law of how the operation adjustment data affects the foam state is generated. Fig.10 A diagram showing experimental results of mining the influence of operation adjustment on foam properties in an interpretable manner provided by an embodiment of the present disclosure is shown. Fig.10The result diagram shows the effect of adjusting the two agents, frother and lime, on the foam size properties. Fig.10 In the figure, the three columns on the left are the first group of results, the three columns on the right are the second group of results, the first row is the two groups of results when the frother is increased, the second row is the two groups of results when the frother is decreased, the third row is the two groups of results when the lime is increased, the fourth row is the two groups of results when the lime is decreased, the fifth row is the two groups of results when the frother is increased and the lime is decreased, and the sixth row is the two groups of results when the frother is decreased and the lime is increased. In each group of results, the left image is the pre-operation image, the middle image is the true value post-operation image, and the right image is the generated post-operation image. In addition, the vector under the pre-operation image in each group of results represents the amount of frother and lime added to the flotation tank before the operation adjustment, the vector under the true value post-operation image represents the amount of frother and lime added to the flotation tank after the operation adjustment, and the value under the generated post-operation image represents the edit amplitude value in the direction of the foam size attribute corresponding to the operation adjustment. According to the operation adjustment, the editing amplitude value corresponding to the foam size attribute, and the change of the foam image before and after the operation, the following rules are obtained: when the foaming agent is increased, the amplitude value of the foam size attribute direction is positive, and the foam becomes smaller; when the foaming agent is decreased, the amplitude value of the foam size attribute direction is negative, and the foam becomes larger. The larger the adjustment amount, the larger the corresponding absolute value of the amplitude value; when the lime is increased, the amplitude value of the foam size attribute direction is negative, and the foam becomes larger; when the lime is decreased, the amplitude value of the foam size attribute direction is positive, and the foam becomes smaller. The larger the adjustment amount, the larger the corresponding absolute value of the amplitude value; when the foaming agent is increased and the lime is decreased, the amplitude value of the foam size attribute direction is positive, and the foam becomes smaller; when the foaming agent is decreased and the lime is increased, the amplitude value of the foam size attribute direction is negative, and the foam becomes larger. The positive direction of the foam size attribute direction corresponds to smaller bubbles, and the negative direction corresponds to larger bubbles.

[0118] This embodiment attaches attribute labels to the semantic directions in the preset semantic direction set, and uses the aforementioned trained foam image prediction model to mine the influence of flotation operation adjustments on flotation state attributes in an explainable manner. This can enhance production personnel's understanding of the influence of operation adjustments on the various attribute changes of flotation foam states, and lay a foundation for guiding the subsequent intelligent optimization control of the flotation process.

[0119] Fig.11 A flow chart of a method for training an image prediction model provided by an embodiment of the present disclosure is shown. Fig.11 It can be seen that the embodiment of the present disclosure at least includes steps S1101-S1103:

[0120] S1101: Acquire a first data set, wherein any first target sample in the first data set is composed of a first sample pre-operation image and first sample operation adjustment data, and a label of the first target sample is a corresponding first sample post-operation image.

[0121] S1102: Construct a first foam image prediction model to be trained; wherein the first foam image prediction model to be trained is the first model to be trained or the second model to be trained; the first model to be trained includes a first encoder, a first generator and a first discriminator, the first encoder is used to process the first sample pre-operation image and the first sample operation adjustment data to obtain a first sample latent code in the latent space for representing the image style, and the first generator is used to process the first sample latent code to obtain a predicted foam image after the operation; the second model to be trained includes a second encoder, a second generator and a second discriminator, the second encoder is used to process the first sample pre-operation image to obtain a second sample latent code in the latent space for representing the image style, and the second generator is used to process the second sample latent code and the first sample operation adjustment data to obtain a predicted foam image after the operation.

[0122] S1103: using the first data set to train the first foam image prediction model to be trained to obtain a trained first foam image prediction model; obtaining the trained foam image prediction model according to the encoder and the generator in the trained first foam image prediction model.

[0123] The specific training process of the image prediction model provided in this embodiment can be found in the introduction and description in the above embodiment, which will not be repeated here.

[0124] Further, in order to better illustrate the process of the training method of the above-mentioned image prediction model, as a refinement and extension of the above-mentioned embodiment, the embodiment of the present invention provides several optional embodiments, but is not limited thereto, which are specifically as follows:

[0125] In a possible implementation, when the second model further includes a mapping network and an editing module, the second model is generated according to the following method: obtaining a second data set, a third data set and a fourth data set; wherein the second data set includes a plurality of real foam images, the third data set includes a plurality of foam images to be reconstructed, any second target sample in the fourth data set is composed of a second sample pre-operation image and second sample operation adjustment data, and the label of the second target sample is the corresponding second sample post-operation image; constructing an initial generative adversarial network, and using the second data set to train the initial generative adversarial network to obtain a trained generative adversarial network; wherein the initial generative adversarial network includes a conversion network, an initial generator and an initial discriminator, the conversion network is used to process a random latent vector to obtain a first target latent code, the initial generator is used to process the first target latent code, a constant input and random noise to obtain a generated foam image; constructing an initial image reconstruction model, the initial image reconstruction model includes: an initial encoder and a generator in a trained generative adversarial network, the initial encoder is used to encode the foam image to be reconstructed to obtain a second target latent code, the generator in the trained generative adversarial network Used to generate a reconstructed foam image based on a second target latent code, a constant input and random noise; use a third data set to train an initial image reconstruction model, update the weight of an initial encoder, and obtain a trained image reconstruction model; construct a second foam image prediction model to be trained, the second foam image prediction model to be trained includes a mapping network, a discriminator, an editing module, a pre-trained encoder and a pre-trained generator, wherein the mapping network is used to process the second sample operation adjustment data to generate a sample image editing amplitude vector, the pre-trained encoder and the pre-trained generator are the encoder and the generator in the trained image reconstruction model, the pre-trained encoder is used to process the second sample image before operation to obtain the corresponding third target latent code, the editing module is used to generate a sample edited latent code based on the sample image editing amplitude vector, a preset semantic direction set and the third target latent code; use a fourth data set to train the second foam image prediction model to be trained, update the weights of the mapping network and the discriminator, and obtain a trained second foam image prediction model; obtain the second model according to the mapping network, the editing module, the encoder and the generator in the trained second foam image prediction model.

[0126] The specific training process of the second model provided in this embodiment can be found in the introduction and description in the above embodiment, and will not be repeated here.

[0127] In a possible implementation manner, the second foam image prediction model to be trained is trained using the fourth data set according to any of the following methods, and the weights of the mapping network and the discriminator are updated to obtain a trained second foam image prediction model:

[0128] Method 1: The second sample pre-operation image and the second sample operation adjustment data of any third target sample in the fourth data set are respectively input into the pre-trained encoder and the mapping network to obtain the corresponding third target latent code and the sample image editing amplitude vector; along each semantic direction in the preset semantic direction set, the third target latent code is edited using the sample image editing amplitude vector corresponding to the third target sample to obtain the first sample edited latent code; the first sample edited latent code is processed using the pre-trained generator to obtain the first predicted image after operation; according to the preset loss function, the first predicted image after operation and the second sample foam image after operation of the third target sample, the weights of the mapping network and the discriminator are updated to finally obtain a trained second foam image prediction model.

[0129] Method 2: Input the second sample pre-operation image and the second sample operation adjustment data of any third target sample in the fourth data set into the pre-trained encoder and the mapping network respectively, and obtain the corresponding third target latent code and sample image editing amplitude vector respectively; select any unused semantic direction from the preset semantic direction set as the semantic direction for image editing; along the semantic direction for image editing, edit the third target latent code using the corresponding editing amplitude in the sample image editing amplitude vector of the third target sample to obtain the second sub-sample edited latent code; use the pre-trained generator to process the second sub-sample edited latent code to obtain the second sub-prediction operation image; update the mapping network, according to the preset loss function, the second sub-prediction operation image and the second sample operation foam image of the third target sample. The weight of the discriminator is obtained, and the step of selecting any unadopted semantic direction from the preset semantic direction set as the semantic direction for image editing is returned, until there is no unadopted semantic direction in the preset semantic direction set, and the intermediate second foam image prediction model is obtained; along each semantic direction in the preset semantic direction set, the third target latent code is edited using the sample image editing amplitude vector corresponding to the third target sample to obtain the edited latent code of the second sample; the edited latent code of the second sample is processed using the pre-trained generator to obtain the second predicted image after operation; according to the preset loss function, the second predicted image after operation and the foam image of the second sample after operation of the third target sample, the weights of the mapping network and the discriminator in the intermediate second foam image prediction model are updated, and finally the trained second foam image prediction model is obtained.

[0130] The specific training method of the trained second foam image prediction model provided in this embodiment can be found in the description in the above embodiment, and will not be repeated here.

[0131] In a possible implementation, the loss function used to train the initial generative adversarial network includes a generator loss function and a first discriminator loss function; the generator loss function is:

[0132]

[0133] The first discriminator loss function is:

[0134]

[0135]

[0136] in, is the generator loss function, E represents the expectation, x' is the generated foam image, represents the i-th generated foam image, To generate the probability distribution of foam images, is the probability that the discriminator considers the i-th generated foam image to be real data, is the first discriminator loss function, x represents the real foam image, represents the probability distribution of the real foam image, is the probability that the discriminator considers the i-th real foam image to be real data, r is the weight coefficient of the regularization loss term, Used to calculate the gradient of the discriminator with respect to its weight parameters.

[0137] In one possible implementation, the initial encoder adopts a feature pyramid network structure.

[0138] Fig.12 FIG. 2 shows a schematic diagram of the structure of the initial encoder provided by an embodiment of the present disclosure. Fig.12 As shown, in this embodiment, the initial encoder adopts a model structure based on a feature pyramid network, which can transform the input foam image into multiple levels of latent codes w with the help of pyramid networks and convolution blocks of different scales, and then input them into a pre-trained generator to obtain a reconstructed image.

[0139] In a possible implementation, the reconstruction loss function used to train the initial image reconstruction model is:

[0140]

[0141] in, represents the reconstruction loss function, is the pixel-level image similarity loss term, is the feature-level image similarity loss term, , is the weight coefficient corresponding to each loss item.

[0142] In specific implementation, for example, It can be determined according to the following formula:

[0143]

[0144] in, is the foam image to be reconstructed, Reconstructed foam image generated for the model.

[0145] For example, it can be determined by the following formula:

[0146]

[0147] It can also be determined according to the following formula:

[0148]

[0149] in, , Indicates two feature-level image similarity loss terms, F() and C() are preset feature extraction models, F() is, for example, a VGG model, an AlexNet model, etc., and C() is, for example, a residual network (ResNet-50 model), etc. This disclosure embodiment is not limited to this.

[0150] Exemplarily, the reconstruction loss function may also be determined according to the following formula:

[0151]

[0152] in, and ( + ) is 1, , , , It can be set according to actual needs, and this embodiment of the present disclosure is not limited to this.

[0153] The loss function provided in this embodiment is composed of two different levels of loss terms, and is adjusted by weight coefficients, so that the model can pay attention to the similarities of both pixel level and feature level during the training process, thereby improving the overall quality and fidelity of the reconstructed image. In addition, by adjusting the weight coefficients, the importance of different loss terms can be balanced according to the requirements of specific tasks.

[0154] In a possible implementation manner, the loss function used to train the second foam image prediction model to be trained includes a mapping network loss function and a second discriminator loss function; the mapping network loss function is:

[0155]

[0156] in, represents the mapping network loss function, represents the pixel-level image similarity loss term, represents the feature-level image similarity loss term, Represents the loss term of the discriminator’s discrimination result on the generated graph, , , is the weight coefficient of each loss item; the loss function of the second discriminator is:

[0157]

[0158]

[0159] in, is the second discriminator loss function, Express expectations, represents the image after the second sample operation, represents the probability distribution of the image after the second sample operation, represents the encoder, represents the discriminator, represents the post-operation foam image generated by the second foam image prediction model generator, represents the probability distribution of the post-operation foam image generated by the second foam image prediction model generator, represents the image after the i-th second sample operation, represents the i-th post-operation foam image generated by the second foam image prediction model generator, r is the weight coefficient of the regularization loss term, Used to calculate the gradient of the discriminator with respect to its weight parameters.

[0160] Here, in specific implementation, For example, it can be determined by the following formula:

[0161]

[0162] For example, it can be determined by the following formula:

[0163]

[0164] It can also be determined according to the following formula:

[0165]

[0166] It can also be determined according to the following formula:

[0167]

[0168] The loss term of the discriminator's discrimination result on the generated graph can be determined, for example, according to the following formula:

[0169]

[0170] in, is the latent code corresponding to the foam image after the i-th second sample operation, The latent code corresponding to the i-th post-operation foam image generated by the second foam image prediction model generator.

[0171] Exemplarily, the mapping network loss function can also be determined according to the following formula:

[0172]

[0173] in, , , The sum of is 1, , , It can be set according to actual needs, and this embodiment of the present disclosure is not limited to this.

[0174] The generator loss function in this embodiment is composed of two different levels of loss terms, and is adjusted by weight coefficients, so that the model can pay attention to the similarities of both pixel level and feature level during training. In addition, the discriminator adds a loss term for the discriminant result of the generated image, thereby realizing adversarial training and significantly improving the overall quality, accuracy and realism of the predicted foam image after operation. In addition, by adjusting the weight coefficients, the importance of different loss terms can be balanced according to the needs of specific tasks.

[0175] The present disclosure also provides an image prediction method, comprising the following steps:

[0176] Step S1: Obtain a second data set, a third data set, and a fourth data set. The second data set includes a plurality of real foam images, the third data set includes a plurality of foam images to be reconstructed, any second target sample in the fourth data set is composed of a second sample pre-operation image and second sample operation adjustment data, and the label of the second target sample is the corresponding second sample post-operation image.

[0177] Step S2: construct an initial generative adversarial network, and use the second data set to train the initial generative adversarial network to obtain a trained generative adversarial network; wherein the initial generative adversarial network includes a conversion network, an initial generator and an initial discriminator, the conversion network is used to process the random latent vector to obtain a first target latent code, and the initial generator is used to process the first target latent code, constant input and random noise to obtain a generated foam image. In a possible implementation, the loss function used to train the initial generative adversarial network includes a generator loss function and a first discriminator loss function; the generator loss function is:

[0178]

[0179] The first discriminator loss function is:

[0180]

[0181]

[0182] Step S3: construct an initial image reconstruction model, the initial image reconstruction model includes: an initial encoder and a generator in a trained generative adversarial network, the initial encoder is used to encode the foam image to be reconstructed to obtain a second target latent code, and the generator in the trained generative adversarial network is used to generate a reconstructed foam image based on the second target latent code, a constant input and random noise. Preferably, the initial encoder adopts a feature pyramid network structure. The reconstruction loss function used to train the initial image reconstruction model is:

[0183]

[0184] Step S4: using the third data set to train the initial image reconstruction model, updating the weights of the initial encoder, and obtaining a trained image reconstruction model.

[0185] Step S5: construct a second foam image prediction model to be trained, the second foam image prediction model to be trained includes a mapping network, a discriminator, an editing module, a pre-trained encoder and a pre-trained generator, wherein the mapping network is used to process the second sample operation adjustment data to generate a sample image editing amplitude vector, the pre-trained encoder and the pre-trained generator are the encoder and the generator in the trained image reconstruction model, the pre-trained encoder is used to process the second sample pre-operation image to obtain the corresponding third target latent code, and the editing module is used to generate a sample edited latent code based on the sample image editing amplitude vector, a preset semantic direction set and the third target latent code. Preferably, the preset semantic direction set is generated according to the following method: extracting weight parameter data of each layer from the generator of the trained generative adversarial network to obtain a weight matrix; constructing a semantic direction optimization model, and solving the semantic direction optimization model to obtain a candidate semantic direction set; for any target candidate semantic direction in the candidate semantic direction set, along the target candidate semantic direction, respectively using a preset first editing amplitude and a preset second editing amplitude, the acquired target to-be-edited latent code is edited to obtain a corresponding first edited latent code and a second edited latent code; respectively inputting the first edited latent code and the second edited latent code into the generator of the trained generative adversarial network to obtain a first edited image and a second edited image; calculating similarity data between the first edited image and the second edited image; based on each similarity data, eliminating elements in the candidate semantic direction set to obtain a candidate semantic direction set; wherein the semantic direction optimization model is:

[0186]

[0187] Step S6: Use the fourth data set to train the second foam image prediction model to be trained, update the weights of the mapping network and the discriminator, and obtain the trained second foam image prediction model; according to the mapping network, editing module, encoder and generator in the trained second foam image prediction model, obtain the second model. Preferably, the trained second foam image prediction model can be obtained by training in the following manner: input the second sample pre-operation image and the second sample operation adjustment data of any third target sample in the fourth data set into the pre-trained encoder and the mapping network respectively, and obtain the corresponding third target latent code and sample image editing amplitude vector respectively; select any unused semantic direction from the preset semantic direction set as the semantic direction for image editing; along the semantic direction for image editing, use the corresponding editing amplitude in the sample image editing amplitude vector of the third target sample to edit the third target latent code to obtain the second sub-sample edited latent code; use the pre-trained generator to process the second sub-sample edited latent code to obtain the second sub-prediction operation image; according to the preset loss function, the second sub-prediction operation image and the second sample operation of the third target sample The method further comprises the following steps: extracting the post-operation foam image, updating the weights of the mapping network and the discriminator, and returning to the step of selecting any unadopted semantic direction from the preset semantic direction set as the semantic direction for image editing, until there is no unadopted semantic direction in the preset semantic direction set, and obtaining an intermediate second foam image prediction model; along each semantic direction in the preset semantic direction set, using the sample image editing amplitude vector corresponding to the third target sample, editing the third target latent code to obtain the second sample edited latent code; processing the second sample edited latent code using the pre-trained generator to obtain the second predicted image after operation; updating the weights of the mapping network and the discriminator in the intermediate second foam image prediction model according to the preset loss function, the second predicted image after operation and the second sample operated foam image of the third target sample, and finally obtaining a trained second foam image prediction model.

[0188] Preferably, the loss function used to train the second foam image prediction model to be trained includes a mapping network loss function and a second discriminator loss function; the mapping network loss function is:

[0189]

[0190] The loss function of the second discriminator is:

[0191]

[0192]

[0193] Step S7: acquiring the foam image before the operation and the operation adjustment data affecting the foam image before the operation in real time.

[0194] Step S8: using the second model, processing the foam image before the operation and the operation adjustment data, and predicting the foam image after the operation.

[0195] Step S9: Use the pre-trained encoder to process the image to be edited to obtain the latent code to be edited.

[0196] Step S10: along any target semantic direction in the preset semantic direction set, the preset third editing amplitude and the preset fourth editing amplitude are used to edit the latent code to obtain the corresponding third edited latent code and fourth edited latent code.

[0197] Step S11: input the third edited latent code and the fourth edited latent code into the pre-trained generator respectively to obtain the third edited image and the fourth edited image.

[0198] Step S12: Based on the generator in the trained generative adversarial network, any target semantic direction in the preset semantic direction set and the corresponding target editing amplitude, the image to be edited is processed to obtain an edited image.

[0199] Step S13: For any first target attribute in the attribute set to be processed, determine the attribute change corresponding to each target semantic direction according to the attribute values ​​of each third edited image and each fourth edited image under the first target attribute, and the first target attribute includes: foam size, foam color and foam shape.

[0200] Step S14: Determine the semantic direction corresponding to the maximum attribute change as the semantic direction to be processed; and determine the first target attribute as the attribute label of the semantic direction to be processed.

[0201] Step S15: Acquire a target operation adjustment data set and an image before the target operation.

[0202] Step S16: For any target operation adjustment data in the target operation adjustment data set, use the trained foam image prediction model to process the target operation adjustment data and the image before the target operation to obtain the target image editing amplitude vector and the image after the target operation corresponding to the target operation adjustment data.

[0203] Step S17: calculating the first attribute value of the image before the target operation under the second target attribute, and calculating the second attribute value set of each image after the target operation under the second target attribute; wherein the attribute value of the second target attribute is affected by the target operation adjustment data.

[0204] Step S18: Generate a rule of how the operation adjustment data affects the foam state according to each target operation adjustment data, the target image editing amplitude vector, the preset semantic direction set, the first attribute value, and the second attribute value set.

[0205] Those skilled in the art will appreciate that, in the above method of the specific implementation manner, the writing order of the steps does not mean a strict execution order, but does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0206] It should be noted that, in practical applications, all possible implementations described above can be combined in any manner to form possible embodiments of the present disclosure, which will not be described one by one here.

[0207] Based on the same concept, the embodiment of the present disclosure also provides an image prediction device. Fig.13 FIG. 1 shows a schematic diagram of the structure of an image prediction device provided by an embodiment of the present disclosure, see Fig.13 As shown, the image prediction device 1300 provided by the embodiment of the present disclosure includes:

[0208] The first acquisition module 1301 is used to acquire in real time the foam image before operation and the operation adjustment data affecting the foam image before operation;

[0209] The prediction module 1302 is used to use the trained foam image prediction model to process the pre-operation foam image and the operation adjustment data to predict the post-operation foam image; wherein the trained foam image prediction model is a first model or a second model; the first model includes a first encoder and a first generator, the first encoder is used to process the pre-operation foam image and the operation adjustment data to obtain a first latent code in a latent space for representing the image style, and the first generator is used to process the first latent code to obtain the post-operation foam image; the second model includes a second encoder and a second generator, the second encoder is used to process the pre-operation foam image to obtain a second latent code in a latent space for representing the image style, and the second generator is used to process the second latent code and the operation adjustment data to obtain the post-operation foam image.

[0210] In a possible implementation manner, the device further includes a first training module, which is used to: obtain a first data set, wherein any first target sample in the first data set is composed of a first sample pre-operation image and first sample operation adjustment data, and the label of the first target sample is the corresponding first sample post-operation image; construct a first foam image prediction model to be trained; wherein the first foam image prediction model to be trained is the first model to be trained or the second model to be trained; the first model to be trained includes a first encoder, a first generator and a first discriminator, wherein the first encoder is used to process the first sample pre-operation image and the first sample operation adjustment data to obtain a first sample latent code in a latent space for characterizing an image style, and the first generator The encoder is used to process the first sample latent code to obtain a predicted foam image after operation; the second model to be trained includes a second encoder, a second generator and a second discriminator, the second encoder is used to process the first sample pre-operation image to obtain a second sample latent code for characterizing the image style in the latent space, and the second generator is used to process the second sample latent code and the first sample operation adjustment data to obtain a predicted foam image after operation; the first foam image prediction model to be trained is trained using the first data set to obtain a trained first foam image prediction model; the trained foam image prediction model is obtained according to the encoder and generator in the trained first foam image prediction model.

[0211] In a possible implementation, in the above-mentioned device, the trained foam image prediction model is the second model; the second model also includes a mapping network and an editing module; the mapping network is used to process the operation adjustment data to generate a corresponding image editing amplitude vector; the editing module is used to edit the second latent code according to the image editing amplitude vector and a preset semantic direction set, generate an edited latent code, and input it into the second generator.

[0212] In a possible implementation, the device further includes a second training module, which is used to: obtain a second data set, a third data set, and a fourth data set; wherein the second data set includes a plurality of real foam images, the third data set includes a plurality of foam images to be reconstructed, any second target sample in the fourth data set is composed of a second sample pre-operation image and a second sample operation adjustment data, and the label of the second target sample is the corresponding second sample post-operation image; construct an initial generative adversarial network, and train the initial generative adversarial network using the second data set to obtain a trained generative adversarial network; wherein the initial generative adversarial network includes a conversion network, an initial generator, and an initial discriminator, the conversion network is used to process a random latent vector to obtain a first target latent code, the initial generator is used to process the first target latent code, a constant input, and random noise to obtain a generated foam image; construct an initial image reconstruction model, the initial image reconstruction model includes: an initial encoder and a generator in the trained generative adversarial network, the initial encoder is used to encode the foam image to be reconstructed to obtain a second target latent code, the generator in the trained generative adversarial network is used to generate a foam image based on the second target latent code, Constant input and random noise are used to generate a reconstructed foam image; the initial image reconstruction model is trained using the third data set, and the weight of the initial encoder is updated to obtain a trained image reconstruction model; a second foam image prediction model to be trained is constructed, and the second foam image prediction model to be trained includes a mapping network, a discriminator, an editing module, a pre-trained encoder and a pre-trained generator, wherein the mapping network is used to process the second sample operation adjustment data to generate a sample image editing amplitude vector, the pre-trained encoder and the pre-trained generator are the encoder and generator in the trained image reconstruction model, the pre-trained encoder is used to process the second sample pre-operation image to obtain the corresponding third target latent code, and the editing module is used to generate a sample edited latent code based on the sample image editing amplitude vector, a preset semantic direction set and the third target latent code; the second foam image prediction model to be trained using the fourth data set, and the weights of the mapping network and the discriminator are updated to obtain a trained second foam image prediction model; the second model is obtained according to the mapping network, the editing module, the encoder and the generator in the trained second foam image prediction model.

[0213] In a possible implementation manner, in the above-mentioned device, the preset semantic direction set is generated according to the following method: extracting weight parameter data of each layer from the generator of the trained generative adversarial network to obtain a weight matrix; constructing a semantic direction optimization model, and solving the semantic direction optimization model to obtain a candidate semantic direction set; for any target candidate semantic direction in the candidate semantic direction set, along the target candidate semantic direction, respectively using a preset first editing amplitude and a preset second editing amplitude, the acquired target to-be-edited latent code is edited to obtain a corresponding first edited latent code and a second edited latent code; respectively inputting the first edited latent code and the second edited latent code into the generator of the trained generative adversarial network to obtain a first edited image and a second edited image; calculating similarity data between the first edited image and the second edited image; based on each of the similarity data, eliminating elements in the candidate semantic direction set to obtain the preset semantic direction set; wherein the semantic direction optimization model is:

[0214]

[0215] in, is the candidate semantic direction set, It is used to calculate the independent variable or independent variable set corresponding to the maximum value of the function. is a d×k real number matrix representing the set of semantic directions, represents the i-th semantic direction vector, for The transpose of is the number of semantic directions, d is the dimension of each semantic direction vector, and A is the weight matrix.

[0216] In a possible implementation manner, in the above-mentioned device, the second training module is used to: input the second sample pre-operation image and the second sample operation adjustment data of any third target sample in the fourth data set into the pre-trained encoder and the mapping network respectively to obtain the corresponding third target latent code and the sample image editing amplitude vector; along each semantic direction in the preset semantic direction set, using the sample image editing amplitude vector corresponding to the third target sample, edit the third target latent code to obtain the first sample edited latent code; use the pre-trained generator to process the first sample edited latent code to obtain the first predicted operation image; according to the preset loss vector, the first predicted latent code is processed by the pre-trained generator to obtain the first predicted operation image. The loss function, the first predicted image after operation and the foam image after the second sample operation of the third target sample are used to update the weights of the mapping network and the discriminator, and finally the trained second foam image prediction model is obtained; or, the second sample pre-operation image and the second sample operation adjustment data of any third target sample in the fourth data set are respectively input into the pre-trained encoder and the mapping network to obtain the corresponding third target latent code and sample image editing amplitude vector respectively; any unused semantic direction is selected from the preset semantic direction set as the semantic direction for image editing; along the semantic direction for image editing, the third target sample is used to generate the image of the third target sample. The method further comprises the steps of: editing the third target latent code according to the edit amplitude corresponding to the edit amplitude in the sample image edit amplitude vector of the target sample, and obtaining the second sub-sample edited latent code; processing the second sub-sample edited latent code using the pre-trained generator to obtain the second sub-prediction operated image; updating the weights of the mapping network and the discriminator according to the preset loss function, the second sub-prediction operated image and the second sample operated foam image of the third target sample, and returning to the step of selecting any unadopted semantic direction from the preset semantic direction set as the semantic direction for image editing, until there is no unadopted semantic direction in the preset semantic direction set. semantic direction, and obtain an intermediate second foam image prediction model; along each semantic direction in the preset semantic direction set, the third target latent code is edited using the sample image editing amplitude vector corresponding to the third target sample to obtain the second sample edited latent code; the second sample edited latent code is processed using the pre-trained generator to obtain a second predicted image after operation; according to the preset loss function, the second predicted image after operation and the second sample operated foam image of the third target sample, the weights of the mapping network and the discriminator in the intermediate second foam image prediction model are updated to finally obtain the trained second foam image prediction model.

[0217] In a possible implementation manner, if the trained second foam image prediction model is obtained by training according to the second method; the above-mentioned device also includes a flotation operation rule mining module, which is used to: use the pre-trained encoder to process the image to be edited to obtain the latent code to be edited; along any target semantic direction in the preset semantic direction set, use the preset third editing amplitude and the preset fourth editing amplitude to edit the latent code to be edited respectively, to obtain the corresponding third edited latent code and fourth edited latent code; input the third edited latent code and the fourth edited latent code into the pre-trained generator respectively to obtain the third edited image and the fourth edited image; for any first target attribute in the attribute set to be processed, according to the attribute values ​​of each of the third edited images and each of the fourth edited images under the first target attribute, determine the attribute change corresponding to each of the target semantic directions, the first target attribute includes at least one of the following: foam size, foam color and foam shape; the largest attribute The semantic direction corresponding to the property change amount is determined as the semantic direction to be processed; and the first target attribute is determined as the attribute label of the semantic direction to be processed; a target operation adjustment data set and a target pre-operation image are obtained; for any target operation adjustment data in the target operation adjustment data set, the target operation adjustment data and the target pre-operation image are processed using the trained foam image prediction model to obtain a target image editing amplitude vector and a target post-operation image corresponding to the target operation adjustment data; a first attribute value of the target pre-operation image under a second target attribute is calculated, and a second attribute value set of each target post-operation image under the second target attribute is calculated; wherein the attribute value of the second target attribute is affected by the target operation adjustment data; and a law of how the operation adjustment data affects the foam state is generated according to each of the target operation adjustment data, the target image editing amplitude vector, the preset semantic direction set, the first attribute value, and the second attribute value set.

[0218] It should be noted that any of the above-mentioned image prediction devices can implement the above-mentioned image prediction method one-to-one, which will not be repeated here.

[0219] Based on the same concept, the embodiment of the present disclosure also provides a training device for an image prediction model. Fig.14 A schematic diagram of the structure of the training device of the image prediction model provided by the embodiment of the present disclosure is shown in FIG. Fig.14 As shown, the image prediction model training device 1400 provided in the embodiment of the present disclosure includes:

[0220] The second acquisition module 1401 is used to acquire a first data set, wherein any first target sample in the first data set is composed of a first sample pre-operation image and first sample operation adjustment data, and the label of the first target sample is the corresponding first sample post-operation image;

[0221] A construction module 1402 is used to construct a first foam image prediction model to be trained; wherein the first foam image prediction model to be trained is the first model to be trained or the second model to be trained; the first model to be trained includes a first encoder, a first generator and a first discriminator, the first encoder is used to process the first sample pre-operation image and the first sample operation adjustment data to obtain a first sample latent code in a latent space for characterizing an image style, and the first generator is used to process the first sample latent code to obtain a predicted foam image after operation; the second model to be trained includes a second encoder, a second generator and a second discriminator, the second encoder is used to process the first sample pre-operation image to obtain a second sample latent code in a latent space for characterizing an image style, and the second generator is used to process the second sample latent code and the first sample operation adjustment data to obtain a predicted foam image after operation;

[0222] The training module 1403 is used to train the first foam image prediction model to be trained using the first data set to obtain a trained first foam image prediction model; and obtain a trained foam image prediction model based on the encoder and generator in the trained first foam image prediction model.

[0223] In a possible implementation manner, when the second model further includes a mapping network and an editing module, the device further includes a third training module, which is used to: obtain a second data set, a third data set and a fourth data set; wherein the second data set includes a plurality of real foam images, the third data set includes a plurality of foam images to be reconstructed, any second target sample in the fourth data set is composed of a second sample pre-operation image and a second sample operation adjustment data, and the label of the second target sample is the corresponding second sample post-operation image; construct an initial generative adversarial network, and train the initial generative adversarial network using the second data set to obtain a trained generative adversarial network; wherein the initial generative adversarial network includes a conversion network, an initial generator and an initial discriminator, the conversion network is used to process a random latent vector to obtain a first target latent code, the initial generator is used to process the first target latent code, a constant input and random noise to obtain a generated foam image; construct an initial image reconstruction model, the initial image reconstruction model includes: an initial encoder and a generator in the trained generative adversarial network, the initial encoder is used to encode the foam image to be reconstructed to obtain a second target latent code, the generator in the trained generative adversarial network is used to base The invention relates to a method for generating a reconstructed foam image based on the second target latent code, a constant input and random noise; the initial image reconstruction model is trained using the third data set, and the weight of the initial encoder is updated to obtain a trained image reconstruction model; a second foam image prediction model to be trained is constructed, and the second foam image prediction model to be trained includes a mapping network, a discriminator, an editing module, a pre-trained encoder and a pre-trained generator, wherein the mapping network is used to process the second sample operation adjustment data to generate a sample image editing amplitude vector, the pre-trained encoder and the pre-trained generator are the encoder and the generator in the trained image reconstruction model, the pre-trained encoder is used to process the second sample pre-operation image to obtain the corresponding third target latent code, and the editing module is used to generate a sample edited latent code based on the sample image editing amplitude vector, a preset semantic direction set and the third target latent code; the second foam image prediction model to be trained is trained using the fourth data set, and the weights of the mapping network and the discriminator are updated to obtain a trained second foam image prediction model; the second model is obtained according to the mapping network, the editing module, the encoder and the generator in the trained second foam image prediction model.

[0224] In a possible implementation manner, in the above-mentioned device, the third training module is used to: input the second sample pre-operation image and the second sample operation adjustment data of any third target sample in the fourth data set into the pre-trained encoder and the mapping network respectively to obtain the corresponding third target latent code and the sample image editing amplitude vector; along each semantic direction in the preset semantic direction set, using the sample image editing amplitude vector corresponding to the third target sample, edit the third target latent code to obtain the first sample edited latent code; use the pre-trained generator to process the first sample edited latent code to obtain the first predicted image after operation; according to the preset The loss function, the first predicted image after operation and the foam image after the second sample operation of the third target sample are used to update the weights of the mapping network and the discriminator, and finally the trained second foam image prediction model is obtained; or, the second sample pre-operation image and the second sample operation adjustment data of any third target sample in the fourth data set are respectively input into the pre-trained encoder and the mapping network to obtain the corresponding third target latent code and sample image editing amplitude vector respectively; any unused semantic direction is selected from the preset semantic direction set as the semantic direction for image editing; along the semantic direction for image editing, the third The method further comprises the steps of: editing the third target latent code according to the corresponding editing amplitude in the sample image editing amplitude vector of the target sample to obtain the second sub-sample edited latent code; processing the second sub-sample edited latent code using the pre-trained generator to obtain the second sub-prediction operated image; updating the weights of the mapping network and the discriminator according to the preset loss function, the second sub-prediction operated image and the second sample operated foam image of the third target sample, and returning to the step of selecting any unadopted semantic direction from the preset semantic direction set as the semantic direction for image editing until there is no unadopted semantic direction in the preset semantic direction set. semantic direction, and obtain an intermediate second foam image prediction model; along each semantic direction in the preset semantic direction set, the third target latent code is edited using the sample image editing amplitude vector corresponding to the third target sample to obtain the second sample edited latent code; the second sample edited latent code is processed using the pre-trained generator to obtain a second predicted image after operation; according to the preset loss function, the second predicted image after operation and the second sample operated foam image of the third target sample, the weights of the mapping network and the discriminator in the intermediate second foam image prediction model are updated to finally obtain the trained second foam image prediction model.

[0225] In a possible implementation, in the above device, the loss function used to train the initial generative adversarial network includes a generator loss function and a first discriminator loss function; the generator loss function is:

[0226]

[0227] The first discriminator loss function is:

[0228]

[0229]

[0230] in, is the generator loss function, E represents the expectation, x' is the generated foam image, represents the i-th generated foam image, is the probability distribution of generating the foam image, is the probability that the discriminator considers the i-th generated foam image to be real data, is the first discriminator loss function, x represents the real foam image, represents the probability distribution of the true foam image, is the probability that the discriminator considers the i-th real foam image to be real data, r is the weight coefficient of the regularization loss term, Used to calculate the gradient of the discriminator with respect to its weight parameters.

[0231] In a possible implementation, in the above device, the initial encoder adopts a feature pyramid network structure.

[0232] In a possible implementation, in the above device, the reconstruction loss function used to train the initial image reconstruction model is:

[0233]

[0234] in, is the reconstruction loss function, is the pixel-level image similarity loss term, is the feature-level image similarity loss term, , is the weight coefficient corresponding to each loss item.

[0235] In a possible implementation, in the above device, the loss function used to train the second foam image prediction model to be trained includes a mapping network loss function and a second discriminator loss function; the mapping network loss function is:

[0236]

[0237] in, represents the mapping network loss function, represents the pixel-level image similarity loss term, represents the feature-level image similarity loss term, Represents the loss term of the discriminator’s discrimination result on the generated graph, , , is the weight coefficient of each loss term; the second discriminator loss function is:

[0238]

[0239]

[0240] in, is the second discriminator loss function, Express expectations, represents the image after the second sample operation, represents the probability distribution of the image after the second sample operation, represents the encoder, represents the discriminator, represents the post-operation foam image generated by the second foam image prediction model generator, represents the probability distribution of the post-operation foam image generated by the second foam image prediction model generator, represents the image after the i-th second sample operation, represents the i-th post-operation foam image generated by the second foam image prediction model generator, r is the weight coefficient of the regularization loss term, Used to calculate the gradient of the discriminator with respect to its weight parameters.

[0241] It should be noted that any of the above-mentioned image prediction model training devices can implement the above-mentioned image prediction model training method in a one-to-one correspondence, which will not be repeated here.

[0242] Fig.15 FIG. 1 shows a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Fig.15 As shown, at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage, etc. Of course, the electronic device may also include hardware required for other services.

[0243] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.15 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0244] The memory is used to store programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0245] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming an image prediction device / image prediction model training device at the logical level. The processor executes the program stored in the memory and is specifically used to execute the above method.

[0246] The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0247] The electronic device can execute the image prediction method provided by the multiple embodiments of the present disclosure, and realize the image prediction device in Fig.13 Alternatively, the training method of the image prediction model provided by the multiple embodiments of the present disclosure may be executed and implemented as a training device for the image prediction model. Fig.14 The functions of the illustrated embodiments will not be described in detail in the embodiments of the present disclosure.

[0248] The embodiments of the present disclosure also propose a computer-readable storage medium, which stores one or more programs, and the one or more programs include instructions. When the instructions are executed by an electronic device including multiple application programs, the electronic device can execute the image prediction method or the image prediction model training method provided by multiple embodiments of the present disclosure.

[0249] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0250] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0251] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0252] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0253] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0254] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0255] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0256] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity or device including the element.

[0257] It should be understood by those skilled in the art that the embodiments of the present disclosure may be provided as methods, systems or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0258] The above are only embodiments of the present disclosure and are not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. An image prediction method, characterized in that: The method comprises: Real-time acquisition of a pre-operation foam image and operation adjustment data affecting the pre-operation foam image; Using the trained foam image prediction model, the foam image before the operation and the operation adjustment data are processed to predict the foam image after the operation; Among them, the trained foam image prediction model is the first model or the second model; the first model includes a first encoder and a first generator, the first encoder is used to process the foam image before operation and the operation adjustment data to obtain a first latent code in the latent space for representing the image style, and the first generator is used to process the first latent code to obtain the foam image after operation; the second model includes a second encoder and a second generator, the second encoder is used to process the foam image before operation to obtain a second latent code in the latent space for representing the image style, and the second generator is used to process the second latent code and the operation adjustment data to obtain the foam image after operation.

2. The method according to claim 1, characterized in that Before the step of using the trained foam image prediction model to process the foam image before the operation and the operation adjustment data to predict the foam image after the operation, the method further includes: Acquire a first data set, wherein any first target sample in the first data set is composed of a first sample pre-operation image and first sample operation adjustment data, and a label of the first target sample is a corresponding first sample post-operation image; Constructing a first foam image prediction model to be trained; wherein the first foam image prediction model to be trained is the first model to be trained or the second model to be trained; the first model to be trained includes a first encoder, a first generator and a first discriminator, the first encoder is used to process the first sample pre-operation image and the first sample operation adjustment data to obtain a first sample latent code in a latent space for characterizing an image style, and the first generator is used to process the first sample latent code to obtain a predicted foam image after operation; the second model to be trained includes a second encoder, a second generator and a second discriminator, the second encoder is used to process the first sample pre-operation image to obtain a second sample latent code in a latent space for characterizing an image style, and the second generator is used to process the second sample latent code and the first sample operation adjustment data to obtain a predicted foam image after operation; The first foam image prediction model to be trained is trained using the first data set to obtain a trained first foam image prediction model; and the trained foam image prediction model is obtained according to the encoder and the generator in the trained first foam image prediction model.

3. The method according to claim 1, characterized in that The trained foam image prediction model is the second model; the second model also includes a mapping network and an editing module; The mapping network is used to process the operation adjustment data to generate a corresponding image editing amplitude vector; The editing module is used to edit the second latent code according to the image editing amplitude vector and a preset semantic direction set, generate an edited latent code, and input the edited latent code into the second generator.

4. The method according to claim 3, characterized in that The second model including the mapping network and the editing module is generated by training according to the following method: Acquire a second data set, a third data set, and a fourth data set; wherein the second data set includes a plurality of real foam images, the third data set includes a plurality of foam images to be reconstructed, any second target sample in the fourth data set is composed of a second sample pre-operation image and second sample operation adjustment data, and the label of the second target sample is the corresponding second sample post-operation image; Constructing an initial generative adversarial network, and training the initial generative adversarial network using the second data set to obtain a trained generative adversarial network; wherein the initial generative adversarial network includes a conversion network, an initial generator, and an initial discriminator, the conversion network is used to process the random latent vector to obtain a first target latent code, and the initial generator is used to process the first target latent code, a constant input, and random noise to obtain a generated foam image; Constructing an initial image reconstruction model, the initial image reconstruction model comprising: an initial encoder and a generator in the trained generative adversarial network, the initial encoder is used to encode the foam image to be reconstructed to obtain a second target latent code, and the generator in the trained generative adversarial network is used to generate a reconstructed foam image based on the second target latent code, a constant input and random noise; Using the third data set to train the initial image reconstruction model, updating the weights of the initial encoder, and obtaining a trained image reconstruction model; Constructing a second foam image prediction model to be trained, the second foam image prediction model to be trained includes a mapping network, a discriminator, an editing module, a pre-trained encoder and a pre-trained generator, wherein the mapping network is used to process the second sample operation adjustment data to generate a sample image editing amplitude vector, the pre-trained encoder and the pre-trained generator are the encoder and the generator in the trained image reconstruction model, the pre-trained encoder is used to process the second sample pre-operation image to obtain a corresponding third target latent code, and the editing module is used to generate a sample edited latent code based on the sample image editing amplitude vector, a preset semantic direction set and the third target latent code; The second foam image prediction model to be trained is trained using the fourth data set, and the weights of the mapping network and the discriminator are updated to obtain a trained second foam image prediction model; the second model is obtained based on the mapping network, editing module, encoder and generator in the trained second foam image prediction model.

5. The method according to claim 4, characterized in that The preset semantic direction set is generated according to the following method: Extracting weight parameter data of each layer from the generator of the trained generative adversarial network to obtain a weight matrix; Constructing a semantic direction optimization model, and solving the semantic direction optimization model to obtain a candidate semantic direction set; For any target candidate semantic direction in the candidate semantic direction set, along the target candidate semantic direction, respectively edit the acquired target latent code to be edited using a preset first editing amplitude and a preset second editing amplitude to obtain a corresponding first edited latent code and a second edited latent code; Inputting the first edited latent code and the second edited latent code into the generator of the trained generative adversarial network respectively to obtain a first edited image and a second edited image; calculating similarity data between the first edited image and the second edited image; Based on each of the similarity degree data, elements in the candidate semantic direction set are eliminated to obtain the preset semantic direction set; Wherein, the semantic direction optimization model is: in, is the candidate semantic direction set, It is used to calculate the independent variable or independent variable set corresponding to the maximum value of the function. is a d×k real number matrix representing the set of semantic directions, represents the i-th semantic direction vector, for The transpose of is the number of semantic directions, d is the dimension of each semantic direction vector, and A is the weight matrix.

6. The method according to claim 4, characterized in that The second foam image prediction model to be trained is trained using the fourth data set according to any of the following methods, and the weights of the mapping network and the discriminator are updated to obtain a trained second foam image prediction model: Method 1: inputting the second sample pre-operation image and the second sample operation adjustment data of any third target sample in the fourth data set into the pre-trained encoder and the mapping network respectively to obtain the corresponding third target latent code and sample image editing amplitude vector; Along each semantic direction in the preset semantic direction set, the third target latent code is edited using the sample image editing amplitude vector corresponding to the third target sample to obtain an edited latent code of the first sample; The pre-trained generator is used to process the first sample edited latent code to obtain a first predicted operated image; according to a preset loss function, the first predicted operated image and the second sample operated foam image of the third target sample, the weights of the mapping network and the discriminator are updated to finally obtain the trained second foam image prediction model; Method 2: Inputting the second sample pre-operation image and the second sample operation adjustment data of any third target sample in the fourth data set into the pre-trained encoder and the mapping network respectively, and obtaining the corresponding third target latent code and sample image editing amplitude vector respectively; Selecting any unused semantic direction from the preset semantic direction set as the semantic direction for image editing; Along the semantic direction for image editing, the third target latent code is edited using the corresponding edit amplitude in the sample image edit amplitude vector of the third target sample to obtain a second sub-sample edited latent code; the second sub-sample edited latent code is processed using the pre-trained generator to obtain a second sub-prediction operation-posted image; According to the preset loss function, the second sub-prediction operation image and the second sample operation foam image of the third target sample, the weights of the mapping network and the discriminator are updated, and the step of returning to selecting any unadopted semantic direction from the preset semantic direction set as the semantic direction for image editing is performed until there is no unadopted semantic direction in the preset semantic direction set, thereby obtaining an intermediate second foam image prediction model; Along each semantic direction in the preset semantic direction set, the third target latent code is edited using the sample image editing amplitude vector corresponding to the third target sample to obtain an edited latent code of the second sample; The pre-trained generator is used to process the edited latent code of the second sample to obtain a second predicted image after operation; according to a preset loss function, the second predicted image after operation and the second sample foam image after operation of the third target sample, the weights of the mapping network and the discriminator in the intermediate second foam image prediction model are updated to finally obtain the trained second foam image prediction model.

7. The method according to claim 6, characterized in that If the trained second foam image prediction model is obtained by training according to the second method; the method further includes: Processing the image to be edited using the pre-trained encoder to obtain a latent code to be edited; Along any target semantic direction in the preset semantic direction set, using a preset third editing amplitude and a preset fourth editing amplitude, respectively edit the latent code to be edited to obtain a corresponding third edited latent code and a fourth edited latent code; Inputting the third edited latent code and the fourth edited latent code into the pre-trained generator respectively to obtain a third edited image and a fourth edited image; For any first target attribute in the set of attributes to be processed, determining the attribute change amount corresponding to each target semantic direction according to the attribute value of each of the third edited images and each of the fourth edited images under the first target attribute, wherein the first target attribute includes at least one of the following: foam size, foam color and foam shape; Determine the semantic direction corresponding to the maximum attribute change as the semantic direction to be processed; and determine the first target attribute as the attribute label of the semantic direction to be processed; Acquire a target operation adjustment data set and a target operation pre-image; For any target operation adjustment data in the target operation adjustment data set, the trained foam image prediction model is used to process the target operation adjustment data and the image before the target operation to obtain a target image editing amplitude vector and a target image after the target operation corresponding to the target operation adjustment data; Calculating a first attribute value of the image before the target operation under a second target attribute, and calculating a second attribute value set of each image after the target operation under the second target attribute; wherein the attribute value of the second target attribute is affected by the target operation adjustment data; According to each of the target operation adjustment data, the target image editing amplitude vector, the preset semantic direction set, the first attribute value, and the second attribute value set, a rule of how the operation adjustment data affects the foam state is generated.

8. A method for training an image prediction model, characterized in that: The method comprises: Acquire a first data set, wherein any first target sample in the first data set is composed of a first sample pre-operation image and first sample operation adjustment data, and a label of the first target sample is a corresponding first sample post-operation image; Constructing a first foam image prediction model to be trained; wherein the first foam image prediction model to be trained is the first model to be trained or the second model to be trained; the first model to be trained includes a first encoder, a first generator and a first discriminator, the first encoder is used to process the first sample pre-operation image and the first sample operation adjustment data to obtain a first sample latent code in a latent space for characterizing an image style, and the first generator is used to process the first sample latent code to obtain a predicted foam image after operation; the second model to be trained includes a second encoder, a second generator and a second discriminator, the second encoder is used to process the first sample pre-operation image to obtain a second sample latent code in a latent space for characterizing an image style, and the second generator is used to process the second sample latent code and the first sample operation adjustment data to obtain a predicted foam image after operation; The first foam image prediction model to be trained is trained using the first data set to obtain a trained first foam image prediction model; and a trained foam image prediction model is obtained according to an encoder and a generator in the trained first foam image prediction model.

9. The method according to claim 8, characterized in that When the second model further includes a mapping network and an editing module, the second model is generated according to the following method: Acquire a second data set, a third data set, and a fourth data set; wherein the second data set includes a plurality of real foam images, the third data set includes a plurality of foam images to be reconstructed, any second target sample in the fourth data set is composed of a second sample pre-operation image and second sample operation adjustment data, and the label of the second target sample is the corresponding second sample post-operation image; Constructing an initial generative adversarial network, and training the initial generative adversarial network using the second data set to obtain a trained generative adversarial network; wherein the initial generative adversarial network includes a conversion network, an initial generator, and an initial discriminator, the conversion network is used to process the random latent vector to obtain a first target latent code, and the initial generator is used to process the first target latent code, a constant input, and random noise to obtain a generated foam image; Constructing an initial image reconstruction model, the initial image reconstruction model comprising: an initial encoder and a generator in the trained generative adversarial network, the initial encoder is used to encode the foam image to be reconstructed to obtain a second target latent code, and the generator in the trained generative adversarial network is used to generate a reconstructed foam image based on the second target latent code, a constant input and random noise; Using the third data set to train the initial image reconstruction model, updating the weights of the initial encoder, and obtaining a trained image reconstruction model; Constructing a second foam image prediction model to be trained, the second foam image prediction model to be trained includes a mapping network, a discriminator, an editing module, a pre-trained encoder and a pre-trained generator, wherein the mapping network is used to process the second sample operation adjustment data to generate a sample image editing amplitude vector, the pre-trained encoder and the pre-trained generator are the encoder and the generator in the trained image reconstruction model, the pre-trained encoder is used to process the second sample pre-operation image to obtain a corresponding third target latent code, and the editing module is used to generate a sample edited latent code based on the sample image editing amplitude vector, a preset semantic direction set and the third target latent code; The second foam image prediction model to be trained is trained using the fourth data set, and the weights of the mapping network and the discriminator are updated to obtain a trained second foam image prediction model; the second model is obtained based on the mapping network, editing module, encoder and generator in the trained second foam image prediction model.

10. The method according to claim 9, characterized in that The second foam image prediction model to be trained is trained using the fourth data set according to any of the following methods, and the weights of the mapping network and the discriminator are updated to obtain a trained second foam image prediction model: Method 1: inputting the second sample pre-operation image and the second sample operation adjustment data of any third target sample in the fourth data set into the pre-trained encoder and the mapping network respectively to obtain the corresponding third target latent code and sample image editing amplitude vector; Along each semantic direction in the preset semantic direction set, the third target latent code is edited using the sample image editing amplitude vector corresponding to the third target sample to obtain an edited latent code of the first sample; The pre-trained generator is used to process the first sample edited latent code to obtain a first predicted operated image; according to a preset loss function, the first predicted operated image and the second sample operated foam image of the third target sample, the weights of the mapping network and the discriminator are updated to finally obtain the trained second foam image prediction model; Method 2: Inputting the second sample pre-operation image and the second sample operation adjustment data of any third target sample in the fourth data set into the pre-trained encoder and the mapping network respectively, and obtaining the corresponding third target latent code and sample image editing amplitude vector respectively; Selecting any unused semantic direction from the preset semantic direction set as the semantic direction for image editing; Along the semantic direction for image editing, the third target latent code is edited using the corresponding edit amplitude in the sample image edit amplitude vector of the third target sample to obtain a second sub-sample edited latent code; the second sub-sample edited latent code is processed using the pre-trained generator to obtain a second sub-prediction operation-posted image; According to the preset loss function, the second sub-prediction operation image and the second sample operation foam image of the third target sample, the weights of the mapping network and the discriminator are updated, and the step of returning to selecting any unadopted semantic direction from the preset semantic direction set as the semantic direction for image editing is performed until there is no unadopted semantic direction in the preset semantic direction set, thereby obtaining an intermediate second foam image prediction model; Along each semantic direction in the preset semantic direction set, the third target latent code is edited using the sample image editing amplitude vector corresponding to the third target sample to obtain an edited latent code of the second sample; The pre-trained generator is used to process the edited latent code of the second sample to obtain a second predicted image after operation; according to a preset loss function, the second predicted image after operation and the second sample foam image after operation of the third target sample, the weights of the mapping network and the discriminator in the intermediate second foam image prediction model are updated to finally obtain the trained second foam image prediction model.

11. The method according to claim 9, characterized in that The loss function used to train the initial generative adversarial network includes a generator loss function and a first discriminator loss function; the generator loss function is: The first discriminator loss function is: in, is the generator loss function, E represents the expectation, x' is the generated foam image, represents the i-th generated foam image, is the probability distribution of generating the foam image, is the probability that the discriminator considers the i-th generated foam image to be real data, is the first discriminator loss function, x represents the real foam image, represents the probability distribution of the true foam image, is the probability that the discriminator considers the i-th real foam image to be real data, r is the weight coefficient of the regularization loss term, Used to calculate the gradient of the discriminator with respect to its weight parameters.

12. The method according to claim 9, characterized in that The initial encoder adopts a feature pyramid network structure.

13. The method according to claim 9, characterized in that The reconstruction loss function used to train the initial image reconstruction model is: in, is the reconstruction loss function, is the pixel-level image similarity loss term, is the feature-level image similarity loss term, , is the weight coefficient corresponding to each loss item.

14. The method according to claim 9, characterized in that The loss function used to train the second foam image prediction model to be trained includes a mapping network loss function and a second discriminator loss function; the mapping network loss function is: in, represents the mapping network loss function, represents the pixel-level image similarity loss term, represents the feature-level image similarity loss term, Represents the loss term of the discriminator’s discrimination result on the generated graph, , , is the weight coefficient of each loss term; the second discriminator loss function is: in, is the second discriminator loss function, Express expectations, represents the image after the second sample operation, represents the probability distribution of the image after the second sample operation, represents the encoder, represents the discriminator, represents the post-operation foam image generated by the second foam image prediction model generator, represents the probability distribution of the post-operation foam image generated by the second foam image prediction model generator, represents the image after the i-th second sample operation, represents the i-th post-operation foam image generated by the second foam image prediction model generator, r is the weight coefficient of the regularization loss term, Used to calculate the gradient of the discriminator with respect to its weight parameters.

15. An image prediction device, characterized in that: The device comprises: A first acquisition module, used for acquiring in real time the foam image before operation and the operation adjustment data affecting the foam image before operation; A prediction module, used to process the pre-operation foam image and the operation adjustment data using a trained foam image prediction model, and predict a post-operation foam image; Among them, the trained foam image prediction model is the first model or the second model; the first model includes a first encoder and a first generator, the first encoder is used to process the foam image before operation and the operation adjustment data to obtain a first latent code in the latent space for representing the image style, and the first generator is used to process the first latent code to obtain the foam image after operation; the second model includes a second encoder and a second generator, the second encoder is used to process the foam image before operation to obtain a second latent code in the latent space for representing the image style, and the second generator is used to process the second latent code and the operation adjustment data to obtain the foam image after operation.

16. A training device for an image prediction model, characterized in that: The device comprises: A second acquisition module is used to acquire a first data set, wherein any first target sample in the first data set is composed of a first sample pre-operation image and first sample operation adjustment data, and a label of the first target sample is a corresponding first sample post-operation image; A construction module, used for constructing a first foam image prediction model to be trained; wherein the first foam image prediction model to be trained is the first model to be trained or the second model to be trained; the first model to be trained includes a first encoder, a first generator and a first discriminator, the first encoder is used for processing the first sample pre-operation image and the first sample operation adjustment data to obtain a first sample latent code in a latent space for characterizing an image style, and the first generator is used for processing the first sample latent code to obtain a predicted foam image after operation; the second model to be trained includes a second encoder, a second generator and a second discriminator, the second encoder is used for processing the first sample pre-operation image to obtain a second sample latent code in a latent space for characterizing an image style, and the second generator is used for processing the second sample latent code and the first sample operation adjustment data to obtain a predicted foam image after operation; A training module is used to train the first foam image prediction model to be trained using the first data set to obtain a trained first foam image prediction model; and to obtain a trained foam image prediction model based on the encoder and generator in the trained first foam image prediction model.

17. An electronic device comprising: processor; And a memory arranged to store computer executable instructions, characterized in that when the executable instructions are executed, the processor executes the steps of the method as claimed in any one of claims 1 to 7, or executes the steps of the method as claimed in any one of claims 8 to 14.

18. A computer-readable storage medium storing one or more programs, characterized in that: When the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device executes the steps of the method as claimed in any one of claims 1 to 7, or executes the steps of the method as claimed in any one of claims 8 to 14.

Images