Image processing method and device, electronic equipment, storage medium and product

Abstract

The application relates to an image processing method and device, electronic equipment, a storage medium and a product. The method comprises the following steps: acquiring a first frame image, the first frame image indicating a preset fluid system at a current time point, the first frame image representing a fluid distribution function corresponding to a fluid grid point in the preset fluid system by an image feature corresponding to a pixel point, the pixel point and the fluid grid point having a position correlation; acquiring a target guide image, the target guide image being used for representing the distribution of a target obstacle object in the preset fluid system; taking the first frame image and the target guide image as inputs, and outputting a second frame image by using a target prediction model, the second frame image indicating the preset fluid system at a next time point. The application provides an image processing scheme considering both efficiency and accuracy for fluid simulation.

Description

Technical Field

[0001] This application relates to the field of Internet communication technology, and in particular to an image processing method, apparatus, electronic device, storage medium and product. Background Technology

[0002] With the development of internet communication technology, various internet products have emerged. Some of these products can use fluid simulation to provide users with related functionalities, such as simulating weather in weather information services or simulating smoke in gaming services. In these technologies, a current frame image is often used to represent the current state of the simulated fluid, and then the image is processed by a fluid simulation algorithm to obtain the next frame image corresponding to the next state. However, this image processing scheme becomes computationally inefficient when the amount of data in the current frame image is large, thus affecting the user's functional service experience. Therefore, a more efficient image processing scheme is needed. Summary of the Invention

[0003] To address at least one of the aforementioned technical problems, this application provides an image processing method, apparatus, electronic device, storage medium, and product:

[0004] According to a first aspect of this application, an image processing method is provided, the method comprising:

[0005] Acquire a first frame image, which indicates a preset fluid system at the current time point. The first frame image uses image features corresponding to pixels to characterize the fluid distribution function corresponding to the fluid grid points in the preset fluid system. The pixels and the fluid grid points have a positional relationship.

[0006] Acquire a target guidance image, which is used to characterize the distribution of target obstacle objects in the preset fluid system;

[0007] Using the first frame image and the target guidance image as input, a second frame image is output using a target prediction model. The second frame image indicates the preset fluid system at the next time point. The target prediction model is obtained by training a preset model based on multiple sample image groups. The sample image group consists of current frame image samples and guidance image samples, and carries the next frame image sample. The preset model includes a first network, a second network, and a third network. The first network is used to encode and decode the current frame image samples sequentially to obtain a base image sample. The second network is used to predict the change information of key regions in the current frame image samples based on the lattice Boltzmann algorithm. The key regions are determined under the guidance of the guidance image samples. The third network is used to predict the next frame image based on the base image samples and the change information. During training, the preset model is adjusted based on the difference between the predicted next frame image and the next frame image samples.

[0008] According to a second aspect of this application, an image processing apparatus is provided, the apparatus comprising:

[0009] Frame image acquisition module: used to acquire a first frame image, the first frame image indicating a preset fluid system at the current time point, the first frame image characterizing the fluid distribution function corresponding to the fluid grid point in the preset fluid system with the image features corresponding to the pixel point, the pixel point and the fluid grid point have a positional relationship;

[0010] Guidance image acquisition module: used to acquire target guidance images, which are used to characterize the distribution of target obstacle objects in the preset fluid system;

[0011] The prediction module is used to take the first frame image and the target guidance image as input, and output a second frame image using a target prediction model. The second frame image indicates the preset fluid system at the next time point. The target prediction model is obtained by training a preset model based on multiple sample image groups. The sample image group consists of current frame image samples and guidance image samples, and the sample image group carries the next frame image sample. The preset model includes a first network, a second network, and a third network. The first network is used to encode and decode the current frame image samples sequentially to obtain a base image sample. The second network is used to predict the change information of key regions in the current frame image samples based on the lattice Boltzmann algorithm. The key regions are determined under the guidance of the guidance image samples. The third network is used to predict the next frame image based on the base image samples and the change information. During training, the preset model is adjusted based on the difference between the predicted next frame image and the next frame image samples.

[0012] According to a third aspect of this application, an electronic device is provided, the electronic device including at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the at least one processor to implement the image processing method as described in the first aspect.

[0013] According to a fourth aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or at least one program being loaded and executed by a processor to implement the image processing method as described in the first aspect.

[0014] According to a fifth aspect of this application, a computer program product is provided, the computer program product comprising at least one instruction or at least one program segment, the at least one instruction or at least one program segment being loaded and executed by a processor to implement the image processing method as described in the first aspect.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application.

[0016] Implementing this application will have the following beneficial effects:

[0017] This application provides an image processing scheme that balances efficiency and accuracy for fluid simulation. Under the premise of representing the state of each fluid grid in a fluid system at a specific time point using frame images, this application takes a first frame image (indicating the current time point) and a target guidance image as input, and uses a target prediction model to output a second frame image (indicating the next time point). Using the lattice Boltzmann algorithm as the preset fluid simulation algorithm, when representing the state of the frame image, based on the characteristics of the lattice Boltzmann algorithm, a relationship is established between the image features corresponding to pixels and the fluid distribution function corresponding to the fluid grid points based on positional association, providing support for the accuracy and effectiveness of fluid simulation. Predicting the second frame image using the target prediction model helps improve the efficiency and accuracy of obtaining the second frame image. Simultaneously, during the training of the target prediction model, the first network processes the global dimensional information of the current frame image sample through encoding and decoding, the second network processes the local dimensional information of the current frame image sample through the lattice Boltzmann algorithm, and the third network integrates the processed global and local dimensional information to predict the next frame image. The model parameters are then adjusted based on the difference between the predicted next frame image and the next frame image sample. The global and local dimensional information of the current frame image sample are processed using different calculation methods. This approach avoids information omissions that could affect the prediction of the next frame. Furthermore, compared to the original data volume of the current frame image sample, the smaller amount of local dimensional information improves the computational efficiency of the lattice Boltzmann algorithm. The local dimensional information also focuses on key regions in the current frame image sample, which helps improve the accuracy of the prediction for the next frame. Therefore, using the trained target prediction model for the next frame image prediction improves the adaptability and reliability of the prediction, and effectively handles situations where the first frame image has a large data volume.

[0018] Other features and aspects of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This diagram illustrates an application environment according to an embodiment of the present application.

[0021] Figure 2 A schematic flowchart of an image processing method according to an embodiment of this application is shown;

[0022] Figure 3 A schematic diagram illustrating the process of training to obtain a target prediction model according to an embodiment of this application is shown;

[0023] Figure 4 A schematic diagram illustrating the process of obtaining a substrate image sample according to an embodiment of this application is shown;

[0024] Figure 5 This diagram illustrates a process for obtaining change information according to an embodiment of this application.

[0025] Figure 6 This diagram illustrates a process for obtaining candidate frame image samples and a fourth obstacle image according to an embodiment of this application.

[0026] Figure 7 A schematic diagram showing a first frame image and a target guidance image according to an embodiment of this application;

[0027] Figure 8 This diagram illustrates the application of a preset model according to an embodiment of this application.

[0028] Figure 9 This diagram illustrates the region segmentation of a current frame image sample according to an embodiment of this application.

[0029] Figure 10 A schematic diagram of the architecture of a first network according to an embodiment of this application is shown;

[0030] Figure 11 This diagram illustrates a device block diagram according to an embodiment of the present application;

[0031] Figure 12 A schematic diagram of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0034] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0035] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0036] The term “exemplary” as used herein means “serving as an example or embodiment.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0037] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0038] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed description. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0039] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0040] LBM (Lattice Boltzmann Method) ： Lattice Boltzmann algorithm.

[0041] Please see Figure 1 , Figure 1 The diagram illustrates an application environment according to an embodiment of this application. The application environment may include a terminal 10 and a server 20. The terminal 10 and server 20 can be directly or indirectly connected via wired or wireless communication. The target object sends an image processing request to the server 20 through the terminal 10. Based on the received image processing request, the server 20 acquires a first frame image and a target guidance image. The first frame image indicates the preset fluid system at the current time point. The first frame image uses image features corresponding to pixels to characterize the fluid distribution function corresponding to the fluid grid points in the preset fluid system. Pixels and fluid grid points have a positional correlation. The target guidance image is used to characterize the distribution of target obstacle objects in the preset fluid system. Then, using the first frame image and the target guidance image as input, a second frame image is output using a target prediction model. The second frame image indicates the preset fluid system at the next time point. The target prediction model used was trained through the following steps: First, using the current frame image sample as input, a first network outputs a base image sample; then, using the current frame image sample and the guide image sample as input, a second network outputs change information; next, using the base image sample and the change information as input, a third network predicts the next frame image; finally, based on the difference between the predicted next frame image and the next frame image sample, the parameters of the preset model are adjusted to obtain the target prediction model. It should be noted that... Figure 1 This is just one example.

[0042] Terminal 10 can be a physical device such as a smartphone, computer (e.g., desktop computer, tablet computer, laptop computer), augmented reality (AR) / virtual reality (VR) device, digital assistant, smart voice interaction device (e.g., smart speaker), smart wearable device, smart home appliance, in-vehicle terminal, etc. The operating system of Terminal 10 can be Android, iOS (a mobile operating system developed by Apple), Linux, Microsoft Windows, etc. Applications can be installed on Terminal 10, such as browser applications, news feed applications, video applications, game applications, navigation applications, etc.

[0043] The server-side component 20 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server may include network communication units, processors, and memory, etc.

[0044] In practical applications, 1) the server that trains the target prediction model through the training steps provided in this application embodiment can be server A, and the server that processes the first frame image and the target guidance image through the image processing method provided in this application embodiment can be server B. Server A and server B can refer to the same server or two different servers. Alternatively, the terminal can also process the first frame image and the target guidance image using the locally stored target prediction model.

[0045] 2) The image processing method provided in this application embodiment can be applied to the physical simulation or fluid motion rendering of game engines. For independent fluid systems, such as weather systems, the image processing method provided in this application embodiment can be used to simulate weather systems related to the game screen. For interactive particle systems, the image processing method provided in this application embodiment can be used to simulate the movement trajectory of a person stepping into a puddle or the flow direction of smoke in a wind system environment.

[0046] Figure 2 This diagram illustrates a flowchart of an image processing method according to an embodiment of this application, as shown below. Figure 2 As shown, this image processing method can be executed by an electronic device, which can be a terminal or a server. The method includes:

[0047] S201: Acquire a first frame image, the first frame image indicating a preset fluid system at the current time point, the first frame image characterizing the fluid distribution function corresponding to the fluid grid point in the preset fluid system with the image features corresponding to the pixel point, the pixel point and the fluid grid point having a positional relationship;

[0048] In this embodiment, the electronic device acquires a first frame image. The first frame image indicates a preset fluid system at the current time point. The preset fluid system can be an independent fluid system or an interactive particle system. When the first frame image characterizes the state of each fluid grid in the preset fluid system at the current time point, it establishes a relationship between the image features corresponding to the pixels and the fluid distribution functions corresponding to the fluid grid points based on the characteristics of the lattice Boltzmann algorithm and position association. It can be understood that the preset fluid system includes multiple fluid grid points, the multiple fluid grid points are of the same size, and the multiple fluids are uniformly distributed. The fluid distribution function corresponding to the fluid grid point is often used to describe the distribution of the fluid grid point group and its relationship with spatial position and time. Taking a three-dimensional fluid grid point as an example, the fluid distribution function corresponding to the fluid grid point is composed of sub-fluid distribution functions in different velocity directions.

[0049] In practical applications, the fluid grid can be viewed as a cube. The fluid grid involves nine velocity directions: one pointing towards the center of the cube, four pointing towards the top, bottom, left, and right of the cube, and four pointing towards the four spatial diagonals. These four spatial diagonals correspond to four spatial diagonals, each connecting a pair of opposite vertices. Accordingly, the fluid distribution function corresponding to the fluid grid is composed of sub-fluid distribution functions in the nine velocity directions, and the image features corresponding to each pixel are composed of sub-image features in nine channel dimensions. This can be represented as... Let f represent the sub-image feature of the i-th channel dimension, and it characterizes the fluid distribution function f corresponding to the fluid grid point. i (x, t). Where i ranges from 0 to 8, and t represents the current time point.

[0050] Furthermore, the lattice Boltzmann algorithm also involves the following evolution equations:

[0051] f i (x+c i Δt, t+Δt)=f i (x, t) + Ω i (x, t)

[0052]

[0053] Among them, c i Ω represents velocity, and Δt represents the time step. iThis represents the collision function. τ represents the relaxation time, which is related to the viscosity of the fluid motion. The equilibrium distribution function can be expressed as follows:

[0054]

[0055] Where ρ represents macroscopic density, w i c represents the velocity weight. s represents the speed of sound, and u represents macroscopic velocity.

[0056] In the D2Q9 architecture (where 2 represents two dimensions and 9 represents nine velocity directions), w i The possible values ​​are as follows:

[0057] w0 = 4 / 9, corresponding to one velocity direction pointing towards the center of the cube;

[0058] w1 = w2 = w3 = w4 = 1 / 9, corresponding to the four velocity directions pointing up, down, left, and right of the cube;

[0059] w5 = w6 = w7 = w8 = 1 / 36, corresponding to the four velocity directions pointing to the four opposite corners of space.

[0060] Using the above evolutionary equations, pseudocode for the lattice Boltzmann algorithm for fluid simulation can be obtained. Specifically, in the initial case (t=0), let ρ and u also be 0. Then, perform iterative solutions at each time step. The iterative solution at one time step includes: 1) updating the fluid distribution function f based on the above evolutionary equations. i (x, t); 2) Based on the fluid distribution function f i (x, t) Calculate ρ and u; 3) Make adaptive corrections to u based on boundary conditions.

[0061] It should be noted that, in addition to the D2Q9 architecture, D1Q3, D2Q9, D3Q15, D3Q19 and D3Q27 can also be used, that is, the dimension is not limited to 2, and the speed direction is not limited to 9.

[0062] S202: Obtain a target guidance image, which is used to characterize the distribution of target obstacle objects in the preset fluid system;

[0063] In this embodiment, the electronic device acquires a target guidance image. The target guidance image is used to characterize the distribution of target obstacle objects in a preset fluid system. For example... Figure 7As shown, the left image, serving as the first frame, focuses on displaying the fluid grid cluster, while the right image, serving as the target guidance image, focuses on displaying the target obstacle object. Generally, the target obstacle object can significantly affect the flow characteristics of the fluid grid cluster. Accordingly, there is a correlation between the first frame image and the target guidance image. For the target obstacle object in the preset fluid system, there can be only one target obstacle object, or it can consist of at least two sub-obstacle objects, as can be seen in the reference. Figure 7 The right image in the text.

[0064] Since the target obstacle may change due to external forces, the target guidance image can also constrain the target obstacle based on the current time point. In practical applications, based on business requirements, the time interval between the first and second frames is relatively short, and it can be assumed that the target obstacle has not changed from the current time point to the next time point. Correspondingly, in some cases, the guidance image corresponding to a previous time point (such as the previous time point) can also be used.

[0065] In practical applications, in conjunction with the aforementioned step S201, This represents the setting of the sub-image features in the i-th channel dimension. The first frame image can be... The target guidance image can be represented as... (i=9) indicates that for a pixel in the target guidance image, a value of 1 can be used if it indicates a target obstacle object, and a value of 0 can be used if it indicates a fluid grid group.

[0066] S203: Using the first frame image and the target guidance image as input, output a second frame image using the target prediction model. The second frame image indicates the preset fluid system at the next time point.

[0067] In this embodiment, using a first frame image and a target guidance image as input, the electronic device outputs a second frame image using a target prediction model. The second frame image, predicted by the target prediction model, reflects the state of each fluid cell in the preset fluid system at the next time point. This leverages the adaptability and reliability of the target prediction model to improve the efficiency and accuracy of obtaining the second frame image. Subsequent page updates can then be based on the second frame image. For example, if the preset fluid system corresponds to a virtual smoke system in a game scene, the first frame image is taken from the current frame of the game screen, and the second image will participate in the rendering update of the next frame. The image processing method provided in this embodiment can improve the smoothness of smoke flow simulation in game applications, thereby enhancing the user's gaming experience. Of course, besides simulating smoke flow, it can also simulate water ripples, etc.

[0068] The target prediction model will be described in detail below. The target prediction model is obtained by training a preset model based on multiple sample image groups. Each sample image group consists of a current frame image sample and a guide image sample, and carries a next frame image sample. The preset model includes a first network, a second network, and a third network. The first network is used to sequentially encode and decode the current frame image sample to obtain a base image sample. The second network is used to predict the change information of key regions in the current frame image sample based on the lattice Boltzmann algorithm. The key regions are determined under the guidance of the guide image sample. The third network is used to predict the next frame image based on the base image sample and the change information. During training, the preset model is adjusted based on the difference between the predicted next frame image and the current frame image sample.

[0069] like Figure 3 , 8 As shown, the target prediction model is obtained through the following steps:

[0070] S301: Using the current frame image sample as input, output the base image sample using the first network;

[0071] S302: Using the current frame image sample and the guidance image sample as input, the change information is output using the second network;

[0072] S303: Using the base image samples and the change information as input, the third network is used to predict the next frame image;

[0073] S304: Based on the difference between the predicted next frame image and the next frame image sample, adjust the parameters of the preset model to obtain the target prediction model.

[0074] The first network is responsible for processing the global dimensionality information of the current frame image sample, which can be achieved through encoding and decoding. Before processing, the global dimensionality information of the current frame image sample is the current frame image sample itself. After processing, the global dimensionality information of the current frame image sample is the basis image sample. The second network is responsible for processing the local dimensionality information of the current frame image sample, which can be achieved through referencing the guide image sample and applying the lattice Boltzmann algorithm. The guide image sample is used to characterize the distribution of obstacle objects in the sample fluid system. Before processing, the local dimensionality information of the current frame image sample is the key region in the current frame image sample. After processing, the local dimensionality information of the current frame image sample is the change information of the predicted key region. The key region is determined under the guidance of the guide image sample and can be a region in the current frame image sample related to the boundary of the obstacle object. The third network is responsible for integrating the processed global and local dimensionality information to predict the next frame image.

[0075] When adjusting the parameters of a preset model using the difference between the predicted next frame image and the next frame image samples, a loss function can be constructed based on the difference, and then the parameters of the preset model can be adjusted using the loss function. For multiple sample image groups, at least one model parameter adjustment can be involved in the training process to obtain the target prediction model. Each model parameter adjustment is based on the difference between the predicted value (the predicted next frame image) and the ideal value (the next frame image sample) corresponding to each sample image group in a batch sample set. A batch sample set can be all the multiple sample image groups, or it can be a part of multiple sample image groups. Model training is continuous, and the target prediction model can be the result of the previous training or the basis for the next training. The target prediction model is a model with high generalization ability obtained through training. Using the target prediction model to predict the next frame image can improve the adaptability and reliability of the next frame image prediction. In fluid simulation, the target prediction model also uses an architecture in which part of the network processes local dimensional information based on the lattice Boltzmann algorithm, and another part of the network processes global dimensional information through encoding and decoding, to achieve fitting of the frame image prediction using the lattice Boltzmann algorithm. This effectively reduces computational complexity, increases computational speed, and saves computational resources while still meeting the required prediction accuracy.

[0076] In practical applications, 1) the target prediction model can be trained for a specific fluid system. Taking a water molecule system as an example, the trained target prediction model can be used to simulate the movement of water molecules in game scenes, such as rain in large-scale scenes and puddles in small-scale scenes. Relevant obstacle objects can indicate wind, human-computer interaction commands, etc. 2) The parameter adjustment of the preset model can be the parameter adjustment of the first network. The first network can be UNet, or it can be CNN, Transformer, etc.

[0077] It should be noted that the current frame image sample can refer to the description of the first frame image in step S201 above, and the guide image sample can refer to the description of the target guide image in step S202 above, which will not be repeated here.

[0078] As one possible implementation, such as Figure 4 As shown, the first network includes an encoding subnetwork and a decoding subnetwork. The encoding subnetwork includes multiple feature extraction units arranged sequentially. Each feature extraction unit consists of a first convolutional layer and a pooling layer for compressing the image scale. The decoding subnetwork includes multiple feature restoration units arranged sequentially. Each feature restoration unit consists of a second convolutional layer and a deconvolutional layer for enlarging the image scale. The step of taking the current frame image sample as input and outputting the base image sample using the first network includes:

[0079] S401: The fused image samples are processed using the coding sub-network to obtain the corresponding coding result, wherein the fused image samples indicate the fusion result of the current frame image samples and the guide image samples;

[0080] S402: The encoding result is processed using the decoding sub-network to obtain the base image sample.

[0081] This section provides the settings and applications of the encoding and decoding structures in the first network. For the encoding sub-network, the arrangement of the feature extraction units' dimensions and their internal structure are defined. For the decoding sub-network, the arrangement of the feature reconstruction units' dimensions and their internal structure are defined. This provides support for the encoding and decoding processing methods used in the first network. It constrains the data flow in the encoding stage, allowing for multiple processes of "increasing channel dimensions via the first convolutional layer and compressing the scale via the pooling layer"; it also constrains the data flow in the decoding stage, allowing for multiple processes of "reducing channel dimensions via the second convolutional layer and amplifying the scale via the deconvolutional layer." Image scale refers to the width and height of the image. Furthermore, the global dimensional information before processing is defined; it is the fused image sample, i.e., the fusion result of the current frame image sample and the guide image sample. Compared to the current frame image sample, the fused image sample carries richer information, and the guide image sample also has a correlation with the current frame image sample, further supporting the use of the fused image sample as global dimensional information before processing. The relationship between the guide image sample and the current frame image sample can be referred to the description of the relationship between the first frame image and the target guide image in step S202 above, and will not be repeated here.

[0082] The architecture of the first network described above enables encoding and decoding to fully capture and utilize feature information at different levels, thereby improving the accuracy of the obtained base image samples. Simultaneously, the fused image samples, carrying richer information, also support the retention of more detailed information in the base image samples. The accuracy and reliability of the base image samples are beneficial for improving the accuracy of subsequent image predictions in the next frame.

[0083] In practical applications, the first network can be adopted as follows: Figure 10The architecture of UNet is shown. UNet borrows the encoding and decoding ideas of FCN networks, extracting deep semantic information from the image by downsampling the input image several times, and then achieving pixel-level classification by upsampling to restore the image size. Accordingly, in the first network of the UNet architecture, a feature extraction unit includes two operation groups: a first operation group containing a first convolutional layer (Conv), a normalization layer (BatchNorm), and an activation layer (LeakyReLU), and a second operation group containing a pooling layer (MaxPool), a normalization layer (BatchNorm), and an activation layer (LeakyReLU); a feature restoration unit includes two operation groups: a third operation group containing a second convolutional layer (Conv), a normalization layer (BatchNorm), and an activation layer (LeakyReLU), and a fourth operation group containing a deconvolutional layer (ConvTranspose), a normalization layer (BatchNorm), and an activation layer (LeakyReLU). The pooling layer can perform 2:1 scale compression to extract a wider range of feature information. Deconvolutional layers can perform a 1:2 scaling. In the data flow direction, the first convolutional layer can extract low-frequency feature information; the second convolutional layer, with the assistance of residual connections, can extract high-frequency feature information.

[0084] The number of feature extraction units and feature reduction units is 5 each. The image scale input to the first feature extraction unit is the same as the image scale output by the last feature reduction unit. Figure 10 The dashed lines in the diagram represent residual connections, which pass the output of the feature extraction unit to the corresponding feature restoration unit for dimensional concatenation with the upsampling results of the preceding deconvolution layer. Simultaneously, three fully connected layers are positioned between multiple feature extraction units and multiple feature restoration units. Each fully connected layer can reach a channel dimension of 2048, equivalent to having an additional 2048x2048 neuron computing nodes. This enhances the focus on more feature information, thereby adapting to complex computations involving large scales and large data volumes.

[0085] Furthermore, regarding the aforementioned step S401, before processing the fused image samples using the coding sub-network to obtain the corresponding coding result, the method may further include the following steps: First, when the scales of the current frame image sample and the guide image sample are both a first scale and the first scale is greater than a second scale, the current frame image sample and the guide image sample are fused to obtain a candidate fused image, wherein the candidate fused image adopts the first scale; then, the candidate fused image is downsampled to obtain the fused image sample, wherein the fused image sample adopts the second scale;

[0086] Correspondingly, in step S402 above, the step of processing the encoding result using the decoding sub-network to obtain the base image sample may include the following steps: inputting the encoding result into the decoding sub-network to obtain a candidate base image, wherein the candidate base image adopts the second scale; performing upsampling processing on the candidate base image to obtain the base image sample, wherein the base image sample adopts the first scale.

[0087] For example, the first scale is 1024*1024, the second scale is 128*128, the current frame image sample (1024*1024*9) has a scale of 1024*1024 and a channel dimension of 9, and the guide image sample (1024*1024*1) has a scale of 1024*1024 and a channel dimension of 1. The current frame image sample (1024*1024*9) and the guide image sample (1024*1024*1) are fused to obtain a candidate fused image (1024*1024*10). The candidate fused image (1024*1024*10) is downsampled to obtain a fused image sample (128*128*10). Downsampling can be achieved using mean pooling. After obtaining the fused image sample (128*128*10), the encoding sub-network processes the fused image sample (128*128*10) to obtain the corresponding encoding result. The encoding result is then input into the decoding sub-network to obtain the candidate base image (128*128*9). Next, the candidate base image (128*128*9) is upsampled to obtain the base image sample (1024*1024*9). Upsampling can be achieved through deconvolution. Regarding the reduction in channel dimension from the fused image sample (128*128*10) to the candidate base image (128*128*9), it should be understood that the next frame image, as the prediction target, should match the current frame image not only in scale but also in channel dimension. Therefore, the channel dimension of the candidate base image is 9. The guiding role of the guiding image sample is also reflected in providing reference information, rather than changing the channel dimension of the candidate base image.

[0088] This approach takes into account the frequent need to process large-scale (e.g., 1024*1024) current frame and guide images in fluid simulation scenarios. The processing steps for larger-scale images are divided into two stages: a scale compression stage before encoding and a scale restoration stage after decoding. This also better suits the original first network architecture. Furthermore, using the scale-compressed image as the encoding / decoding object and the scale-restored decoded result as the base image sample balances the efficiency and accuracy of obtaining the base image sample, thus better meeting the needs of fluid simulation scenarios.

[0089] Of course, the specific values ​​of the first and second scales can be flexibly adjusted according to business needs.

[0090] As one possible implementation, such as Figure 5 , 9 As shown, the step of taking the current frame image sample and the guide image sample as input, and using the second network to output the change information, includes:

[0091] S501: The second network is used to segment the current frame image sample into regions to obtain multiple first local images, the scale of the first local images being a preset scale;

[0092] S502: Using the second network, based on the distribution of the sample obstacle objects represented by the guide image samples and the correlation between the guide image samples and the current frame image samples, determine a plurality of second local images containing the representation boundaries of the sample obstacle objects from the plurality of first local images;

[0093] S503: The second network is used to process the multiple second local images based on the lattice Boltzmann algorithm to obtain the corresponding multiple third local images, and the multiple third local images are determined to be the change information.

[0094] For obtaining multiple first local images, if the preset scale is 64*64, the scale of the current frame image sample (1024*1024*9) is 1024*1024 and the channel dimension is 9. The current frame image sample is scaled according to the preset scale to obtain 256 first local images (64*64*9).

[0095] Because the current frame image sample and the guide image sample are related, the former focuses on displaying the fluid grid cluster in the sample fluid system, while the latter focuses on displaying the sample obstacle object. This provides support for determining multiple second local images containing the boundary of the sample obstacle object from multiple first local images. It should be understood that regardless of whether there is only one sample obstacle object or it consists of at least two sub-obstacle objects, the boundary of the sample obstacle corresponds to at least one curve. The curve can be a closed curve or an open curve. Based on the relationship between the current frame image sample and the guide image sample, the boundary of the sample obstacle can be projected onto the current frame image sample to obtain the representation boundary of the sample obstacle. Correspondingly, the representation boundary of the sample obstacle corresponds to at least one representation curve. The representation curve can be a closed curve or an open curve. All or part of the representation curve is projected onto a second local image to achieve a non-realistic representation of all or part of the representation curve in the second local image. While displaying all or part of the representation curve, the second local image also displays the fluid grid points in the fluid grid cluster of the sample fluid system that are close to the sample obstacle object, which is key information that can affect the fluid simulation. Multiple second local images represent key regions within the current frame image sample. Predicting changes in these key regions involves using the lattice Boltzmann algorithm to predict the corresponding third local image for each second local image. One third local image reflects changes in one second local image, and multiple third local images reflect changes in the key regions.

[0096] This section provides methods for defining key regions in the current frame image sample and obtaining information on changes in these key regions. Using the boundary of the sample obstacle object as an anchor point and a preset scale as a constraint, the quality of the multiple local images representing the key regions is ensured. The boundary of the sample obstacle object influences the display items in the second local image, making it crucial information that affects fluid simulation. The preset scale influences the scale of the second local image, ensuring it remains within an appropriate range. Excessive scale introduces information that negatively impacts computational efficiency, while insufficient scale introduces information related to "fluid grid points near the sample obstacle object within the fluid grid group of the sample fluid system," thus improving the accuracy of predictions for the next frame image.

[0097] It should be noted that the current frame image sample focuses on displaying the fluid grid clusters in the sample fluid system, and this is not a limitation on its ability to exclude sample obstacle objects. The current frame image sample may or may not display sample obstacle objects. Because the current frame image sample focuses on displaying the fluid grid clusters in the sample fluid system, even if the current frame image sample displays sample obstacle objects, the display of sample obstacle objects may be incomplete in some cases, such as when the fluid grid clusters occlude the display of sample obstacle objects. Correspondingly, although the aforementioned second local image provides a non-realistic representation of all or part of the representation curve, it can still realistically display the boundaries of sample obstacle objects.

[0098] Furthermore, for the aforementioned step S303, the step of using the base image sample and the change information as input and using the output of the third network to predict the next frame image may include the following steps: for each third local image, using the third network to determine the current local image at the corresponding position in the base image sample according to the position of the second local image corresponding to the third local image in the current frame image, and replacing or covering the current local image with the third local image to achieve the prediction of the next frame image.

[0099] For example, the current frame image sample (1024*1024*9) has a scale of 1024*1024 and a channel dimension of 9. The base image sample (1024*1024*9) has a scale of 1024*1024 and a channel dimension of 9. The second local image (64*64*9) has a scale of 64*64 and a channel dimension of 9. The third local image (64*64*9) has a scale of 64*64 and a channel dimension of 9. If there are multiple third local images, there are M third local images, and the third local image m is the m-th of the M third local images. The third local image m is the prediction result of the second local image m. When the second local image m is located in region i of the current frame image sample, the current local image j is determined in region j of the base image sample, and the current local image j is replaced or covered by the third local image m. The position of region j of the base image sample matches the position of region i of the current frame image sample, such as both indicating the upper left corner region.

[0100] The prediction results of the second local image are used to replace or cover the current local image of the corresponding region in the base image sample, thereby realizing the prediction of the next frame image. The region segmentation of the image and the setting of multiple second local images, i.e., key regions in the current frame image sample, provide support for the replacement or coverage between local images, which helps to improve the efficiency of the next frame image prediction.

[0101] As a possible implementation, for the aforementioned step S303, after predicting the next frame image using the third network output with the base image samples and the change information as input, the method may further include the following steps: First, after completing the prediction of the next frame image for the current batch of sample sets, for each sample image group in the current batch of sample sets, determine the difference between the predicted next frame image and the next frame image sample to obtain a difference set, wherein the current batch of sample sets includes a preset number of sample image groups; then, based on the difference set, adjust the parameters of the preset model to obtain the target prediction model.

[0102] Considering that tuning parameters based on the difference between the predicted value (the predicted next frame image) and the ideal value (the next frame image sample) of a sample image group would result in a long model iteration time, this paper proposes tuning parameters based on the difference set corresponding to the current batch of sample sets. The loss function reflects the accumulation of differences, which can improve the model iteration speed and shorten the time to obtain the target prediction model, allowing the target prediction model to participate in the processing of the first frame image and the target guidance image more promptly. For example, when tuning parameters based on the difference set corresponding to the current batch of sample sets, the following loss function can be constructed based on the difference set, and then the parameters of the preset model can be adjusted using the loss function:

[0103]

[0104] Where T represents the preset number, X t+k G(X) represents the k-th sample image group in a set of preset sample image groups. t+k ) represents the predicted value corresponding to the k-th sample image group. This represents the ideal value corresponding to the k-th sample image group. This represents the absolute error or mean square error between the predicted value and the ideal value corresponding to the k-th sample image group. In practical applications, T can be 10.

[0105] As one possible implementation, such as Figure 6 As shown, the method further includes:

[0106] S601: Randomly generate a candidate image set using a directed distance field algorithm, the candidate image set including multiple first obstacle images;

[0107] S602: For each of the first obstacle images, with the target number as the time step, the first obstacle image is processed by a direct calculation method to obtain a corresponding candidate image sequence. The target number of second obstacle images in the candidate image sequence are arranged in the order of image generation time.

[0108] S603: Extract sequence segments from the candidate image sequence, the sequence segments including multiple third obstacle images;

[0109] S604: Determine multiple fourth obstacle images from the sequence segments according to a preset interval;

[0110] S605: Using the multiple fourth obstacle images as a reference to guide the image sequence, the lattice Boltzmann algorithm is used to simulate the preset fluid object to obtain multiple candidate frame image samples. The multiple candidate frame image samples and the multiple fourth obstacle images are the data sources for constructing the sample image group.

[0111] The Signed Distance Field (SDF) algorithm is a function that defines the shortest distance from any point in space to the nearest surface. For a point p and a surface s, the SDF value d can be represented as the distance from point p to the nearest point on surface s. If point p is inside surface s, d is negative; if p is outside surface s, d is positive. Specifically, it can be expressed as follows:

[0112] d(p)=min||pq i ||

[0113] Among them, ||pq i || represents the relationship between point p and point q on surface s. i The Euclidean distance between them.

[0114] In image generation, the SDF algorithm can be used to implicitly represent 3D shapes and render 2D images from arbitrary angles. When randomly generating the first obstacle image using the SDF algorithm, a random vector can be used as input. For example, 500 first obstacle images (1024*1024) can be generated. Exemplarily, the random vector can constrain the 3D pose of the potential obstacle object. The resulting 3D potential obstacle object can then have its directed distance calculated by limiting sampling points, and multiple sampling points can be projected onto a preset 2D plane. On the preset 2D plane, the 2D region of the obstacle object contributed by the sampling points is determined with the projection point corresponding to each sampling point as the center and the directed distance corresponding to each sampling point as the radius. The sampling points can be randomly determined on the potential obstacle object.

[0115] For each first obstacle image, using the target number as the time step, a direct calculation method is used to evolve the first obstacle image, resulting in a corresponding candidate image sequence. The application of this direct calculation method focuses on simulating the changes of the first obstacle image over time. For example, at a time step of 16000, the candidate image sequence corresponding to the first obstacle image includes 16000 second obstacle images, where second obstacle images generated earlier have higher rankings in the candidate image sequence. For the candidate image sequence corresponding to the first obstacle image, a sequence segment containing multiple third obstacle images is extracted. For example, the extracted sequence segment represents 160 consecutive second obstacle images in the candidate image sequence. Under a preset interval constraint, multiple fourth obstacle images are determined from the sequence segment. For example, for every 160 consecutive third obstacle images in the sequence segment, the last image in the sequence is retained out of every 8 images, resulting in 20 fourth obstacle images. These multiple fourth obstacle images can reflect changes in the preset obstacle object and can constitute a baseline guidance image sequence. Based on this, the lattice Boltzmann algorithm is used to simulate a predefined fluid object to obtain multiple candidate frame image samples. The XLB fluid simulation framework can be used when simulating the predefined fluid object using the lattice Boltzmann algorithm. Multiple candidate frame image samples and multiple fourth obstacle images serve as the data source for constructing the sample image set.

[0116] The above content serves the preparation stage of the sample image set, ensuring that a higher quality sample image set can participate in model training. The randomness of the first obstacle image is utilized, and based on this, candidate image sequences are obtained through evolution. Furthermore, some images in the candidate image sequence will serve as reference guide images in the generation of candidate frame image samples. This data source preparation logic helps to improve the information richness of the sample image set, thereby enhancing the adaptability of the target prediction model trained based on the sample image set.

[0117] Furthermore, the aforementioned preset interval constrains the selection of multiple fourth obstacle images, which will constitute a baseline guide image sequence to participate in the fluid simulation, thereby obtaining multiple candidate frame image samples. Correspondingly, the preset interval also determines the time interval between the next frame image predicted by the trained target prediction model and the current frame image. For example, 8 time steps.

[0118] In practical applications, during the preparation of data sources, candidate data can be filtered to discard images with calculation errors (e.g., evolution failure to converge, or insufficient physical representation of the evolution process) or insufficient features (e.g., some obstacle images completely shield the fluid external force, resulting in minimal image changes). Of course, manual review can also be used to remove images where the LBM solver itself has failed, and images that cause significant differences between predicted and original values.

[0119] After selecting high-quality data sources, the mean and variance of the frame image sets and the obstacle image sets can be calculated and saved according to their dimensions (such as channel dimension and spatial dimension (i.e., scale)). Then, all image data are processed by subtracting the corresponding mean and dividing by the corresponding variance to transform all image data into a standard normal distribution with a mean of 1 and a variance of 0, thus constructing multiple sample image groups.

[0120] The image processing method provided in this application can effectively process frame images of 1024*1024 or larger. Simultaneously, when applying the target prediction model for inference, it also supports user-interactive adjustments to the guidance image. The XLB fluid simulation framework is commonly used in related technologies to predict the next frame image. The XLB fluid simulation framework provides a Python-based, differentiable, massively parallel lattice Boltzmann library to support fluid simulation. Compared to the performance of the XLB fluid simulation framework in predicting the next frame image, experiments show that the target prediction model has a 1.67x speedup for "1024*1024, simple internal boundary" scenarios and a 1.22x speedup for "1024*1024, complex internal boundary" scenarios, as shown in Table 1 below:

[0121]

[0122] Table 1

[0123] Compared to simple internal boundaries, complex internal boundaries are mainly characterized by having more sub-obstacle objects that constitute the obstacle object, and the boundary shapes of the sub-obstacle objects are more tortuous.

[0124] The target prediction model can be trained using a cluster of four RTX 3080 graphics cards, and experiments have shown that model convergence can be achieved in 18 hours. The trained target prediction model exhibits an error of approximately 2e-4 on the training set (the dataset used during model training) and approximately 1.7e-2 on the test set (a dataset never used during model training). Furthermore, the open-source deep learning framework JAX can be used for model training. For model inference involving application scenarios, dedicated CUDA programs can be written to achieve faster simulations.

[0125] As can be seen from the technical solutions provided in the embodiments of this application above, the embodiments of this application provide an image processing scheme that balances efficiency and accuracy for fluid simulation. Under the setting of representing the state of each fluid grid in a fluid system at a specific time point using frame images, the embodiments of this application take a first frame image (indicating the current time point) and a target guidance image as input, and use a target prediction model to output a second frame image (indicating the next time point). Using the lattice Boltzmann algorithm as the preset fluid simulation algorithm, when representing the state of the frame image, based on the characteristics of the lattice Boltzmann algorithm, a relationship is established between the image features corresponding to the pixels and the fluid distribution function corresponding to the fluid grid points based on positional association, providing support for the accuracy and effectiveness of fluid simulation. Predicting the second frame image using the target prediction model helps improve the efficiency and accuracy of obtaining the second frame image. Simultaneously, during the training of the target prediction model, the first network processes the global dimensional information of the current frame image sample through encoding and decoding, the second network processes the local dimensional information of the current frame image sample through the lattice Boltzmann algorithm, and the third network integrates the processed global and local dimensional information to predict the next frame image. The model parameters are then adjusted based on the difference between the predicted next frame image and the actual next frame image sample. The global and local dimensional information of the current frame image sample are processed using different calculation methods. This avoids information omissions that could affect the prediction of the next frame image. Furthermore, compared to the original data volume of the current frame image sample, the smaller amount of local dimensional information improves the efficiency of the lattice Boltzmann algorithm, and the focus on key regions in the current frame image sample also contributes to the accuracy of the next frame image prediction. Therefore, using the trained target prediction model for next frame image prediction improves the adaptability and reliability of the next frame image prediction and effectively handles situations where the first frame image has a large amount of data.

[0126] This application also provides an image processing apparatus, such as... Figure 11 As shown, the image processing apparatus 110 includes:

[0127] Frame image acquisition module 1101: used to acquire a first frame image, the first frame image indicating a preset fluid system at the current time point, the first frame image characterizing the fluid distribution function corresponding to the fluid grid point in the preset fluid system with the image features corresponding to the pixel point, the pixel point and the fluid grid point having a positional relationship;

[0128] Guidance image acquisition module 1102: used to acquire target guidance image, the target guidance image being used to characterize the distribution of target obstacle objects in the preset fluid system;

[0129] Prediction module 1103: Used to take the first frame image and the target guidance image as input, and output a second frame image using a target prediction model. The second frame image indicates the preset fluid system at the next time point. The target prediction model is obtained by training a preset model based on multiple sample image groups. The sample image group consists of current frame image samples and guidance image samples, and carries next frame image samples. The preset model includes a first network, a second network, and a third network. The first network is used to encode and decode the current frame image samples sequentially to obtain base image samples. The second network is used to predict the change information of key regions in the current frame image samples based on the lattice Boltzmann algorithm. The key regions are determined under the guidance of the guidance image samples. The third network is used to predict the next frame image based on the base image samples and the change information. During training, the preset model is adjusted based on the difference between the predicted next frame image and the next frame image samples.

[0130] In one embodiment, the target prediction model is trained by the following steps: taking the current frame image sample as input, the base image sample is output using the first network; taking the current frame image sample and the guide image sample as input, the change information is output using the second network; taking the base image sample and the change information as input, the next frame image is predicted using the third network; and the parameters of the preset model are adjusted based on the difference between the predicted next frame image and the next frame image sample to obtain the target prediction model.

[0131] In one embodiment, the first network includes an encoding subnetwork and a decoding subnetwork. The encoding subnetwork includes a plurality of feature extraction units arranged in sequence, each of which consists of a first convolutional layer and a pooling layer for compressing the image scale. The decoding subnetwork includes a plurality of feature restoration units arranged in sequence, each of which consists of a second convolutional layer and a deconvolutional layer for enlarging the image scale. The step of taking the current frame image sample as input and outputting the base image sample using the first network includes: processing the fused image sample using the encoding subnetwork to obtain a corresponding encoding result, wherein the fused image sample indicates the fusion result of the current frame image sample and the guiding image sample; and processing the encoding result using the decoding subnetwork to obtain the base image sample.

[0132] In one embodiment, the device further includes a fusion module;

[0133] The fusion module is configured to: fuse the current frame image sample and the guide image sample to obtain a candidate fused image, wherein the scale of the current frame image sample and the guide image sample is both a first scale and the first scale is greater than a second scale; and perform downsampling processing on the candidate fused image to obtain the fused image sample, wherein the fused image sample uses the second scale.

[0134] The step of processing the encoding result using the decoding subnetwork to obtain the base image sample includes: inputting the encoding result into the decoding subnetwork to obtain a candidate base image, wherein the candidate base image adopts the second scale; and performing upsampling processing on the candidate base image to obtain the base image sample, wherein the base image sample adopts the first scale.

[0135] In one embodiment, the step of using the current frame image sample and the guide image sample as input and outputting the change information using the second network includes: using the second network to segment the current frame image sample into regions to obtain multiple first local images, wherein the scale of the first local images is a preset scale; using the second network to determine multiple second local images containing the representational boundaries of the sample obstacle objects from the multiple first local images based on the distribution of the sample obstacle objects represented by the guide image sample and the correlation between the guide image sample and the current frame image sample; using the second network to process the multiple second local images based on the lattice Boltzmann algorithm to obtain multiple corresponding third local images, and determining the multiple third local images as the change information.

[0136] In one embodiment, the step of using the base image samples and the change information as input and the output of the third network to predict the next frame image includes: for each third local image, using the third network to determine the current local image at the corresponding position in the base image samples based on the position of the second local image corresponding to the third local image in the current frame image, and replacing or covering the current local image with the third local image to achieve the prediction of the next frame image.

[0137] In one embodiment, the device further includes a parameter tuning module;

[0138] The parameter tuning module is used to, after completing the prediction of the next frame image for the current batch of sample sets, determine the difference between the predicted next frame image and the next frame image sample for each sample image group in the current batch of sample sets, so as to obtain a difference set, wherein the current batch of sample sets includes a preset number of sample image groups; and adjust the parameters of the preset model based on the difference set to obtain the target prediction model.

[0139] In one embodiment, the apparatus further includes a data source construction module;

[0140] The data source construction module is used to: randomly generate a candidate image set using a directed distance field algorithm, the candidate image set including multiple first obstacle images; for each first obstacle image, using the target number as the time step, perform evolution processing on the first obstacle image using a direct calculation method to obtain a corresponding candidate image sequence, in which the target number of second obstacle images are arranged in chronological order of image generation time; extract sequence segments from the candidate image sequence, the sequence segments including multiple third obstacle images; determine multiple fourth obstacle images from the sequence segments at preset intervals; use the multiple fourth obstacle images as a reference to guide the image sequence, and use the lattice Boltzmann algorithm to simulate a preset fluid object to obtain multiple candidate frame image samples, the multiple candidate frame image samples and the multiple fourth obstacle images are the data source for constructing the sample image group.

[0141] It should be noted that the apparatus and method embodiments described in the device embodiments are based on the same inventive concept.

[0142] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0143] This application also provides a computer-readable storage medium storing at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the above-described method. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0144] This application also provides an electronic device, which includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the at least one processor to implement the above method.

[0145] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0146] Figure 12 A block diagram of an electronic device according to an embodiment of this application is shown. (Refer to...) Figure 12 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions executable by the processing component 1922. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0147] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0148] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0149] This application may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having on it at least one instruction or at least a program segment for causing a processor to implement various aspects of this application.

[0150] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0151] At least one instruction or program segment described herein may be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives at least one instruction or at least one program segment from the network and forwards the instruction or program segment to a computer-readable storage medium in the respective computing / processing device.

[0152] At least one instruction or at least one program segment used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C+, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. At least one instruction or at least one program segment may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information of at least one instruction or at least one program to implement various aspects of this application.

[0153] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by at least one instruction or at least a piece of program.

[0154] The at least one instruction or at least one program segment can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, it creates means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. The at least one instruction or at least one program segment can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0155] At least one instruction or at least one program may be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions specified in the blocks may occur in a different order than those specified in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0157] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technological improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, characterized in that, The method includes: Acquire a first frame image, which indicates a preset fluid system at the current time point. The first frame image uses image features corresponding to pixels to characterize the fluid distribution function corresponding to the fluid grid points in the preset fluid system. The pixels and the fluid grid points have a positional relationship. Acquire a target guidance image, which is used to characterize the distribution of target obstacle objects in the preset fluid system; Using the first frame image and the target guidance image as input, a second frame image is output using a target prediction model. The second frame image indicates the preset fluid system at the next time point. The target prediction model is obtained by training a preset model based on multiple sample image groups. The sample image group consists of current frame image samples and guidance image samples, and carries the next frame image sample. The preset model includes a first network, a second network, and a third network. The first network is used to encode and decode the current frame image samples sequentially to obtain a base image sample. The second network is used to predict the change information of key regions in the current frame image samples based on the lattice Boltzmann algorithm. The key regions are determined under the guidance of the guidance image samples. The third network is used to predict the next frame image based on the base image samples and the change information. During training, the preset model is adjusted based on the difference between the predicted next frame image and the next frame image samples.

2. The method according to claim 1, characterized in that, The target prediction model is obtained through the following steps: Using the current frame image sample as input, the first network outputs the base image sample; Using the current frame image sample and the guidance image sample as input, the second network outputs the change information; Using the base image samples and the change information as input, the third network is used to predict the next frame image; Based on the difference between the predicted next frame image and the next frame image sample, the parameters of the preset model are adjusted to obtain the target prediction model.

3. The method according to claim 1 or 2, characterized in that, The first network includes an encoding subnetwork and a decoding subnetwork. The encoding subnetwork includes multiple feature extraction units arranged sequentially. Each feature extraction unit consists of a first convolutional layer and a pooling layer for compressing the image scale. The decoding subnetwork includes multiple feature restoration units arranged sequentially. Each feature restoration unit consists of a second convolutional layer and a deconvolutional layer for enlarging the image scale. The step of taking the current frame image sample as input and outputting the base image sample using the first network includes: The fused image samples are processed using the coding sub-network to obtain the corresponding coding result, wherein the fused image sample indicates the fusion result of the current frame image sample and the guide image sample; The encoding result is processed using the decoding sub-network to obtain the base image sample.

4. The method according to claim 3, characterized in that, Before processing the fused image samples using the coding sub-network to obtain the corresponding coding result, the method further includes: When the scale of the current frame image sample and the guide image sample are both the first scale and the first scale is greater than the second scale, the current frame image sample and the guide image sample are fused to obtain a candidate fused image, and the candidate fused image adopts the first scale; The candidate fused image is downsampled to obtain the fused image sample, and the fused image sample adopts the second scale; The step of processing the encoding result using the decoding sub-network to obtain the base image sample includes: The encoding result is input into the decoding sub-network to obtain a candidate basis image, wherein the candidate basis image adopts the second scale; The candidate base image is upsampled to obtain the base image sample, and the base image sample adopts the first scale.

5. The method according to claim 1 or 2, characterized in that, The step of taking the current frame image sample and the guide image sample as input and outputting the change information using the second network includes: The second network is used to segment the current frame image sample into regions to obtain multiple first local images, the scale of which is a preset scale; The second network is used to determine multiple second local images containing the representational boundaries of the sample obstacle objects from the multiple first local images based on the distribution of the sample obstacle objects represented by the guide image samples and the correlation between the guide image samples and the current frame image samples. The second network is used to process the multiple second local images based on the lattice Boltzmann algorithm to obtain multiple corresponding third local images, and the multiple third local images are determined to be the change information.

6. The method according to claim 5, characterized in that, The step of using the base image samples and the change information as input, and using the output of the third network to predict the next frame image, includes: For each of the third local images, the third network determines the current local image at the corresponding position in the base image sample based on the position of the second local image corresponding to the third local image in the current frame image, and replaces or covers the current local image with the third local image to achieve the prediction of the next frame image.

7. The method according to claim 1 or 2, characterized in that, After using the base image samples and the change information as input, and performing image prediction for the next frame using the output of the third network, the method further includes: When the prediction of the next frame image for the current batch of sample sets is completed, for each sample image group in the current batch of sample sets, the difference between the predicted next frame image and the next frame image sample is determined to obtain a difference set, wherein the current batch of sample sets includes a preset number of sample image groups. Based on the difference set, the parameters of the preset model are adjusted to obtain the target prediction model.

8. The method according to claim 1 or 2, characterized in that, The method further includes: A candidate image set is randomly generated using a directed distance field algorithm, and the candidate image set includes multiple first obstacle images; For each first obstacle image, the first obstacle image is processed by direct calculation method with the target number as the time step to obtain a corresponding candidate image sequence. The target number of second obstacle images in the candidate image sequence are arranged in the order of image generation time. Extract sequence segments from the candidate image sequence, the sequence segments including multiple third obstacle images; Multiple fourth obstacle images are determined from the sequence segments at preset intervals; Using the multiple fourth obstacle images as a reference to guide the image sequence, the lattice Boltzmann algorithm is used to simulate the preset fluid object to obtain multiple candidate frame image samples. The multiple candidate frame image samples and the multiple fourth obstacle images are the data sources for constructing the sample image group.

9. An image processing apparatus, characterized in that, The device includes: Frame image acquisition module: used to acquire a first frame image, the first frame image indicating a preset fluid system at the current time point, the first frame image characterizing the fluid distribution function corresponding to the fluid grid point in the preset fluid system with the image features corresponding to the pixel point, the pixel point and the fluid grid point have a positional relationship; Guidance image acquisition module: used to acquire target guidance images, which are used to characterize the distribution of target obstacle objects in the preset fluid system; The prediction module is used to take the first frame image and the target guidance image as input, and output a second frame image using a target prediction model. The second frame image indicates the preset fluid system at the next time point. The target prediction model is obtained by training a preset model based on multiple sample image groups. The sample image group consists of current frame image samples and guidance image samples, and the sample image group carries the next frame image sample. The preset model includes a first network, a second network, and a third network. The first network is used to encode and decode the current frame image samples sequentially to obtain a base image sample. The second network is used to predict the change information of key regions in the current frame image samples based on the lattice Boltzmann algorithm. The key regions are determined under the guidance of the guidance image samples. The third network is used to predict the next frame image based on the base image samples and the change information. During training, the preset model is adjusted based on the difference between the predicted next frame image and the next frame image samples.

10. An electronic device, characterized in that, The electronic device includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the at least one processor to implement the image processing method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the image processing method as described in any one of claims 1-8.

12. A computer program product, characterized in that, The computer program product includes at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the image processing method as described in any one of claims 1-8.

Images