A method for constructing an image sequence prediction model and an image sequence prediction method

Abstract

The present invention discloses a method for constructing an image sequence prediction model and an image sequence prediction method, the method comprising: inputting a preset training image sequence for model training into the image sequence prediction model; obtaining a time step to be interpolated; for the time step to be interpolated, using a preset interpolation initial image and a hidden state outputted at the previous time step of the time step to be interpolated as inputs of neurons at the time step to be interpolated, and then outputting the hidden state of the time step to be interpolated to the neurons at the next time step to complete data interpolation at the time step to be interpolated, wherein the neuron is a ConvGRU‑ODE unit; outputting a predicted image sequence of the image sequence prediction model, calculating a loss function according to a label image corresponding to the training image sequence and a predicted image sequence, and training the image sequence prediction model until the loss function converges. The present invention can realize image prediction based on an image sequence with non-equal time intervals.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing based on deep learning, and in particular to a method for constructing an image sequence prediction model and an image sequence prediction method. Background Art

[0002] With the development of deep learning technology and image processing technology, image sequence prediction has made good research progress in application scenarios such as video prediction and target trajectory prediction. However, existing deep learning-based image sequence prediction methods, such as image sequence prediction methods based on ConvLSTM models, require image sequences with equal time intervals as training data during model training. In some application scenarios of image sequence prediction, the known image sequences are of unequal time intervals. For example, Figure 1 The figure shows the infrared scanning images of the whole disk generated by the Fengyun-4-A satellite. The generation time of the five images is 0:00, 0:15, 1:00, 2:00, and 2:45, and the time intervals of the five images are 15 minutes, 45 minutes, 60 minutes, and 45 minutes, respectively. It can be seen that the five images are image sequences with non-equal time intervals. In the image extrapolation scenario based on the satellite images of the Fengyun-4-A satellite, the known satellite image sequences have non-equal time intervals, and the existing image sequence methods based on equal time intervals cannot accurately realize image sequence prediction. Summary of the invention

[0003] The problem solved by the present invention is that the existing image sequence method cannot accurately predict the image sequence based on the image sequence with unequal time intervals.

[0004] The present invention proposes a method for constructing an image sequence prediction model, wherein the image sequence prediction model includes an encoder and a decoder, wherein the encoder includes at least one layer of ConvGRU-ODE units, and the hidden state solution formula of the ConvGRU-ODE unit is the Euler solution formula of the hidden state ordinary differential equation; the method for constructing the image sequence prediction model includes:

[0005] Inputting a preset training image sequence for model training into the image sequence prediction model;

[0006] Get the time step to be interpolated;

[0007] For time steps other than the time step to be interpolated, the actually input training image and the hidden state of the previous time step are used as input, and the hidden state of the time step is output to the neuron of the next time step;

[0008] For the time step to be interpolated, the preset interpolation initial image and the hidden state outputted at the previous time step of the time step to be interpolated are used as inputs of the neurons of the time step to be interpolated, and then the hidden state of the time step to be interpolated is outputted to the neurons of the next time step to complete the data interpolation at the time step to be interpolated, wherein the neuron is the ConvGRU-ODE unit;

[0009] A predicted image sequence output by the image sequence prediction model is obtained, a loss function is calculated according to the label image corresponding to the training image sequence and the predicted image sequence, and the image sequence prediction model is trained until the loss function converges.

[0010] Optionally, the hidden state solution formula of the ConvGRU-ODE unit is as follows:

[0011]

[0012]

[0013] Among them, h(t) refers to the hidden state at time step t, h(t-1) refers to the hidden state at time step (t-1), ΔT refers to the time interval between the corresponding moment of time step t and the corresponding moment of time step (t-1), z(t) refers to the value of the update gate at time step t, and g(t) refers to the value of the candidate activation gate at time step t.

[0014] Optionally, obtaining the time step to be interpolated includes:

[0015] Acquire an interpolation mark, wherein the interpolation mark is generated based on a time interval between two adjacent training images in the training image sequence and a preset fixed time interval;

[0016] Determine two adjacent training images that need to be interpolated and the number of images that need to be interpolated according to the interpolation identifier;

[0017] The time step to be interpolated is determined according to the two adjacent training images to be interpolated and the number of interpolations required.

[0018] Optionally, the decoder includes at least one layer of core processing units, the core processing unit is a ConvGRU-ODE unit or one of the ConvGRU units, and the number of core processing unit layers in the decoder is equal to the number of ConvGRU-ODE unit layers in the encoder.

[0019] Optionally, the encoder includes two or more layers of ConvGRU-ODE units. For any layer of the ConvGRU-ODE units in the encoder, it outputs the hidden state to the ConvGRU-ODE units of the next layer at the same time step and the ConvGRU-ODE units in the encoder of the same layer at the next time step or the core processing unit in the decoder of the same layer at the next time step.

[0020] Optionally, the decoder includes two or more layers of core processing units, and for any layer of the core processing units in the decoder, it outputs the hidden state to the core processing units of the previous layer at the same time step and the core processing units of the same layer at the next time step.

[0021] Optionally, in the encoder, each layer of the ConvGRU-ODE unit includes at least one downsampling layer before it, and in the decoder, each layer of the core processing unit includes at least one upsampling layer after it.

[0022] The present invention also proposes an image sequence prediction method, comprising:

[0023] Acquire a benchmark image sequence, wherein the benchmark image sequence is an image sequence with unequal time intervals or an image sequence with equal time intervals; input the benchmark image sequence into a constructed image sequence prediction model to obtain a predicted image sequence output by the image sequence prediction model, wherein the image sequence prediction model is constructed based on the construction method of the image sequence prediction model as described above.

[0024] The present invention also proposes a computer device, including a computer-readable storage medium storing a computer program and a processor. When the computer program is read and executed by the processor, the method for constructing an image sequence prediction model or the method for predicting an image sequence as described above is implemented.

[0025] The present invention also proposes a computer-readable storage medium, which stores a computer program. When the computer program is read and executed by a processor, the method for constructing an image sequence prediction model or the method for predicting an image sequence is implemented.

[0026] The present invention deeply combines ConvGRU with GRU-ODE, converts the hidden state solution formula of the ConvGRU-ODE unit into the Euler solution formula of the hidden state ordinary differential equation, realizes the image interpolation of the specified time, and interpolates the image sequence with non-equal time intervals into the image sequence with equal time intervals, which is convenient for realizing image prediction based on the image sequence with non-equal time intervals, and overcomes the problem that the existing model cannot predict the image sequence with non-equal time intervals. At the same time, during the training process, by deeply combining ConvGRU with GRU-ODE, the image prediction algorithm and the interpolation algorithm are trained simultaneously by back propagation, the robustness of the image sequence prediction model is enhanced, and the error accumulation caused by the module training is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Full disk infrared scan generated for the Fengyun-4A satellite;

[0028] Figure 2 A schematic diagram of an embodiment of an image sequence prediction model in an embodiment of the present invention;

[0029] Figure 3 Schematic diagram of an embodiment of a method for constructing an image sequence prediction model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0031] In one embodiment of the present invention, the image sequence prediction model includes an encoder and a decoder, the encoder includes at least one layer of ConvGRU-ODE unit, and the hidden state solution formula of the ConvGRU-ODE unit is the Euler solution formula of the hidden state ordinary differential equation.

[0032] Among them, the ConvGRU-ODE unit is a convolutional gate recurrent-ordinary differential unit.

[0033] ConvGRU is a fusion of convolution and GRU (Gated Recurrent Network). By converting the point multiplication operation in GRU into a convolution operation, ConvGRU has the advantage of extracting both temporal and spatial features at the same time. The calculation formula of ConvGRU is as follows:

[0034] z t =σ(W xz *x t +W hz *h t-1 +b z ),

[0035] R t =σ(Wxr *x t +W hr *h t-1 +b r ),

[0036] g t =f(W xh *x t +R t ☉(W hh *h t-1 )+b h ),

[0037] h t =(1-z t )☉g t +z t ☉h t-1 ,

[0038] Among them, z t is the update gate, R t is the reset gate, g t is the candidate activation state, h t is the hidden state. t is the input of the current time step, h t-1 is the hidden state of the previous time step, W xz , W hz , W xr , W hr , W xh , W hh is the weight, b z 、b r 、b h is the bias, σ and f are activation functions, “*” refers to the convolution operation, and “☉” refers to the multiplication of the corresponding elements of the matrix.

[0039] ODE refers to ordinary differential equations. By expressing the calculation formula of the hidden state in ConvGRU as the Euler solution formula of ordinary differential equations, the data interpolation at a specified time can be achieved. The calculation formula of the hidden state in ConvGRU is as follows:

[0040] h t =(1-z t )☉g t +z t ☉h t-1 ,

[0041] Derivative the calculation formula, we can get the ordinary differential equation:

[0042] Δh t =h t -h t-1 =(1-z t )☉gt +z t ☉h t-1 -h t-1 ,

[0043]

[0044] The Euler method is used to obtain h(t) at the specified time, that is, the Euler solution formula of the hidden state ordinary differential equation of the ConvGRU-ODE unit is obtained:

[0045]

[0046] Among them, h(t) refers to the hidden state at time step t, h(t-1) refers to the hidden state at time step (t-1), z(t) refers to the value of the update gate at time step t, and g(t) refers to the value of the candidate activation gate at time step t. The meaning is: h(t) is derived with respect to the moment corresponding to the t time step, ΔT refers to the time interval between the moment corresponding to the t time step and the moment corresponding to the (t-1) time step, and the time interval is preset. For example, the time interval between the moment corresponding to the t time step and the moment corresponding to the (t-1) time step can be preset to 15 minutes or 25 minutes.

[0047] By expressing the hidden state solution formula of the ConvGRU-ODE unit as the Euler solution formula of the hidden state ordinary differential equation, when the image data of a partial time series is known, the Euler solution formula can be used to interpolate the value at a specified time to achieve time-controllable interpolation to meet specific interpolation requirements.

[0048] Optionally, the decoder includes at least one layer of core processing units, the core processing unit is a ConvGRU-ODE unit or one of the ConvGRU units, and the number of core processing unit layers in the decoder is equal to the number of ConvGRU-ODE unit layers in the encoder.

[0049] Among them, because no interpolation is required during the decoding process, the core processing unit can be a ConvGRU unit.

[0050] Furthermore, in one embodiment, the encoder includes two or more layers of the ConvGRU-ODE units. Since the number of core processing unit layers in the decoder is equal to the number of ConvGRU-ODE unit layers in the encoder, that is, the number of layers is equal, the number of core processing unit layers in the decoder is also two or more. For any layer of the ConvGRU-ODE unit in the encoder, it outputs the hidden state to the next layer of the ConvGRU-ODE unit at the same time step and the ConvGRU-ODE unit or the ConvGRU unit at the same layer at the next time step.

[0051] Among them, for the last layer of ConvGRU-ODE unit in the encoder, on the one hand, it outputs the hidden state to the ConvGRU-ODE unit or the ConvGRU unit of the same layer in the next time step, and on the other hand, the hidden state can be finally output.

[0052] For the ConvGRU-ODE unit of any layer in the encoder, when its next time step is the ConvGRU-ODE unit in the encoder, it outputs the hidden state to the ConvGRU-ODE unit of the same layer in the next time step; when its next time step is the core processing unit in the decoder, it outputs the hidden state to the ConvGRU-ODE unit or ConvGRU unit of the same layer in the next time step.

[0053] The layer number of any ConvGRU-ODE unit in the encoder and the layer number of any core processing unit in the decoder are fixed, and the layer number refers to the nth layer. In the above implementation, the same layer of "the same layer in the next time step" refers to the layer number. The first layer is the previous layer of the second layer, the second layer is the previous layer of the third layer, and so on. The second layer is the next layer of the first layer, the third layer is the next layer of the second layer, and so on. Optionally, the closer the input feature map size is to the original map size, the lower or higher the layer numbers of the ConvGRU-ODE unit and the core processing unit are. For example, Figure 2 In the figure, for the first time step, the image I1 is output to the ConvGRU-ODE unit after the first downsample, and its input feature map size is closest to the original image size, which is defined as the first layer ConvGRU-ODE unit. The following two layers are the second layer ConvGRU-ODE unit and the third layer ConvGRU-ODE unit. For the n+1th time step, the input feature map size of the ConvGRU unit that outputs the image In+1 is closest to the original image size, which is defined as the first layer ConvGRU unit.

[0054] Figure 2A schematic diagram of an embodiment of an image sequence prediction model is shown. The encoder of the image sequence prediction model includes three layers of downsampling layers and three layers of ConvGRU-ODE units, and the decoder includes three layers of ConvGRU units and three layers of upsampling layers. Take the first layer of ConvGRU-ODE units at the first time step corresponding to image I1, the second layer of ConvGRU-ODE units at the first time step corresponding to image I1, and the first layer of ConvGRU-ODE units at the nth time step corresponding to image In as examples. For the first layer of ConvGRU-ODE units at the first time step of the input image I1, it includes two outputs: one output is to input the hidden state to the second layer of ConvGRU-ODE units at the first time step, and the other output is to input the hidden state to the first layer of ConvGRU-ODE units at the second time step. For the second layer ConvGRU-ODE unit of the first time step corresponding to image I1, it contains two outputs: one output is to input the hidden state to the third layer ConvGRU-ODE unit of the first time step, and the other output is to input the hidden state to the second layer ConvGRU-ODE unit of the second time step. For the first layer ConvGRU-ODE unit of the nth time step corresponding to image In, it contains two outputs: one output is to input the hidden state to the second layer ConvGRU-ODE unit of the nth time step, and the other output is to input the hidden state to the first layer ConvGRU-ODE unit of the decoder of the n+1th time step.

[0055] Furthermore, in one embodiment, the decoder includes two or more layers of core processing units, and for any layer of core processing units in the decoder, the hidden state is output to the core processing unit of the previous layer at the same time step and the core processing unit of the same layer at the next time step. In particular, for the topmost core processing unit in the decoder, on the one hand, it outputs the hidden state to the core processing unit of the same layer at the next time step, and on the other hand, it outputs the final prediction result.

[0056] Figure 2A schematic diagram of an embodiment of an image sequence prediction model is shown. The decoder includes three layers of ConvGRU units and three layers of upsampling layers. Taking the third layer ConvGRU unit of the n+1th time step corresponding to the image In+1 and the second layer ConvGRU unit of the n+1th time step corresponding to the image In+1 as examples, for the third layer ConvGRU unit of the n+1th time step corresponding to the image In+1, it includes two outputs: one output is to input the hidden state to the second layer ConvGRU unit of the n+1th time step, and the other output is to input the hidden state to the third layer ConvGRU unit of the n+2th time step. Taking the second layer ConvGRU unit of the n+1th time step corresponding to the image In+1 as an example, it includes two outputs: one output is to input the hidden state to the first layer ConvGRU unit of the n+1th time step, and the other output is to input the hidden state to the second layer ConvGRU unit of the n+2th time step.

[0057] Optionally, in the encoder, each layer of the ConvGRU-ODE unit includes at least one downsampling layer before it, and in the decoder, each layer of the core processing unit includes at least one upsampling layer after it.

[0058] Through the downsampling layer, images of different dimensions and sizes are obtained, so that the ConvGRU-ODE unit can extract feature information of different dimensions and scales. Through the upsampling layer, the image dimension is reduced and the image size is gradually transformed to the original image size.

[0059] Figure 2 FIG. 2 shows a schematic diagram of an embodiment of an image sequence prediction model. Figure 2 The image sequence prediction model includes an encoder and a decoder. The encoder is composed of three layers of downsampling layers and three layers of ConvGRU-ODE units (convolution gate recurrent-ordinary differential units). The decoder is composed of three layers of ConvGRU units (convolution gate recurrent units) and three layers of upsampling layers. The image sequence prediction model outputs m predicted images (In+1, In+2, ..., In+m) based on n input images (I1, I2 ... In), where n and m may be equal or unequal. After the n input images are encoded by the encoder, they are decoded by the decoder to obtain t predicted images.

[0060] Furthermore, after the basic framework of the image sequence prediction model is constructed by the above steps, the image sequence prediction model construction method described below is further used to construct a practical image sequence prediction model. Figure 3 , the method for constructing the image sequence prediction model includes:

[0061] Step S100: input a preset training image sequence for model training into an encoder in the image sequence prediction model.

[0062] The training data includes multiple image sequences and their corresponding label images, where the image sequence refers to an image sequence with a certain time sequence, and the label image is also an image sequence.

[0063] Since the image sequence prediction model adopts the basic architecture of ConvGRU, when the training image sequence is input into the encoder, the training image sequence is input in chronological order, with one training image being input in turn at each time step.

[0064] The training data may include image sequences with unequal time intervals and image sequences with equal time intervals, and the image sequence prediction model obtained by training can perform image prediction based on image sequences with unequal time intervals or image sequences with equal time intervals. The training data may also only include image sequences with unequal time intervals, so as to realize image prediction based on image sequences with unequal time intervals.

[0065] Step S200, obtaining the time step to be interpolated.

[0066] For image sequences with unequal time intervals input into the image sequence prediction model, data interpolation is required to achieve image prediction. Therefore, the time step to be interpolated must be determined first.

[0067] Optionally, when the training image sequence is input into the encoder, an interpolation identifier for determining the time step to be interpolated is also input. Specifically, step S200 includes:

[0068] Acquire an interpolation mark, wherein the interpolation mark is generated based on a time interval between two adjacent training images in the training image sequence and a preset fixed time interval;

[0069] Determine two adjacent training images that need to be interpolated and the number of images that need to be interpolated according to the interpolation identifier;

[0070] The time step to be interpolated is determined according to the two adjacent training images to be interpolated and the number of interpolations required.

[0071] The interpolation flag can be input into the encoder together with the training image sequence. The interpolation flag can be pre-calculated and can include the flags of two adjacent training images and the number of interpolations required. For example, the interpolation flag (1-2, 2) indicates that the number of interpolations required between the first image and the second image is 2.

[0072] The two adjacent training images that need to be interpolated refer to the time between the two adjacent training images that need to be interpolated. The number of interpolations required refers to the number of images that need to be interpolated between the two adjacent training images.

[0073] The interpolation mark is generated based on the time interval between two adjacent training images in the training image sequence and the preset fixed time interval. Specifically, the time interval between two adjacent training images is divided by the preset fixed time interval to obtain a quotient, and (quotient-1) is used as the number of interpolations required between the two adjacent training images. For example, the time interval between two adjacent training images is 45 minutes, and the preset fixed time interval is 15 minutes. The former is divided by the latter 45÷15 to obtain a quotient=3, and (3-1) is used as the number of interpolations required between the two adjacent training images.

[0074] After determining the two adjacent training images to be interpolated and the number of interpolations required, the time step to be interpolated can be determined based on the temporal relationship between the two adjacent training images and the image to be interpolated and the time steps corresponding to the two adjacent training images. Figure 1 As shown, assuming that the preset fixed time interval is 15 minutes, the generation time of the input image sequence is 0:00, 0:15, 1:00, 2:00, and 2:45 respectively, the time interval between the first image and the second image is 15 minutes, then the number of interpolations required between the first image and the second image is (15÷15-1)=0, the time interval between the second image and the third image is 45 minutes, then the number of interpolations required between the second image and the third image is (45÷15-1)=2, the time interval between the third image and the fourth image is 60 minutes, then the number of interpolations required between the third image and the fourth image is (60÷15-1)=3, the time interval between the fourth image and the fifth image is 45 minutes, then the number of interpolations required between the fourth image and the fifth image is If (45÷15-1)=2, then no interpolation is required between the 1st and 2nd images, then the 1st and 2nd images are the 1st and 2nd time steps respectively, and 2 images need to be interpolated between the 2nd and 3rd images, then the time steps to be interpolated between the 2nd and 3rd images are the 3rd and 4th time steps, and the 3rd image is the 5th time step. 3 images need to be interpolated between the 3rd and 4th images, then the time steps to be interpolated between the 3rd and 4th images are the 6th, 7th and 8th time steps, and the 4th image is the 9th time step. The number of interpolations required between the 4th and 5th images is 2, then the time steps to be interpolated between the 4th and 5th images are the 10th and 11th time steps, and the 5th image is the 12th time step.

[0075] By obtaining the interpolation mark generated based on the time interval between two adjacent training images in the training image sequence and the preset fixed time interval, the two adjacent training images that need to be interpolated and the number of interpolations required are determined, and then the time step to be interpolated is determined, so that the image sequence prediction model can clearly identify the time step to be interpolated, so as to determine the time step that needs to be interpolated for each image sequence with different unequal time intervals, so as to realize image prediction based on image sequences with different unequal time intervals.

[0076] Step S300: For time steps other than the time step to be interpolated, the actually input training image and the hidden state of the previous time step are used as input, and the hidden state of the time step is output to the neuron of the next time step.

[0077] The time steps other than the time step to be interpolated include the first time step and the time steps other than the first time step. For the first time step, the actually input training image and the preset initial hidden state are used as the neuron input of the first time step, and the hidden state of the first time step is output to the neuron of the next time step. For the time step to be interpolated and the time step other than the first time step, the actually input training image and the hidden state of the previous time step are used as input, and the hidden state of the current time step is output to the neuron of the next time step.

[0078] Step S400, for the time step to be interpolated, the preset interpolation initial image and the hidden state outputted at the previous time step of the time step to be interpolated are used as the input of the neuron of the time step to be interpolated, and then the hidden state of the time step to be interpolated is outputted to the neuron of the next time step to complete the data interpolation at the time step to be interpolated, wherein the neuron is the ConvGRU-ODE unit.

[0079] The preset interpolation initial image may be an all-zero image.

[0080] From the above, we can know that the hidden state solution formula of the ConvGRU-ODE unit is the Euler solution formula of the hidden state ordinary differential equation, which can realize the data interpolation of the interpolation time step. The Euler solution formula is as follows:

[0081]

[0082]

[0083] h(t) refers to the hidden state at time step t, h(t-1) refers to the hidden state at time step (t-1), z(t) refers to the value of the update gate at time step t, and g(t) refers to the value of the candidate activation gate at time step t. ΔT refers to the time interval between the corresponding moment of time step t and the corresponding moment of time step (t-1).

[0084] Through the Euler solution formula of the hidden state ordinary differential equation mentioned above, the hidden state of the time step to be interpolated can be calculated, and then the interpolation of the time step to be interpolated can be realized.

[0085] Step S500: After the encoder executes the above steps, the decoder decodes the output of the encoder to obtain a predicted image sequence output by the image sequence prediction model.

[0086] The output of the encoder is the hidden state output by the ConvGRU-ODE unit of the encoder at the last time step. In particular, if the encoder contains three layers of ConvGRU-ODE units, the output of the encoder is the three hidden states output by the three layers of ConvGRU-ODE units at the last time step.

[0087] Step S600, calculating a loss function according to the label image corresponding to the training image sequence and the predicted image sequence, and training the image sequence prediction model until the loss function converges.

[0088] The image sequence prediction model is trained by supervised learning. The output of the image sequence prediction model is obtained through forward propagation, and then the parameters of the image sequence prediction model are updated through back propagation. After multiple iterations, the image sequence prediction model converges.

[0089] It should be noted that the encoder and decoder are actually algorithms for implementing encoding functions and algorithms for implementing decoding functions, and the encoding and decoding algorithms are executed by a unified controller to implement the above steps S100 to S600.

[0090] By deeply combining ConvGRU with GRU-ODE, the hidden state solution formula of the ConvGRU-ODE unit is converted to the Euler solution formula of the hidden state ordinary differential equation, and the image interpolation of the specified time is realized, so as to interpolate the image sequence with non-equal time intervals into the image sequence with equal time intervals, which is convenient for realizing image prediction based on the image sequence with non-equal time intervals, and overcomes the problem that the existing model cannot predict the image sequence with non-equal time intervals. At the same time, during the training process, by deeply combining ConvGRU with GRU-ODE, the image prediction algorithm and the interpolation algorithm are trained simultaneously by back propagation, which strengthens the robustness of the image sequence prediction model and reduces the error accumulation caused by module training.

[0091] In one embodiment of the present invention, the image sequence prediction method includes:

[0092] Acquire a benchmark image sequence, wherein the benchmark image sequence is an image sequence with unequal time intervals or an image sequence with equal time intervals; input the benchmark image sequence into a constructed image sequence prediction model to obtain a predicted image sequence output by the image sequence prediction model, wherein the image sequence prediction model is constructed based on the construction method of the image sequence prediction model as described above.

[0093] The reference image sequence refers to a known image sequence, which can be a satellite image, a video image or other time-series image data. By inputting it into the constructed image sequence prediction model, a predicted image sequence can be obtained. The reference image sequence can be an image sequence with unequal time intervals or an image sequence with equal time intervals. The image sequence prediction model constructed by the image sequence prediction model construction method described above can realize image prediction based on image sequences with unequal time intervals and image sequences with equal time intervals.

[0094] The image sequence prediction model constructed by the image sequence prediction model construction method described above combines GRU-ODE with the encoding and decoding structure of ConvGRU to realize image interpolation and image sequence prediction on two-dimensional images, and further realize image prediction at non-equal time intervals.

[0095] In one embodiment of the present invention, a computer device includes a computer-readable storage medium storing a computer program and a processor. When the computer program is read and executed by the processor, the method for constructing an image sequence prediction model as described above or the method for predicting an image sequence as described above is implemented. The beneficial effects of the computer device of the present invention relative to the prior art are consistent with the method for constructing an image sequence prediction model as described above or the method for predicting an image sequence as described above, and are not described in detail herein.

[0096] In one embodiment of the present invention, a computer-readable storage medium stores a computer program, and when the computer program is read and executed by a processor, the method for constructing an image sequence prediction model or the method for predicting an image sequence as described above is implemented.

[0097] The reader should understand that in the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0098] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for constructing an image sequence prediction model, characterized in that: The image sequence prediction model includes an encoder and a decoder, the encoder includes at least one layer of ConvGRU-ODE unit, and the hidden state solution formula of the ConvGRU-ODE unit is the Euler solution formula of the hidden state ordinary differential equation; the construction method of the image sequence prediction model includes: Inputting a preset training image sequence for model training into the image sequence prediction model; Get the time step to be interpolated; For time steps other than the time step to be interpolated, the actually input training image and the hidden state of the previous time step are used as input, and the hidden state of the time step is output to the neuron of the next time step; For the time step to be interpolated, the preset interpolation initial image and the hidden state outputted at the previous time step of the time step to be interpolated are used as inputs of the neurons of the time step to be interpolated, and then the hidden state of the time step to be interpolated is outputted to the neurons of the next time step to complete the data interpolation at the time step to be interpolated, wherein the neuron is the ConvGRU-ODE unit; A predicted image sequence output by the image sequence prediction model is obtained, a loss function is calculated according to the label image corresponding to the training image sequence and the predicted image sequence, and the image sequence prediction model is trained until the loss function converges.

2. The method for constructing an image sequence prediction model according to claim 1, wherein: The hidden state solution formula of the ConvGRU-ODE unit is as follows: Among them, h(t) refers to the hidden state at time step t, h(t-1) refers to the hidden state at time step (t-1), ΔT refers to the time interval between the corresponding moment of time step t and the corresponding moment of time step (t-1), z(t) refers to the value of the update gate at time step t, and g(t) refers to the value of the candidate activation gate at time step t.

3. The method for constructing an image sequence prediction model according to claim 1, wherein: The obtaining of the time step to be interpolated comprises: Acquire an interpolation mark, wherein the interpolation mark is generated based on a time interval between two adjacent training images in the training image sequence and a preset fixed time interval; Determine two adjacent training images that need to be interpolated and the number of images that need to be interpolated according to the interpolation identifier; The time step to be interpolated is determined according to the two adjacent training images to be interpolated and the number of interpolations required.

4. The method for constructing an image sequence prediction model according to any one of claims 1 to 3, characterized in that: The decoder includes at least one layer of core processing units, which are ConvGRU-ODE units or ConvGRU units, and the number of core processing unit layers in the decoder is equal to the number of ConvGRU-ODE unit layers in the encoder.

5. The method for constructing an image sequence prediction model according to claim 4, characterized in that: The encoder includes two or more layers of ConvGRU-ODE units. For any layer of the ConvGRU-ODE units in the encoder, it outputs the hidden state to the ConvGRU-ODE units of the next layer at the same time step and the ConvGRU-ODE units in the encoder of the same layer at the next time step or the core processing unit in the decoder of the same layer at the next time step.

6. The method for constructing an image sequence prediction model according to claim 5, characterized in that: The decoder includes two or more layers of core processing units. For any layer of the core processing units in the decoder, it outputs the hidden state to the core processing units of the previous layer at the same time step and the core processing units of the same layer at the next time step.

7. The method for constructing an image sequence prediction model according to claim 4, characterized in that: In the encoder, each layer of the ConvGRU-ODE unit includes at least one downsampling layer before it, and in the decoder, each layer of the core processing unit includes at least one upsampling layer after it.

8. An image sequence prediction method, characterized in that: include: Acquire a reference image sequence, wherein the reference image sequence is an image sequence with unequal time intervals or an image sequence with equal time intervals; The reference image sequence is input into a constructed image sequence prediction model to obtain a predicted image sequence output by the image sequence prediction model, wherein the image sequence prediction model is constructed based on the construction method of the image sequence prediction model according to any one of claims 1 to 7.

9. A computer device, characterized in that: It comprises a computer-readable storage medium storing a computer program and a processor, wherein when the computer program is read and executed by the processor, the method for constructing an image sequence prediction model as claimed in any one of claims 1 to 7 or the image sequence prediction method as claimed in claim 8 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is read and executed by a processor, the method for constructing an image sequence prediction model according to any one of claims 1 to 7 or the image sequence prediction method according to claim 8 is implemented.

Images