Remote sensing image sequence repair method, device, electronic equipment and storage medium

Abstract

The present invention provides a method, device, electronic device and storage medium for repairing a remote sensing image sequence, which relates to the field of image processing. The method for repairing a remote sensing image sequence comprises: inputting a remote sensing image sequence to be repaired and a missing information mask sequence into a target repair network, performing repair processing at different scales through S repair modules in sequence, and obtaining S intermediate image sequences; repairing the missing values ​​in the remote sensing image sequence to be repaired according to the Sth intermediate image sequence output by the Sth repair module, and obtaining a target repair image sequence. The present invention solves the problem of low precision in remote sensing image repair, and achieves the effect of improving the precision of remote sensing image repair.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method, device, electronic equipment and storage medium for repairing a remote sensing image sequence. Background Art

[0002] When taking remote sensing images, there may be serious missing images. For example, cloud cover may prevent some areas from being captured by the sensor, resulting in large missing areas in the remote sensing image. Or the sensor may malfunction or malfunction during the shooting process, resulting in the failure to correctly collect data in certain areas, which in turn leads to local missing images.

[0003] Remote sensing images with local missing pixels need to be repaired. However, due to the uncertainty and complexity of missing data in remote sensing images, it is difficult to accurately infer the values ​​of missing pixels, resulting in low accuracy of remote sensing image repair. Summary of the invention

[0004] The problem solved by the present invention is the low precision of remote sensing image restoration.

[0005] In order to solve the above problems, the present invention provides a method, device, electronic equipment and storage medium for repairing a remote sensing image sequence.

[0006] In a first aspect, the present invention provides a method for repairing a remote sensing image sequence, comprising:

[0007] Inputting the remote sensing image sequence to be repaired and the missing information mask sequence into the target repair network, wherein the missing information mask sequence is used to mark the position of the missing value in the remote sensing image sequence to be repaired, and the target repair network includes S repair modules, wherein S is an integer greater than 1;

[0008] Performing restoration processing at different scales in sequence through S restoration modules, respectively, to obtain S intermediate image sequences, wherein the input data of the i-th restoration module among the S restoration modules includes the missing information mask sequence and the intermediate image sequence output by the i-1-th restoration module, the restoration scale of the i-th restoration module is smaller than the restoration scale of the i-1-th restoration module, i is greater than or equal to 2, and the input data of the first restoration module is the remote sensing image sequence to be restored and the missing information mask sequence;

[0009] The missing values ​​in the remote sensing image sequence to be repaired are repaired according to the Sth intermediate image sequence output by the Sth repair module to obtain a target repaired image sequence.

[0010] Optionally, the restoration processing is performed in sequence by S restoration modules at different scales to obtain S intermediate image sequences, including: when using the i-th restoration module to perform the restoration processing, the i-1-th intermediate image sequence is divided into blocks to obtain an image sequence block set, wherein the size of each block in the image sequence block set is the restoration scale of the i-th restoration module; the image sequence block set is embedded to obtain an embedded vector sequence; feature extraction processing is performed according to the embedded vector sequence and the missing information mask sequence by a spatiotemporal feature extraction submodule to obtain spatiotemporal correlation features; and the spatiotemporal correlation features are inversely embedded to obtain the i-th intermediate image sequence.

[0011] Optionally, the spatiotemporal feature extraction submodule performs feature extraction processing according to the embedded vector sequence and the missing information mask sequence, including: performing a masked self-attention operation on the embedded vector sequence along the time direction according to the missing information mask sequence to obtain a first associated feature; performing a masked self-attention operation on the first associated feature along the spatial direction according to the missing information mask sequence to obtain a second associated feature; and processing the second associated feature through a feedforward neural network to obtain the spatiotemporal associated feature.

[0012] Optionally, the performing a masked self-attention operation on the embedded vector sequence along the time direction according to the missing information mask block to obtain a first association feature includes: performing a mapping operation on the embedded vector sequence to obtain a query vector, a key vector, and a value vector; determining the first attention score matrix according to the query vector and the key vector; according to the missing information mask block, setting the attention scores corresponding to the positions in the first attention score matrix where the missing rate is greater than or equal to a first threshold to negative infinity, and setting the attention scores corresponding to the diagonals in the first attention score matrix to negative infinity according to the diagonal mask matrix to obtain a second attention score matrix; and using the second attention score matrix to learn the temporal association features in the embedded vector sequence to obtain the first association features.

[0013] Optionally, before inputting the remote sensing image sequence to be repaired and the missing information mask sequence into the target repair network, the method also includes: obtaining a training remote sensing image sequence, wherein the images included in the training remote sensing image sequence are complete remote sensing images; performing simulated degradation processing on the images in the training remote sensing image sequence to obtain a degraded remote sensing image sequence and a degraded missing value mask sequence; and iteratively training the initial repair network using the degraded remote sensing image sequence and the degraded missing value mask sequence to obtain the target repair network.

[0014] Optionally, the iterative training of the initial repair network using the degraded remote sensing image sequence and the degraded missing value mask sequence includes: when the initial repair network is trained for the mth time, the degraded remote sensing image sequence and the degraded missing value mask sequence are input into the mth training network, wherein m is a positive integer, and when m=1, the mth training network is the initial repair network; repair processing is performed at different scales by the S training repair modules included in the mth training network to obtain S training intermediate image sequences; loss values ​​are calculated based on the S training intermediate image sequences and the training remote sensing image sequence to obtain a target loss value; when the target loss value does not meet a preset condition, the network parameters in the mth training network are modified to obtain the m+1th training network, and the m+1th training network is used for the next training; when the target loss value meets a preset condition, the mth training network is determined as the target repair network.

[0015] Optionally, the calculating the loss value according to the S training intermediate image sequences and the training remote sensing image sequence to obtain the target loss value includes:

[0016] The target loss value is determined by the following formula

[0017]

[0018] in, is the loss value between n and y, n is the i-th training intermediate image sequence in the S training intermediate image sequences, and y is the training remote sensing image sequence;

[0019] Determined by the following formula

[0020]

[0021] in, is the pixel-level loss value between η and y, is the structural loss value between η and y, is the perceptual loss between η and y, λ s represents the weight of the structural loss value, λ p In a second aspect, the present invention provides a remote sensing image sequence restoration device, comprising:

[0022] An input unit, used for inputting a remote sensing image sequence to be repaired and a missing information mask sequence into a target repair network, wherein the missing information mask sequence is used to mark the position of missing values ​​in the remote sensing image sequence to be repaired, and the target repair network includes S repair modules, wherein S is an integer greater than 1;

[0023] A first restoration unit is configured to perform restoration processing at different scales through S restoration modules in sequence to obtain S intermediate image sequences, wherein the input data of the i-th restoration module among the S restoration modules includes the missing information mask sequence and the intermediate image sequence output by the i-1-th restoration module, the restoration scale of the i-th restoration module is smaller than the restoration scale of the i-1-th restoration module, i is greater than or equal to 2, and the input data of the first restoration module is the remote sensing image sequence to be restored and the missing information mask sequence;

[0024] The second restoration unit is used to restore missing values ​​in the remote sensing image sequence to be restored according to the Sth intermediate image sequence output by the Sth restoration module to obtain a target restored image sequence.

[0025] In a third aspect, the present invention provides an electronic device, including a memory and a processor;

[0026] The memory is used to store computer programs;

[0027] The processor is used to implement the remote sensing image sequence repair method as described in the first aspect when executing the computer program.

[0028] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is read and executed by a processor, the remote sensing image sequence repair method as described in the first aspect is implemented.

[0029] The beneficial effects of the remote sensing image sequence repair method, device, electronic device and storage medium provided by the present invention are as follows: the remote sensing image sequence to be repaired and the missing information mask sequence are used as the input of the target repair network. When the target repair network is repairing, it can determine the position in the remote sensing image that needs to be repaired, and can repair the missing position in the remote sensing image in a targeted manner, thereby ensuring the accuracy of the repair of the remote sensing image sequence. S repair modules are repaired in a step-by-step repair manner, that is, the output of the previous repair module is used as the input of the next repair module. Since the repair scale of the next repair module is smaller than the repair scale of the previous repair module, the repair result of the previous repair module is repaired in a smaller scale to obtain a more refined repair result. The step-by-step refinement repair strategy is adopted to progressively repair the missing values ​​of the remote sensing image sequence to be repaired, so that the repair of the missing values ​​is more accurate. Therefore, the problem of low precision of remote sensing image repair is solved, and the effect of improving the precision of remote sensing image repair is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of a process of repairing a remote sensing image sequence according to an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of the structure of a target repair network according to an embodiment of the present invention;

[0032] Figure 3 A structural schematic diagram of a repair module according to an embodiment of the present invention;

[0033] Figure 4 A schematic diagram of the structure of a spatiotemporal feature extraction module according to an embodiment of the present invention;

[0034] Figure 5 A schematic diagram of a process of training a repair network according to an embodiment of the present invention;

[0035] Figure 6 It is a structural schematic diagram of a remote sensing image sequence repair device according to an embodiment of the present invention;

[0036] Figure 7 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be interpreted as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0038] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0039] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optionally" means "optional embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0040] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0041] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0042] like Figure 1 As shown, a remote sensing image sequence repair method provided by an embodiment of the present invention includes:

[0043] Step S101, inputting a remote sensing image sequence to be repaired and a missing information mask sequence into a target repair network, wherein the missing information mask sequence is used to mark the position of missing values ​​in the remote sensing image sequence to be repaired, and the target repair network includes S repair modules.

[0044] In this embodiment, the remote sensing image sequence to be repaired is a time-series remote sensing image sequence, and the remote sensing image sequence to be repaired includes multiple remote sensing images, and the data of at least one remote sensing image among the multiple remote sensing images has missing values. The mask in the missing information mask sequence is used to indicate the position of the missing value in the remote sensing image sequence to be repaired, and the missing information mask sequence is usually composed of 0 and 1, wherein 0 indicates that the data at the corresponding position has a missing value, and 1 indicates that the data at the corresponding position is valid and has no missing values. For example, the remote sensing image sequence to be repaired is a sequence of 10 remote sensing images, each of which is 100×100 pixels in size, and there are some missing values ​​in at least one remote sensing image. The missing information mask sequence can be represented as a 10×100×100 multidimensional array. The mask values ​​in the missing information mask sequence correspond one-to-one to the pixels in the remote sensing image sequence to be repaired, that is, each mask value corresponds to a unique pixel position in a unique image in the remote sensing image sequence to be repaired, and the mask value indicates whether there is a missing value at the corresponding pixel position. The mask value of a certain position in the missing information mask sequence is 0, indicating that there is a missing value at a corresponding pixel position in a corresponding remote sensing image in the remote sensing image sequence to be repaired.

[0045] Through the missing information mask sequence, the target repair network can repair the missing positions in the remote sensing image in a targeted manner to ensure the accuracy of the repair.

[0046] Step S102, performing restoration processing at different scales in turn through S restoration modules to obtain S intermediate image sequences, wherein the input data of the i-th restoration module among the S restoration modules includes the missing information mask sequence and the intermediate image sequence output by the i-1-th restoration module, the restoration scale of the i-th restoration module is smaller than the restoration scale of the i-1-th restoration module, i is greater than or equal to 2, and the input data of the first restoration module is the remote sensing image sequence to be restored and the missing information mask sequence.

[0047] It should be noted that S is an integer greater than 1, i is an integer greater than or equal to 2, and i is less than or equal to S.

[0048] In this embodiment, after the remote sensing image sequence to be repaired and the missing information mask sequence are input into the target repair network, the repair work is implemented through the target repair network. Specifically, the target repair network includes S repair modules, the S repair modules are residual connections, and the S repair modules perform repair processing at different scales respectively. Each repair module outputs an intermediate image sequence to obtain S intermediate image sequences.

[0049] Step S103: repair missing values ​​in the remote sensing image sequence to be repaired according to the Sth intermediate image sequence output by the Sth repair module to obtain a target repaired image sequence.

[0050] Specifically, the structure of the target repair network is as follows: Figure 2 As shown, the remote sensing image sequence to be repaired and the missing information mask sequence are input into the target repair network, and the target repair network includes three repair modules, wherein the input of the first repair module is the remote sensing image sequence to be repaired and the missing information mask sequence, the output of the first repair module is the first intermediate image sequence, the input of the second repair module is the first intermediate image sequence and the missing information mask sequence output by the first repair module, the output of the second repair module is the second intermediate image sequence, the input of the third repair module is the second intermediate image sequence and the missing information mask sequence output by the second repair module, and the output of the third repair module is the third intermediate image sequence.

[0051] By analogy, when i is greater than or equal to 2, the input of the i-th restoration module is the first intermediate image sequence and the missing information mask sequence output by the i-1-th restoration module, the output of the i-th restoration module is the i-th intermediate image sequence, and the output of the i-th restoration module is the S-th intermediate image sequence, resulting in a total of S intermediate image sequences.

[0052] After obtaining the Sth intermediate image sequence, the Sth intermediate image sequence is used to repair the missing values ​​in the remote sensing image sequence to be repaired, that is, the missing values ​​in the remote sensing sequence to be repaired are replaced or filled with the data values ​​of the corresponding positions in the Sth intermediate image sequence to obtain the target repaired image sequence. Compared with the remote sensing image sequence to be repaired, the target repaired image sequence only modifies the data at the missing value positions, and the data at other positions remain unchanged.

[0053] Through the above embodiment, the remote sensing image sequence to be repaired and the missing information mask sequence are used as the input of the target repair network. When repairing, the target repair network can determine the position in the remote sensing image that needs to be repaired, and can repair the missing position in the remote sensing image in a targeted manner to ensure the accuracy of the repair of the remote sensing image sequence. The S repair modules use a step-by-step repair method to perform repair processing, that is, the output of the previous repair module is used as the input of the next repair module. Since the repair scale of the next repair module is smaller than the repair scale of the previous repair module, the repair result of the previous repair module is repaired with a smaller scale to obtain a more refined repair result. The step-by-step refinement repair strategy is used to progressively repair the missing values ​​of the remote sensing image sequence to be repaired, so that the repair of the missing values ​​is more accurate. Therefore, the problem of low accuracy of remote sensing image repair is solved, and the effect of improving the accuracy of remote sensing image repair is achieved.

[0054] Optionally, the restoration processing is performed in sequence by S restoration modules at different scales to obtain S intermediate image sequences, including: when using the i-th restoration module to perform the restoration processing, the i-1-th intermediate image sequence is divided into blocks to obtain an image sequence block set, wherein the size of each block in the image sequence block set is the restoration scale of the i-th restoration module; the image sequence block set is embedded to obtain an embedded vector sequence; feature extraction processing is performed according to the embedded vector sequence and the missing information mask sequence by a spatiotemporal feature extraction submodule to obtain spatiotemporal correlation features; and the spatiotemporal correlation features are inversely embedded to obtain the i-th intermediate image sequence.

[0055] In this embodiment, since the S restoration modules in the target restoration network all perform restoration processing, the restoration processing process performed by each restoration module on the input image sequence is similar.

[0056] In addition, the restoration processing process of the first restoration module is similar to the restoration processing process of the above-mentioned i-th restoration module, and the remote sensing image sequence to be restored is divided into blocks to obtain a set of image sequence blocks, and the size of each block is the restoration scale of the first restoration module; the image sequence block set is embedded to obtain an embedded vector sequence; the spatiotemporal feature extraction submodule performs feature extraction processing according to the embedded vector sequence and the missing information mask sequence to obtain spatiotemporal correlation features; the spatiotemporal correlation features are inversely embedded to obtain the first intermediate image sequence.

[0057] Taking the repair process of the i-th repair module among S repair modules as an example, the structure of the repair module is as follows: Figure 3 As shown, a restoration module includes an embedding submodule, a spatiotemporal feature extraction submodule, and an inverse processing submodule. The restoration processing process of the i-th restoration module includes: performing block processing on the image sequence (i-1-th intermediate image sequence) input to the restoration module, and performing block processing according to a preset size (corresponding to the restoration scale) so that the size of each block obtained after the block processing is the restoration scale of the restoration module, and obtaining an image sequence block set.

[0058] In the subsequent processing process, the blocks in the image sequence block set are processed as the smallest unit. Specifically, the image sequence block set is embedded to obtain an embedded vector sequence, wherein the embedding process is the process of mapping the high-dimensional discrete features in each block into a low-dimensional dense embedded representation to facilitate the learning and processing of the spatiotemporal feature extraction submodule.

[0059] Specifically, the embedding process of the image sequence block set can be expressed by the following formula:

[0060]

[0061] Among them, α is the embedding vector sequence, is the i-1th intermediate image sequence, P (i) is the block size of the i-th repair module.

[0062] Optionally, the i-1th intermediate image sequence may be connected to the missing information mask sequence and then embedded. Specifically, it may be expressed by the following formula:

[0063]

[0064] Among them, α is the embedding vector sequence, is the i-1th intermediate image sequence, P (i) is the block size of the i-th repair module, M is the missing information mask sequence, Embedding represents embedding processing, and concat represents concatenation operation.

[0065] After obtaining the embedded vector sequence, the embedded vector sequence is input into the spatiotemporal feature extraction submodule to extract the spatiotemporal correlation features, which can be expressed as:

[0066] β=MFE(α,M)

[0067] Among them, β is the spatiotemporal correlation feature, α is the embedding vector sequence, and MFE represents the extraction operation of the spatiotemporal feature extraction submodule.

[0068] Optionally, the i-th intermediate image sequence It can be determined by the following formula:

[0069]

[0070] Among them, β is the spatiotemporal correlation feature, P (i) is the block size of the i-th repair module, and Unembedding represents the inverse embedding process.

[0071] In another embodiment, the intermediate image sequence after the inverse processing can be spliced ​​with the intermediate image sequence input to the restoration module to obtain the i-th intermediate image sequence, that is, the i-th intermediate image sequence can also be determined by the following formula:

[0072]

[0073] Among them, β is the spatiotemporal correlation feature, P (i) is the block size of the i-th repair module, Unembedding represents the inverse embedding process, is the i-1th intermediate image sequence, Xin The remote sensing image to be restored.

[0074] Furthermore, the spatiotemporal feature extraction submodule includes a masked spatiotemporal attention unit, which processes the input embedding vector sequence to extract spatiotemporal correlation features.

[0075] Specifically, the masked spatiotemporal attention unit extracts spatiotemporal correlation features including: first, performing a masked self-attention operation on the embedded vector sequence along the time direction according to the missing information mask sequence to obtain the first correlation feature, that is, for the same block position in space, performing a masked self-attention operation along the time direction, learning the temporal correlation features in the input, and obtaining the output under the masked time attention, that is, the first correlation feature. Then, performing a masked self-attention operation on the first correlation feature along the spatial direction according to the missing information mask sequence to obtain the second correlation feature, that is, for the remote sensing image at each moment, performing a masked self-attention operation along the spatial direction, learning the control correlation features of the remote sensing image at each moment, obtaining the output of the masked spatial attention, and obtaining the second correlation feature. The second correlation feature includes both temporal correlation features and spatial correlation features. The second correlation feature is mapped through a feedforward neural network to obtain the spatiotemporal correlation feature. That is, the spatiotemporal feature extraction submodule performs feature extraction processing according to the embedded vector sequence and the missing information mask sequence, including: performing a masked self-attention operation on the embedded vector sequence along the time direction according to the missing information mask sequence to obtain a first associated feature; performing a masked self-attention operation on the first associated feature along the spatial direction according to the missing information mask sequence to obtain a second associated feature; and processing the second associated feature through a feedforward neural network to obtain the spatiotemporal associated feature.

[0076] In one embodiment, the spatiotemporal feature extraction module may include L layers of masked spatiotemporal attention units for superposition, and the input or output needs to be layer normalized before and after each masked spatiotemporal attention unit performs a masked self-attention operation. The structure of an optional spatiotemporal feature extraction module is as follows: Figure 4 As shown, the spatiotemporal feature extraction module is superimposed with L layers of masked spatiotemporal attention units, where L is an integer greater than or equal to 1. The processing process of the L-layer masked spatiotemporal attention unit is as follows: the embedded vector sequence is position-encoded and input into the first masked spatiotemporal attention unit, and the feature sequence output by the first masked spatiotemporal attention unit is input into the second masked spatiotemporal attention unit, and the feature sequence output by the previous layer of masked spatiotemporal attention units is input into the next layer of masked spatiotemporal attention units. The spatiotemporal correlation feature MFE (E, M) finally extracted by the spatiotemporal feature extraction module is superimposed with the L-layer masked spatiotemporal attention units. Specifically, it can be expressed as:

[0077] MFE(E,M)={MSTA(E,M)}L

[0078] Among them, MSTA represents the output of a single-layer masked spatiotemporal attention unit, M is the missing information mask sequence, and E is the input sequence of a single-layer spatiotemporal attention unit.

[0079] Among them, the single-layer masked spatiotemporal attention unit (MSTA) consists of masked temporal attention (MTA), masked spatial attention (MSA), layer normalization (LN) and feedforward neural network (FFN). The extraction process of the feature vector extracted by the single-layer masked spatiotemporal attention unit is as follows:

[0080] U=MTA(LN(E),M)

[0081] V=MSA(LN(U),M)

[0082] MSTA(E,M)=FFN(LN(V))

[0083] Among them, U is the first associated feature, MTA is the masked self-attention calculation along the time direction, V is the second associated feature, MSA is the masked self-attention calculation along the spatial direction, LN represents the layer normalization operation, FFN represents the processing operation of the feedforward neural network, MSTA(E, M) represents the processing result of a single-layer masked spatiotemporal attention unit, and M is the missing information mask sequence.

[0084] Furthermore, for the input sequence (embedded vector sequence) Masked temporal attention first transforms the dimension of E so that the time dimension and the feature dimension are at the end, and obtains d with a length of T emb Dimensional sequence t Next, for the same block position in space, the masked self-attention operation is performed along the time direction, as shown in the following formula:

[0085]

[0086] MTA(E,M)=Reshape(MaskedAttn(e t , M))

[0087] Among them, Reshape represents the dimension change operation, MTA(E, M) is the masked self-attention calculation along the time direction, and MaskedAttn(e t , M) is the applied mask operation, M is the missing information mask sequence, and N is the number of blocks for each image.

[0088] Similarly, after the input sequence is transformed into the corresponding dimension, for each remote sensing image, the masked self-attention operation is performed along the spatial direction to obtain the output of the masked spatial attention (MSA), as shown in the following formula:

[0089]

[0090] MSA(E,M)=Reshape(MaskedAttn(e s , M))

[0091] Among them, Reshape represents the dimension change operation, MSA(E, M) is the masked self-attention calculation along the spatial direction, and MaskedAttn(e s , M) is the application mask operation, and M is the missing information mask sequence.

[0092] Specifically, the masked self-attention operation is performed on the embedded vector sequence along the time direction according to the missing information mask block to obtain the first association feature, including: mapping the embedded vector sequence to obtain a query vector, a key vector, and a value vector; determining the first attention score matrix according to the query vector and the key vector; according to the missing information mask block, setting the attention scores corresponding to the positions in the first attention score matrix where the missing rate is greater than or equal to the first threshold to negative infinity, and setting the attention scores corresponding to the diagonals in the first attention score matrix to negative infinity according to the diagonal mask matrix to obtain a second attention score matrix; using the second attention scores to learn the temporal association features in the embedded vector sequence to obtain the first association features.

[0093] According to the missing information mask sequence, a masked self-attention operation is performed on the first associated feature along the spatial direction to obtain a second associated feature, including: mapping the first associated feature to obtain a query vector, a key vector, and a value vector, and determining a third attention score matrix according to the query vector and the key vector; according to the missing information mask block, the attention scores corresponding to the positions of the third attention score matrix where the missing rate is greater than or equal to the first threshold are set to negative infinity, and the attention scores corresponding to the diagonals in the first attention score matrix are set to negative infinity according to the diagonal mask matrix to obtain a fourth attention score matrix; and the fourth attention scores are used to learn spatial associated features to obtain the second associated features.

[0094] Specifically, the input sequence X (embedded vector sequence or first associated feature) is mapped to the query vector Q, key vector K and value vector V. The scaled dot product can be used to efficiently calculate the attention score between Q and K. Then the Softmax function is applied to obtain the attention weight matrix A, which is expressed as:

[0095] [Q, K, V] = X·W QKV

[0096]

[0097] A=Softmax(Score)

[0098] Among them, Score represents the first attention score matrix or the third attention score matrix, W QKV It is the projection matrix used to calculate the Q, K, V vectors, d k is the dimension of vector K, and the superscript T indicates the matrix transpose operation.

[0099] In order to solve the problem of large regional differences in the distribution of missing values ​​in time series remote sensing data and improve the feature mining capability, we added missing value masks and diagonal masks to the self-attention operation. That is, some attention scores in the first attention score matrix or the third attention score matrix are masked, and which attention scores are masked is determined according to the missing value mask sequence and the diagonal mask matrix. The missing rate of each block is calculated according to the missing value mask sequence. For blocks with a missing rate greater than or equal to the first threshold, the attention score of the block to other blocks is set to negative infinity, and the attention score corresponding to the diagonal is set to negative infinity according to the diagonal mask matrix, which can be expressed as:

[0100]

[0101] Mask(i, j) = 1 if and only if (MR(E(i)) > C max or i=j)

[0102] Among them, Score (i，j) is the attention score corresponding to the (i, j) position in the attention score matrix, indicating the attention score of the i-th block to the j-th block, C max is the first threshold, Mask(i, j) represents the mask value corresponding to the position (i, j). Score′ is the attention score matrix (the second attention score matrix or the fourth attention score matrix) to which the attention mask is applied, where MR(E(i)) represents the missing rate of the i-th block.

[0103] Through the above embodiments, the missing value mask sets the attention score of the block with too high data missing rate to other blocks to -∞, shielding the impact of the missing value, so that the model focuses on extracting effective information from the non-missing data to repair the missing value without being affected by the missing data; the diagonal mask sets the diagonal of the attention matrix to -∞, so that reasoning can only be performed with the help of data in the past time, avoiding over-reliance on future data, so that it can capture the correlation of spatiotemporal features in high-dimensional space.

[0104] After applying the attention mask operation to the attention scores, the attention score matrix is ​​converted into an attention weight matrix using the Softmax function, and the expression of a single masked attention head is obtained, which can be expressed as:

[0105] A′=Softmax(Score′)

[0106] H (i) =A′V

[0107] Where A′ represents the attention weight matrix after applying the mask, Score′ is the attention score matrix after applying the attention mask, and H (i) Represents a single masked attention head.

[0108] Use a linear projection head to fuse h attention heads and add a residual connection to get the output of the masked self-attention operation, as shown in the following formula:

[0109] MaskedAttn(X,M)=Proj([H (1) , H (2) , …, H (h) ])+X

[0110] Among them, H (i) represents a single masked attention head, i = 1, 2, ..., h, MaskedAttn(·) represents a masked self-attention operation, Proj represents a linear projection head, h represents the number of attention heads, and X is the input sequence of the masked self-attention operation.

[0111] In an optional embodiment, before using the target repair network to perform repair work, it is necessary to train the network model to obtain the target repair network. The process of training the repair network is as follows: Figure 5As shown, first of all, it is necessary to determine the training data: obtain a training remote sensing image sequence without missing values, that is, the images in the training remote sensing image sequence are complete remote sensing images, and perform simulated degradation processing on the images in the training remote sensing image sequence to obtain a degraded remote sensing image sequence and a degraded missing value mask sequence. The simulated degradation processing is to remove part of the data in the complete remote sensing image to obtain a degraded remote sensing image sequence with missing values. The repair network in training can repair the degraded remote sensing image sequence, and compare it with the training remote sensing image sequence to determine the accuracy of the repair effect of the repair network. In the process of simulating degradation, a degraded missing value mask sequence can be obtained, and the positions of missing values ​​in the remote sensing image sequence are marked. In each training process, S training repair modules in the training network are used to perform repair processing at different scales to obtain S training intermediate image sequences; loss values ​​are calculated according to the S training intermediate image sequences and the training remote sensing image sequence, respectively, and a target loss value is determined according to multiple loss values. The parameters of the training network are optimized according to the target loss value, including optimizing the parameters of the training repair module, and then the next training is carried out until the target loss value meets the preset conditions and the training is stopped to obtain the target repair network. At this time, the S training repair modules are the S repair modules in the target repair network. That is, before inputting the remote sensing image sequence to be repaired and the missing information mask sequence into the target repair network, the method also includes: obtaining a training remote sensing image sequence, wherein the images in the training remote sensing image sequence are complete remote sensing images; performing simulated degradation processing on the images in the training remote sensing image sequence to obtain a degraded remote sensing image sequence and a degraded missing value mask sequence; and iteratively training the initial repair network using the degraded remote sensing image sequence and the degraded missing value mask sequence to obtain the target repair network.

[0112] Furthermore, iterative training of the initial repair network may include multiple trainings, wherein each training process is similar, and the mth training includes the following steps: inputting the degraded remote sensing image sequence and the degraded missing value mask sequence into the mth training network, wherein m is a positive integer, and when m=1, the mth training network is the initial repair network; performing repair processing at different scales through the S training repair modules in the mth training network to obtain S training intermediate image sequences; calculating loss values ​​based on the S training intermediate image sequences and the training remote sensing image sequence to obtain a target loss value; when the target loss value does not meet a preset condition, modifying the network parameters in the mth training network to obtain the m+1th training network, and using the m+1th training network for the next training; when the target loss value meets a preset condition, determining the mth training network as the target repair network.

[0113] Among them, the preset condition may be that the target loss value is less than or equal to a second threshold, and the second threshold is a preset threshold.

[0114] Optionally, calculating the loss value according to the S training intermediate image sequences and the training remote sensing image sequence to obtain a target loss value includes:

[0115] The target loss value is determined by the following formula

[0116]

[0117] in, is the loss value between n and y, n is the i-th training intermediate image sequence in the S training intermediate image sequences, and y is the training remote sensing image sequence;

[0118] Determined by the following formula

[0119]

[0120] in, is the pixel-level loss value between η and y, is the structural loss value between η and y, is the perceptual loss between η and y, λ s represents the weight of the structural loss value, λ p A weight representing the perceptual loss value.

[0121] In this embodiment, "pixel-structure-perception" is used to calculate the target loss value, wherein the pixel-level loss value regards the generated intermediate remote sensing image sequence as a collection of a series of pixels, compares the generated remote sensing image sequence with the training remote sensing image sequence pixel by pixel, and the pixel-level loss value is the basic loss of the restoration task; the structural loss is based on the principle of structural similarity, and the structural similarity index (SSIM) is used to measure the difference between the generated remote sensing image sequence and the training remote sensing image sequence; the perceptual loss uses the VGG16 network pre-trained based on the ImageNet dataset as the feature extraction network, extracts the features of the remote sensing image sequence and the training remote sensing image sequence, and obtains the corresponding perceptual feature vectors respectively, and then determines the perceptual loss value according to the perceptual feature vectors corresponding to the two image sequences.

[0122] Furthermore, the pixel-level loss value between η and y is Calculated by the following formula:

[0123]

[0124] Wherein, T represents the time dimension, C represents the image width, H represents the image height, W represents the number of channels, η is the i-th training intermediate image sequence in the S training intermediate image sequences, and y is the training remote sensing image sequence.

[0125] Furthermore, the structural loss value between η and y Calculated by the following formula:

[0126]

[0127] Wherein, n is the i-th training intermediate image sequence in the S training intermediate image sequences, y is the training remote sensing image sequence, μ η With μ y denote the means of η and y respectively, and Denote the variance of η and y, σ ηy represents the covariance of η and y, and C1 and C2 are constants.

[0128] Furthermore, the perceptual loss value between η and y Calculated by the following formula:

[0129]

[0130] Among them, ψ(·) represents the feature extraction network, d f Represents the feature dimension.

[0131] like Figure 6 As shown, an embodiment of the present invention provides a remote sensing image sequence repair device, comprising:

[0132] An input unit, used for inputting a remote sensing image sequence to be repaired and a missing information mask sequence into a target repair network, wherein the missing information mask sequence is used to mark the position of missing values ​​in the remote sensing image sequence to be repaired, and the target repair network includes S repair modules, wherein S is an integer greater than 1;

[0133] A first restoration unit is configured to perform restoration processing at different scales through S restoration modules in sequence to obtain S intermediate image sequences, wherein the input data of the i-th restoration module among the S restoration modules includes the missing information mask sequence and the intermediate image sequence output by the i-1-th restoration module, the restoration scale of the i-th restoration module is smaller than the restoration scale of the i-1-th restoration module, i is greater than or equal to 2, and the input data of the first restoration module is the remote sensing image sequence to be restored and the missing information mask sequence;

[0134] The second restoration unit is used to restore missing values ​​in the remote sensing image sequence to be restored according to the Sth intermediate image sequence output by the Sth restoration module to obtain a target restored image sequence.

[0135] In an optional embodiment, the above-mentioned first restoration unit is also used to implement restoration processing at different scales in sequence through S restoration modules in the following manner to obtain S intermediate image sequences: when using the i-th restoration module to perform restoration processing, the i-1-th intermediate image sequence is divided into blocks to obtain an image sequence block set, wherein the size of each block in the image sequence block set is the restoration scale of the i-th restoration module; the image sequence block set is embedded to obtain an embedded vector sequence; feature extraction processing is performed according to the embedded vector sequence and the missing information mask sequence through the spatiotemporal feature extraction submodule to obtain spatiotemporal correlation features; the spatiotemporal correlation features are inversely embedded to obtain the i-th intermediate image sequence.

[0136] In an optional embodiment, the above-mentioned first repair unit is also used to implement feature extraction processing according to the embedded vector sequence and the missing information mask sequence through the spatiotemporal feature extraction submodule in the following manner: performing a masked self-attention operation on the embedded vector sequence along the time direction according to the missing information mask sequence to obtain a first associated feature; performing a masked self-attention operation on the first associated feature along the spatial direction according to the missing information mask sequence to obtain a second associated feature; processing the second associated feature through a feedforward neural network to obtain the spatiotemporal associated feature.

[0137] In an optional embodiment, the first repair unit is further used to perform a masked self-attention operation on the embedded vector sequence along the time direction according to the missing information mask block to obtain a first association feature in the following manner: mapping the embedded vector sequence to obtain a query vector, a key vector, and a value vector; determining the first attention score matrix according to the query vector and the key vector; setting the attention scores corresponding to the positions in the first attention score matrix where the missing rate is greater than or equal to the first threshold to negative infinity according to the missing information mask block, and setting the attention scores corresponding to the diagonals in the first attention score matrix to negative infinity according to the diagonal mask matrix to obtain a second attention score matrix; using the second attention score matrix to learn the temporal association features in the embedded vector sequence to obtain the first association features.

[0138] In an optional embodiment, the above-mentioned device is also used to obtain a training remote sensing image sequence before inputting the remote sensing image sequence to be repaired and the missing information mask sequence into the target repair network, wherein the images included in the training remote sensing image sequence are complete remote sensing images; simulate degradation processing is performed on the images in the training remote sensing image sequence to obtain a degraded remote sensing image sequence and a degraded missing value mask sequence; and the initial repair network is iteratively trained using the degraded remote sensing image sequence and the degraded missing value mask sequence to obtain the target repair network.

[0139] In an optional embodiment, the above-mentioned device is also used to implement iterative training of the initial repair network using the degraded remote sensing image sequence and the degraded missing value mask sequence in the following manner: when the initial repair network is trained for the mth time, the degraded remote sensing image sequence and the degraded missing value mask sequence are input into the mth training network, wherein m is a positive integer, and when m=1, the mth training network is the initial repair network; the S repair modules included in the mth training network are used to perform repair processing at different scales to obtain S training intermediate image sequences; loss values ​​are calculated based on the S training intermediate image sequences and the training remote sensing image sequence to obtain a target loss value; when the target loss value does not meet the preset conditions, the network parameters in the mth training network are modified to obtain the m+1th training network, and the m+1th training network is used for the next training; when the target loss value meets the preset conditions, the mth training network is determined as the target repair network.

[0140] In an optional embodiment, the above-mentioned device is further used to calculate the loss value according to the S training intermediate image sequences and the training remote sensing image sequence in the following manner to obtain the target loss value:

[0141] The target loss value is determined by the following formula

[0142]

[0143] in, is the loss value between n and y, n is the i-th training intermediate image sequence in the S training intermediate image sequences, and y is the training remote sensing image sequence;

[0144] Determined by the following formula

[0145]

[0146] in, is the pixel-level loss value between η and y, is the structural loss value between η and y, is the perceptual loss between η and y, λ s represents the weight of the structural loss value, λ p A weight representing the perceptual loss value.

[0147] The remote sensing image sequence repairing device of this embodiment is used to implement the remote sensing image sequence repairing method described above. Its advantages over the prior art are the same as the advantages of the remote sensing image sequence repairing method described above over the prior art, and will not be repeated here.

[0148] like Figure 7 As shown, an electronic device provided by an embodiment of the present invention includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement the remote sensing image sequence repair method as described above when executing the computer program.

[0149] The advantages of the electronic device of this embodiment compared with the prior art are the same as the advantages of the above-mentioned remote sensing image sequence repair method compared with the prior art, which will not be repeated here.

[0150] Another embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is read and executed by a processor, the above-mentioned remote sensing image sequence repair method is implemented.

[0151] An electronic device that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present invention described and / or required herein.

[0152] The electronic device includes a computing unit, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for the operation of the device can also be stored. The computing unit, ROM and RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0153] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc. In the present application, the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present invention. In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0154] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A method for repairing a remote sensing image sequence, characterized in that: include: Inputting the remote sensing image sequence to be repaired and the missing information mask sequence into the target repair network, wherein the missing information mask sequence is used to mark the position of the missing value in the remote sensing image sequence to be repaired, and the target repair network includes S repair modules, wherein S is an integer greater than 1; Performing restoration processing at different scales in sequence through S restoration modules, respectively, to obtain S intermediate image sequences, wherein the input data of the i-th restoration module among the S restoration modules includes the missing information mask sequence and the intermediate image sequence output by the i-1-th restoration module, the restoration scale of the i-th restoration module is smaller than the restoration scale of the i-1-th restoration module, i is greater than or equal to 2, and the input data of the first restoration module is the remote sensing image sequence to be restored and the missing information mask sequence; Replacing or filling missing values ​​in the remote sensing image sequence to be restored with data values ​​at corresponding positions in the Sth intermediate image sequence among the S intermediate image sequences, to obtain a target restored image sequence; The method of performing restoration processing at different scales by the S restoration modules in sequence to obtain S intermediate image sequences comprises: when the i-th restoration module is used to perform restoration processing, the i-1-th intermediate image sequence is subjected to block processing to obtain an image sequence block set, wherein the size of each block in the image sequence block set is the restoration scale of the i-th restoration module; Embedding the image sequence block set to obtain an embedding vector sequence; performing feature extraction processing according to the embedding vector sequence and the missing information mask sequence through a spatiotemporal feature extraction submodule to obtain spatiotemporal correlation features; performing inverse embedding processing on the spatiotemporal correlation features to obtain the i-th intermediate image sequence; The feature extraction processing is performed according to the embedding vector sequence and the missing information mask sequence by the spatiotemporal feature extraction submodule, including: performing a masked self-attention operation on the embedding vector sequence along the time direction according to the missing information mask sequence to obtain a first associated feature; performing a masked self-attention operation on the first associated feature along the space direction according to the missing information mask sequence to obtain a second associated feature; processing the second associated feature through a feedforward neural network to obtain the spatiotemporal associated feature; Among them, the performing a masked self-attention operation on the embedded vector sequence along the time direction according to the missing information mask sequence to obtain a first association feature includes: performing a mapping operation on the embedded vector sequence to obtain a query vector, a key vector, and a value vector; determining a first attention score matrix according to the query vector and the key vector; according to the missing information mask sequence, setting the attention scores corresponding to the positions in the first attention score matrix where the missing rate is greater than or equal to the first threshold to negative infinity, and setting the attention scores corresponding to the diagonals in the first attention score matrix to negative infinity according to the diagonal mask matrix to obtain a second attention score matrix; using the second attention score matrix to learn the temporal association features in the embedded vector sequence to obtain the first association feature; The first attention score matrix is ​​determined by the following formula: Where Score represents the first attention score matrix, Q is the query vector, K is the key vector, and d k is the dimension of the key vector, the superscript Represents a matrix transpose operation.

2. The method for repairing a remote sensing image sequence according to claim 1, characterized in that: Before inputting the remote sensing image sequence to be repaired and the missing information mask sequence into the target repair network, the method further includes: Acquire a training remote sensing image sequence, wherein the images included in the training remote sensing image sequence are complete remote sensing images; Performing simulated degradation processing on the images in the training remote sensing image sequence to obtain a degraded remote sensing image sequence and a degraded missing value mask sequence; The initial restoration network is iteratively trained using the degraded remote sensing image sequence and the degraded missing value mask sequence to obtain the target restoration network.

3. The method for repairing a remote sensing image sequence according to claim 2, characterized in that: The iterative training of the initial restoration network using the degraded remote sensing image sequence and the degraded missing value mask sequence comprises: When the initial restoration network is trained for the mth time, the degraded remote sensing image sequence and the degraded missing value mask sequence are input into the mth training network, wherein m is a positive integer, and when m=1, the mth training network is the initial restoration network; Performing restoration processing at different scales respectively through the S training restoration modules included in the m-th training network to obtain S training intermediate image sequences; Calculate the loss value according to the S training intermediate image sequences and the training remote sensing image sequence to obtain a target loss value; When the target loss value does not meet the preset conditions, modify the network parameters in the mth training network to obtain the m+1th training network, and use the m+1th training network for the next training; When the target loss value meets a preset condition, the mth training network is determined as the target restoration network.

4. The method for repairing a remote sensing image sequence according to claim 3, characterized in that: The step of calculating the loss value based on the S training intermediate image sequences and the training remote sensing image sequence to obtain the target loss value includes: The target loss value is determined by the following formula in, is the loss value between n and y, n is the i-th training intermediate image sequence in the S training intermediate image sequences, and y is the training remote sensing image sequence; Determined by the following formula in, is the pixel-level loss value between η and y, is the structural loss value between η and y, is the perceptual loss between η and y, λ s represents the weight of the structural loss value, λ p A weight representing the perceptual loss value.

5. A remote sensing image sequence repair device, characterized in that: include: An input unit, used for inputting a remote sensing image sequence to be repaired and a missing information mask sequence into a target repair network, wherein the missing information mask sequence is used to mark the position of missing values ​​in the remote sensing image sequence to be repaired, and the target repair network includes S repair modules, wherein S is an integer greater than 1; A first restoration unit is configured to perform restoration processing at different scales through S restoration modules in sequence to obtain S intermediate image sequences, wherein the input data of the i-th restoration module among the S restoration modules includes the missing information mask sequence and the intermediate image sequence output by the i-1-th restoration module, the restoration scale of the i-th restoration module is smaller than the restoration scale of the i-1-th restoration module, i is greater than or equal to 2, and the input data of the first restoration module is the remote sensing image sequence to be restored and the missing information mask sequence; A second restoration unit is used to replace or fill missing values ​​in the remote sensing image sequence to be restored with data values ​​of corresponding positions in the Sth intermediate image sequence among the S intermediate image sequences, so as to obtain a target restored image sequence; The first restoration unit is further used to implement restoration processing at different scales through S restoration modules in sequence in the following manner to obtain S intermediate image sequences, including: when using the i-th restoration module to perform restoration processing, performing block processing on the i-1-th intermediate image sequence to obtain an image sequence block set, wherein the size of each block in the image sequence block set is the restoration scale of the i-th restoration module; performing embedding processing on the image sequence block set to obtain an embedding vector sequence; performing feature extraction processing on the embedding vector sequence and the missing information mask sequence through the spatiotemporal feature extraction submodule to obtain spatiotemporal correlation features; performing inverse embedding processing on the spatiotemporal correlation features to obtain the i-th intermediate image sequence; The first repair unit is further used to implement feature extraction processing according to the embedding vector sequence and the missing information mask sequence through the spatiotemporal feature extraction submodule in the following manner, including: performing a masked self-attention operation on the embedding vector sequence along the time direction according to the missing information mask sequence to obtain a first associated feature; performing a masked self-attention operation on the first associated feature along the space direction according to the missing information mask sequence to obtain a second associated feature; processing the second associated feature through a feedforward neural network to obtain the spatiotemporal associated feature; The first repair unit is also used to perform a masked self-attention operation on the embedded vector sequence along the time direction according to the missing information mask sequence in the following manner to obtain a first association feature, including: performing a mapping operation on the embedded vector sequence to obtain a query vector, a key vector, and a value vector; determining a first attention score matrix according to the query vector and the key vector; setting the attention scores corresponding to the positions in the first attention score matrix where the missing rate is greater than or equal to the first threshold to negative infinity according to the missing information mask sequence, and setting the attention scores corresponding to the diagonals in the first attention score matrix to negative infinity according to the diagonal mask matrix to obtain a second attention score matrix; using the second attention score matrix to learn the temporal association features in the embedded vector sequence to obtain the first association feature; The first attention score matrix is ​​determined by the following formula: Where Score represents the first attention score matrix, Q is the query vector, K is the key vector, and d k is the dimension of the key vector, the superscript Represents a matrix transpose operation.

6. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the remote sensing image sequence repair method as described in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is read and executed by a processor, the method for repairing a remote sensing image sequence according to any one of claims 1 to 4 is implemented.

Images