Image prediction method
The visual Transformer model through sliding feature learning encodes and processes the images, solving the problems of large amount of calculation and low efficiency in image processing, and improving image segmentation accuracy and efficiency.
Patent Information
- Application Number
- CN202410974564.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-22
Smart Images

Figure CN119006308A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of image processing, and in particular relates to an image prediction method. Background Art
[0002] Image segmentation and image processing are computer technologies related to computer vision and image processing. Deep learning algorithms have been widely used in the entire field of computer vision, including the field of image processing, for example, medical image segmentation. The strategy of traditional neural networks for image segmentation is to input each pixel and its neighborhood into a convolutional neural network for training and prediction. This method is simple and direct, but it cannot have a high receptive field, resulting in poor results and large computational complexity. Subsequently, it was proposed to use a fully convolutional network (FCN) for image processing. The innovation of the fully convolutional network is that the sample image size is no longer restricted, the applicability is wider, the redundant structure is reduced, and the operation efficiency is more efficient. However, the disadvantage of this method is that the image detail information will be lost, and the segmentation accuracy needs to be further improved.
[0003] At the same time, Transformer is a deep learning model architecture for natural language processing and sequence-to-sequence tasks. It introduces a self-attention mechanism that can simultaneously consider the contextual relationship of each element in the input sequence. It has been applied to the field of image processing and can focus on image detail information, thereby improving image processing accuracy. However, when the existing Transformer model is applied to the field of image processing, the focus on image detail information is often repeated, resulting in large computational complexity and low processing efficiency. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide an image prediction method, which segments and processes images based on a visual Transformer model learned through sliding features, thereby reducing the amount of calculation in the image processing process, thereby improving the efficiency of image processing and enhancing the segmentation model's ability to learn contextual feature relationships.
[0005] In order to achieve the above technical purpose, the technical solution of the present invention is:
[0006] An image prediction method comprises the following steps:
[0007] Dividing the image to be identified into sub-data of preset feature dimensions, and adding position codes to the sub-data to obtain input data;
[0008] Encoding the input data by a preset network encoder to obtain encoded data, wherein the preset encoder includes at least one sliding module, the sliding module includes two sliding feature learning units and a region merging unit, the sliding feature learning unit is used to extract feature data of the input data, and the region merging unit is used to downsample the feature data to obtain the encoded data;
[0009] Inputting the encoded data into a preset network decoder through a preset bottleneck layer to obtain decoded data, wherein the preset network decoder is a decoder constructed based on a deep convolutional network, and the preset network encoder and the preset network decoder are jump-connected;
[0010] A predicted image corresponding to the image to be identified is obtained through a prediction unit according to the decoded data.
[0011] Optionally, the step of dividing the image to be identified into sub-data of preset feature dimensions includes:
[0012] The image to be identified is evenly divided into non-overlapping regional images with preset feature dimensions through a regional division algorithm, and the regional images are used as the sub-data.
[0013] Optionally, the step of adding a position code to the sub-data to obtain input data includes:
[0014] The sub-data is projected into the embedding space through a linear layer, and a relative position encoding is added to the sub-data to obtain input data.
[0015] Optionally, the sliding feature learning unit includes a first Transformer block and a second Transformer block, and the step of encoding the input data by a preset network encoder to obtain the encoded data includes:
[0016] Inputting the input data into the first Transformer block to obtain initial feature data of the input data;
[0017] Inputting the initial feature data into the second Transformer block to obtain feature data of the input data;
[0018] The feature data is downsampled by a region merging unit of the preset network encoder to obtain the encoded data.
[0019] Optionally, the first Transformer block includes a first subnetwork consisting of a first normalization layer and a first multi-head attention network, and a second subnetwork consisting of a second normalization layer and a first perceptron, the second Transformer block includes a third subnetwork consisting of a third normalization layer and a second multi-head attention network, and a fourth subnetwork consisting of a fourth normalization layer and a second perceptron, the first subnetwork is residually connected to the second subnetwork, the second subnetwork is residually connected to the third subnetwork, and the third subnetwork is residually connected to the fourth subnetwork, the first multi-head attention network is a window-based multi-head attention network, and the second multi-head attention network is a window sliding-based multi-head attention network.
[0020] Optionally, the mathematical description of the first Transformer block is as follows:
[0021]
[0022] The mathematical description of the second Transformer block is as follows:
[0023]
[0024] in, and x m They represent the outputs of the first multi-head attention network and the first perceptron in the m-th sliding module, respectively. and x m+1 They respectively represent the outputs of the second multi-head attention network and the second perceptron in the m+1th sliding module.
[0025] Optionally, the mathematical description of the sliding feature learning unit is as follows:
[0026]
[0027] Wherein, Q, K, and V represent the query, key, and value in the sliding feature learning unit, respectively. M 2 Represents the number of regional images in the window, d represents the dimension of Q or K, and the value of B comes from the bias matrix
[0028] Optionally, the preset bottleneck layer includes two sliding feature learning units connected in series, the input end of the preset bottleneck layer is connected to the output end of the preset network encoder, and the output end of the preset bottleneck layer is connected to the input end of the preset network decoder.
[0029] Optionally, the preset network decoder includes at least one decoding unit, the decoding unit includes a decoding subnetwork and a deconvolution subnetwork, the decoding subnetwork is composed of a convolution layer, a BN layer, a ReLu function layer and a deconvolution layer, the convolution layer, the BN layer, the ReLu function layer and the deconvolution layer are connected in series in sequence, and the output end of the deconvolution layer is connected to the input end of the deconvolution subnetwork.
[0030] Optionally, the step of obtaining a predicted image corresponding to the image to be identified through a prediction unit according to the decoded data includes:
[0031] Inputting the decoded data into the prediction convolution layer of the prediction unit to obtain convolution data;
[0032] A predicted image corresponding to the image to be recognized is generated according to the convolution data through the softmax activation function of the prediction unit.
[0033] The beneficial effects of the present invention are:
[0034] The present invention encodes the input data through a preset network encoder to obtain encoded data, the preset encoder includes at least one sliding module, the sliding module includes two sliding feature learning units and a region merging unit, the sliding feature learning unit is used to extract feature data of the input data, and the region merging unit is used to downsample the feature data to obtain the encoded data, that is, the image is encoded through the Transformer model of sliding feature learning, which reduces the amount of calculation in the image processing process, thereby improving the efficiency of image processing and the ability to learn contextual relationships. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 is a flow chart of the image prediction method of the present invention;
[0036] Figure 2 A schematic diagram of a model for executing an image prediction method in an embodiment;
[0037] Figure 3 is a schematic diagram of the structure of an image prediction device in an embodiment;
[0038] Figure 4 is a schematic diagram of the structure of an electronic device in an embodiment;
[0039] Figure 5 Schematic diagram of the structure of a computer-readable storage medium in an embodiment. DETAILED DESCRIPTION
[0040] The embodiments of the present invention will be further described below in conjunction with the accompanying drawings.
[0041] like Figure 1 As shown, the present invention proposes an image prediction method, comprising the following steps:
[0042] S101, dividing the image to be identified into sub-data of preset feature dimensions, and adding position codes to the sub-data to obtain input data;
[0043] In a possible implementation manner, the step of dividing the image to be identified into sub-data of preset feature dimensions includes:
[0044] The image to be identified is evenly divided into non-overlapping regional images with preset feature dimensions through a regional division algorithm, and the regional images are used as the sub-data.
[0045] Exemplarily, the patch partition layer evenly divides the image into small, non-overlapping regions (region images) of equal size with a feature dimension of 4×4×3.
[0046] In a possible implementation manner, the step of adding a position code to the sub-data to obtain input data includes:
[0047] The sub-data is projected into the embedding space through a linear layer, and a position code is added to the sub-data to obtain input data.
[0048] Exemplarily, the input data of the preset network encoder is obtained by projecting into the embedding space through a linear embedding layer and adding relative position encoding for the small area (region image).
[0049] S102, encoding the input data through a preset network encoder to obtain encoded data, the preset encoder includes at least one sliding module, the sliding module includes two sliding feature learning units and a region merging unit, the sliding feature learning unit is used to extract feature data of the input data, and the region merging unit is used to downsample the feature data to obtain the encoded data;
[0050] Exemplarily, when the region merging unit works with 2x downsampling, to ensure that feature information is not omitted, three sliding modules are required, that is, six sliding feature learning units and three region merging units, to repeat the 2x downsampling three times. Specifically, after the first sliding module outputs the first sampling data, the first sampling data is input into the second sliding module. Similarly, the second sampling data output by the second sliding module is input into the third sliding module to obtain the encoded data.
[0051] S103, inputting the encoded data into a preset network decoder through a preset bottleneck layer to obtain decoded data, wherein the preset network decoder is a decoder constructed based on a deep convolutional network, and the preset network encoder and the preset network decoder are jump-connected;
[0052] In a possible implementation, the preset bottleneck layer includes two sliding feature learning units connected in series, the input end of the preset bottleneck layer is connected to the output end of the preset network encoder, and the output end of the preset bottleneck layer is connected to the input end of the preset network decoder.
[0053] Exemplarily, in order to reduce the number of parameters and thus reduce the amount of calculation, and to more effectively and intuitively perform data training and feature extraction after dimensionality reduction, the preset bottleneck layer includes two sliding feature learning units connected in series.
[0054] Exemplarily, the preset network encoder and the preset network decoder can make up for the spatial features lost in the decoding process through jump connections, thereby solving the problem of gradient disappearance in the deep network.
[0055] S104: Obtain a predicted image corresponding to the image to be recognized through a prediction unit according to the decoded data.
[0056] The input data is encoded by a preset network encoder to obtain encoded data, the preset encoder includes at least one sliding module, the sliding module includes two sliding feature learning units and a region merging unit, the sliding feature learning unit is used to extract feature data of the input data, and the region merging unit is used to downsample the feature data to obtain the encoded data, that is, the image is encoded by the Transformer model of sliding feature learning, which reduces the amount of calculation in the image processing process and thus improves the efficiency of image processing.
[0057] In a possible implementation, Figure 2As shown, the sliding feature learning unit includes a first Transformer block and a second Transformer block, the first Transformer block includes a first sub-network consisting of a first normalization layer and a first multi-head attention network, and, a second sub-network consisting of a second normalization layer and a first perceptron, the second Transformer block includes a third sub-network consisting of a third normalization layer and a second multi-head attention network, and, a fourth sub-network consisting of a fourth normalization layer and a second perceptron, the first sub-network is residually connected to the second sub-network, the second sub-network is residually connected to the third sub-network, the third sub-network is residually connected to the fourth sub-network, the first multi-head attention network is a window-based multi-head attention network, the second multi-head attention network is a window-sliding based multi-head attention network, the step of encoding the input data by a preset network encoder to obtain the encoded data includes:
[0058] Inputting the input data into the first Transformer block to obtain initial feature data of the input data;
[0059] Inputting the initial feature data into the second Transformer block to obtain feature data of the input data;
[0060] The feature data is downsampled by a region merging unit of the preset network encoder to obtain the encoded data.
[0061] Exemplarily, a sliding window is an algorithm for processing sequence data (such as time series, text, images, etc.). It divides the data into a series of overlapping subsequences, each of which is called a window. The sliding window algorithm captures important features of the data by sliding a window on the data sequence for further analysis and processing. By adopting a window-based multi-head attention network (first multi-head attention network) and a window sliding-based multi-head attention network (second multi-head attention network), a sliding window is used on the data sequence to capture important features of the data to ensure that the features are not missed.
[0062] In one possible implementation, the mathematical description of the first Transformer block is as follows:
[0063]
[0064] The mathematical description of the second Transformer block is as follows:
[0065]
[0066] in, and x mThey represent the outputs of the first multi-head attention network and the first perceptron in the m-th sliding module, respectively. and x m+1 They respectively represent the outputs of the second multi-head attention network and the second perceptron in the m+1th sliding module.
[0067] In a possible implementation, the mathematical description of the sliding feature learning unit is as follows:
[0068]
[0069] Wherein, Q, K, and V represent the query, key, and value in the sliding feature learning unit, respectively. M 2 Represents the number of regional images in the window, d represents the dimension of Q or K, and the value of B comes from the bias matrix
[0070] In a possible implementation, Figure 2 As shown, the preset network decoder includes at least one decoding unit, the decoding unit includes a decoding subnetwork and a deconvolution subnetwork, the decoding subnetwork is composed of a convolution layer, a BN layer, a ReLu function layer and a deconvolution layer, the convolution layer, the BN layer, the ReLu function layer and the deconvolution layer are connected in series in sequence, and the output end of the deconvolution layer is connected to the input end of the deconvolution subnetwork.
[0071] For example, although Transformer has a strong ability to learn global information, its ability to capture local information cannot surpass the convolutional network. Therefore, the decoder is designed based on CNN, and the output is upsampled using a deconvolution layer. The upsampling process is repeated cyclically until the same resolution as the image to be processed is reached. Specifically, multiple decoding units are required to set up for the cyclically repeated upsampling process. When the upsampling process is repeated n times, the number of decoding units is greater than or equal to n.
[0072] Specifically, the convolution layer is a 3X3 convolution layer, and the deconvolution layer is a 2X2 deconvolution layer.
[0073] In a possible implementation, Figure 2 As shown, the step of obtaining a predicted image corresponding to the image to be identified through a prediction unit according to the decoded data includes:
[0074] Inputting the decoded data into the prediction convolution layer of the prediction unit to obtain convolution data;
[0075] A predicted image corresponding to the image to be recognized is generated according to the convolution data through the softmax activation function of the prediction unit.
[0076] Exemplarily, the prediction convolution layer is a 1X1X1 convolution layer.
[0077] like Figure 3 As shown, an embodiment of the present application provides a profiling device based on evaluation data, the device comprising:
[0078] A preprocessing module 201 is used to divide the image to be identified into sub-data of preset feature dimensions and add position codes to the sub-data to obtain input data;
[0079] The encoding module 202 is used to encode the input data through a preset network encoder to obtain encoded data, wherein the preset encoder includes at least one sliding module, and the sliding module includes two sliding feature learning units and a region merging unit. The sliding feature learning unit is used to extract feature data of the input data, and the region merging unit is used to downsample the feature data to obtain the encoded data;
[0080] A decoding module 203, used for inputting the encoded data into a preset network decoder through a preset bottleneck layer to obtain decoded data, wherein the preset network decoder is a decoder constructed based on a deep convolutional network, and the preset network encoder and the preset network decoder are jump-connected;
[0081] The prediction module 204 is configured to obtain a predicted image corresponding to the image to be identified through a prediction unit according to the decoded data.
[0082] In one embodiment, Figure 4 As shown, an electronic device is proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: dividing the image to be identified into sub-data of preset feature dimensions, and adding position codes to the sub-data to obtain input data; encoding the input data through a preset network encoder to obtain encoded data, wherein the preset encoder includes at least one sliding module, wherein the sliding module includes two sliding feature learning units and a region merging unit, wherein the sliding feature learning unit is used to extract feature data of the input data, and the region merging unit is used to downsample the feature data to obtain the encoded data; inputting the encoded data into a preset network decoder through a preset bottleneck layer to obtain decoded data, wherein the preset network decoder is a decoder constructed based on a deep convolutional network, and the preset network encoder and the preset network decoder are jump-connected; and obtaining a predicted image corresponding to the image to be identified through a prediction unit according to the decoded data.
[0083] In one embodiment, Figure 5As shown, a computer-readable storage medium is proposed, and the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the following steps are implemented: dividing the image to be identified into sub-data of preset feature dimensions, and adding position codes to the sub-data to obtain input data; encoding the input data through a preset network encoder to obtain encoded data, and the preset encoder includes at least one sliding module, and the sliding module includes two sliding feature learning units and a region merging unit, and the sliding feature learning unit is used to extract feature data of the input data, and the region merging unit is used to downsample the feature data to obtain the encoded data; inputting the encoded data into a preset network decoder through a preset bottleneck layer to obtain decoded data, and the preset network decoder is a decoder constructed based on a deep convolutional network, and the preset network encoder and the preset network decoder are jump-connected; according to the decoded data, a predicted image corresponding to the image to be identified is obtained through a prediction unit.
Claims
1. An image prediction method, characterized in that: The method comprises the following steps: Dividing the image to be identified into sub-data of preset feature dimensions, and adding position codes to the sub-data to obtain input data; Encoding the input data by a preset network encoder to obtain encoded data, wherein the preset encoder includes at least one sliding module, the sliding module includes two sliding feature learning units and a region merging unit, the sliding feature learning unit is used to extract feature data of the input data, and the region merging unit is used to downsample the feature data to obtain the encoded data; Input the encoded data into a preset network decoder through a preset bottleneck layer to obtain decoded data, wherein the preset network decoder is a decoder constructed based on a deep convolutional network, and the preset network encoder and the preset network decoder are jump-connected; A predicted image corresponding to the image to be identified is obtained through a prediction unit according to the decoded data.
2. The image prediction method according to claim 1, characterized in that: The step of dividing the image to be identified into sub-data of preset feature dimensions includes: The image to be identified is evenly divided into non-overlapping regional images with preset feature dimensions through a regional division algorithm, and the regional images are used as the sub-data.
3. The image prediction method according to claim 1, characterized in that: The step of adding a position code to the sub-data to obtain input data comprises: The sub-data is projected into an embedding space through a linear layer, and a position code is added to the sub-data to obtain the input data.
4. The image prediction method according to claim 1, characterized in that: The sliding feature learning unit includes a first Transformer block and a second Transformer block, and the step of encoding the input data by a preset network encoder to obtain encoded data includes: Inputting the input data into the first Transformer block to obtain initial feature data of the input data; Inputting the initial feature data into the second Transformer block to obtain feature data of the input data; The feature data is downsampled by a region merging unit of the preset network encoder to obtain the encoded data.
5. The image prediction method according to claim 4, characterized in that: The first Transformer block includes a first subnetwork consisting of a first normalization layer and a first multi-head attention network, and a second subnetwork consisting of a second normalization layer and a first perceptron. The second Transformer block includes a third subnetwork consisting of a third normalization layer and a second multi-head attention network, and a fourth subnetwork consisting of a fourth normalization layer and a second perceptron. The first subnetwork is residually connected to the second subnetwork, the second subnetwork is residually connected to the third subnetwork, and the third subnetwork is residually connected to the fourth subnetwork. The first multi-head attention network is a window-based multi-head attention network, and the second multi-head attention network is a window-sliding based multi-head attention network.
6. The image prediction method according to claim 5, characterized in that: The mathematical description of the first Transformer block is as follows: The mathematical description of the second Transformer block is as follows: in, and x m They represent the outputs of the first multi-head attention network and the first perceptron in the m-th sliding module, respectively. and x m+1 They respectively represent the outputs of the second multi-head attention network and the second perceptron in the m+1th sliding module.
7. The image prediction method according to claim 1, characterized in that: The mathematical description of the sliding feature learning unit is as follows: Wherein, Q, K, and V represent the query, key, and value in the sliding feature learning unit, respectively. M 2 Represents the number of regional images in the window, d represents the dimension of Q or K, and the value of B comes from the bias matrix 8. The image prediction method according to claim 1, characterized in that: The preset bottleneck layer includes two sliding feature learning units connected in series, the input end of the preset bottleneck layer is connected to the output end of the preset network encoder, and the output end of the preset bottleneck layer is connected to the input end of the preset network decoder.
9. The image prediction method according to claim 1, characterized in that: The preset network decoder includes at least one decoding unit, and the decoding unit includes a decoding subnetwork and a deconvolution subnetwork. The decoding subnetwork is composed of a convolution layer, a BN layer, a ReLu function layer and a deconvolution layer. The convolution layer, the BN layer, the ReLu function layer and the deconvolution layer are connected in series in sequence, and the output end of the deconvolution layer is connected to the input end of the deconvolution subnetwork.
10. The image prediction method according to claim 1, characterized in that: The step of obtaining a predicted image corresponding to the image to be identified through a prediction unit according to the decoded data comprises: Inputting the decoded data into the prediction convolution layer of the prediction unit to obtain convolution data; A predicted image corresponding to the image to be recognized is generated according to the convolution data through the softmax activation function of the prediction unit.
Citation Information
Patent Citations
Image segmentation model with coding and decoding structure combined with attention mechanism and training method thereof
CN114445420A
Medical image segmentation method and system based on double-branch embedded attention mechanism
CN116309650A
Blind sidewalk area extraction method of sliding window attention network based on codec
CN117612122A
