Image segmentation method, device, program product and medium
By introducing multi-scale convolution operation and linear attention mechanism in the image segmentation model, combined with Gaussian error linear unit activation function, the inefficiency problem of preset deep learning model architecture in receptive field and multi-scale processing is solved, and the Improved performance and efficiency of image segmentation models.
Patent Information
- Application Number
- CN202411132770.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-19
AI Technical Summary
Image segmentation technology based on preset deep learning model architecture has problems of high computational complexity and low efficiency in receptive field and multi-scale processing, and it is difficult to directly apply to high-resolution image processing.
Build a basic image segmentation model based on preset convolutional neural network, use multi-scale convolution operation and linear attention mechanism for feature extraction and fusion, combine Gaussian error linear unit activation function, configure the loss function and Optimizer to form the target image segmentation model.
It significantly reduces the computational complexity, improves the computing efficiency and the model's perception of features of different scales, and achieves the dual improvement of image segmentation model performance and computing efficiency.
Smart Images

Figure CN118674731B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to an image segmentation method, device, program product and medium. Background Art
[0002] Image segmentation is an important technology in the field of computer vision. In recent years, in the field of image processing, applications based on preset deep learning model architectures have achieved remarkable results. Their excellent global modeling capabilities have enabled them to show great potential in image segmentation tasks. Although image segmentation technologies based on preset deep learning model architectures have performed well in many tasks, they still have significant problems in receptive field and multi-scale processing. Not only is it difficult to directly apply to the processing of high-resolution images due to its high computational complexity, which limits the receptive field, but it also does not have a natural advantage in processing multi-scale information, which often requires additional modules or improvements to improve its performance in multi-scale scenarios. Therefore, in practical applications, the implementation of image segmentation technology based on preset deep learning model architectures is not ideal. Summary of the invention
[0003] In view of this, the purpose of the present invention is to provide an image segmentation method, device, program product and medium, which can achieve dual improvement of image segmentation model performance and computational efficiency. The specific scheme is as follows:
[0004] In a first aspect, the present application discloses an image segmentation method, comprising:
[0005] Build a basic image segmentation model based on a preset convolutional neural network;
[0006] In the basic image segmentation model, a multi-scale convolution operation is used to extract features from a data set used for image segmentation to obtain features of different scales;
[0007] A linear attention mechanism is constructed based on a preset deep learning model architecture, and a Gaussian error linear unit activation function is used as a constructor of the linear attention mechanism, and then the features are fused through the linear attention mechanism to obtain a multi-scale fused feature map;
[0008] The multi-scale fusion feature map is upsampled and concatenated, and a corresponding loss function and optimizer are configured for the basic image segmentation model to obtain a target image segmentation model, so as to perform image segmentation using the target image segmentation model.
[0009] Optionally, before constructing the basic image segmentation model based on the preset convolutional neural network, the method further includes:
[0010] Collect a data set for image segmentation, and perform data preprocessing and data enhancement on the image data in the data set; wherein the data preprocessing is used to standardize the image data; and the data enhancement is used to perform any one or a combination of random cropping, rotation, flipping, and color jittering on the image data.
[0011] Optionally, the constructing of a basic image segmentation model based on a preset convolutional neural network includes:
[0012] Initialize a preset convolutional neural network, remove the last fully connected layer in the preset convolutional neural network, and then select a corresponding target version type for the preset convolutional neural network to obtain the basic image segmentation model.
[0013] Optionally, the method of extracting features from a data set used for image segmentation by using a multi-scale convolution operation in the basic image segmentation model to obtain features of different scales includes:
[0014] Determine the sequence data corresponding to each feature vector in the data set used for image segmentation;
[0015] Mapping the sequence data to obtain corresponding query matrix, key matrix and value matrix;
[0016] In the basic image segmentation model, multi-scale convolution operations are used to extract features from the query matrix, the key matrix and the value matrix respectively to obtain features of different scales.
[0017] Optionally, the step of fusing the features through the linear attention mechanism to obtain a multi-scale fused feature map includes:
[0018] The query matrix, the key matrix and the value matrix after feature extraction are calculated using the calculation formula of the linear attention mechanism based on the Gaussian error linear unit activation function, and the output matrices obtained after the calculation are sequentially matrix-connected and linearly transformed to obtain the multi-scale fusion feature map.
[0019] Optionally, the image segmentation method further includes:
[0020] Obtain multi-scale fusion feature maps based on the parallel output of the multi-head attention mechanism, and perform matrix connection and linear transformation on the parallel output multi-scale fusion feature maps in sequence to obtain the target output;
[0021] Accordingly, performing upsampling and splicing operations on the multi-scale fusion feature map, and configuring a corresponding loss function and optimizer for the basic image segmentation model to obtain a target image segmentation model, includes:
[0022] The target output is upsampled and concatenated, and a corresponding loss function and optimizer are configured for the basic image segmentation model to obtain a target image segmentation model.
[0023] Optionally, configuring a corresponding loss function and an optimizer for the basic image segmentation model to obtain a target image segmentation model includes:
[0024] The corresponding loss function and optimizer are configured for the basic image segmentation model, and the gradient clipping technology is used to eliminate the gradient explosion to obtain the target image segmentation model.
[0025] Optionally, the upsampling and splicing operations on the multi-scale fusion feature map include:
[0026] The multi-scale fusion feature map is up-sampled using a transposed convolution operation, and during the up-sampling process, the features of the current layer are concatenated with the features of the corresponding encoding layer.
[0027] Optionally, the process of performing upsampling and splicing operations on the multi-scale fusion feature map and configuring a corresponding loss function and optimizer for the basic image segmentation model to obtain a target image segmentation model further includes:
[0028] Setting a batch size and a number of training rounds during training for the basic image segmentation model;
[0029] The basic image segmentation model is trained based on the batch size and the training rounds, and the current basic image segmentation model is verified using a verification set after each training round, so as to optimize the parameters of the basic image segmentation model according to the verification results.
[0030] Optionally, after performing upsampling and splicing operations on the multi-scale fusion feature map and configuring a corresponding loss function and optimizer for the basic image segmentation model to obtain a target image segmentation model, the method further includes:
[0031] The target image segmentation model is used to perform verification on a test set, and the target image segmentation model is evaluated using preset evaluation indicators.
[0032] Optionally, the using the target image segmentation model to perform verification on a test set and using a preset evaluation index to evaluate the target image segmentation model includes:
[0033] Obtain the segmentation results of the image segmentation of the test set using the target image segmentation model, and perform visual analysis on the segmentation results according to the preset evaluation index to verify the segmentation effect of the target image segmentation model.
[0034] Optionally, after performing upsampling and splicing operations on the multi-scale fusion feature map and configuring a corresponding loss function and optimizer for the basic image segmentation model to obtain a target image segmentation model, so as to perform image segmentation using the target image segmentation model, the method further includes:
[0035] Determine a target application scenario, and export the target image segmentation model into a target deployment format that can be deployed in the target application scenario;
[0036] The target image segmentation model is deployed in the target application scenario according to the target deployment format, so as to perform image segmentation using the target image segmentation model based on the target deployment format.
[0037] In a second aspect, the present application discloses an electronic device, comprising:
[0038] Memory for storing computer programs;
[0039] A processor is used to load and execute the computer program to implement the aforementioned image segmentation method.
[0040] In a third aspect, the present application discloses a computer program product, including a computer program / instruction, which implements the steps of the aforementioned image segmentation method when executed by a processor.
[0041] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program implements the aforementioned image segmentation method when executed by a processor.
[0042] The present application provides an image segmentation method, comprising: constructing a basic image segmentation model based on a preset convolutional neural network; using a multi-scale convolution operation in the basic image segmentation model to extract features of a data set used for image segmentation to obtain features of different scales; constructing a linear attention mechanism based on a preset deep learning model architecture, and using a Gaussian error linear unit activation function as a constructor of the linear attention mechanism, and then fusing the features through the linear attention mechanism to obtain a multi-scale fused feature map; upsampling and splicing the multi-scale fused feature map, and configuring a corresponding loss function and optimizer for the basic image segmentation model to obtain a target image segmentation model, so as to use the target image segmentation model to perform image segmentation.
[0043] The beneficial technical effects of the present application are as follows: at the algorithm level, by introducing a linear attention mechanism based on a Gaussian error linear unit activation function, the computational complexity is greatly reduced in the realization of the global receptive field, while ensuring the global receptive field of the model, the consumption of computing resources is significantly reduced, thereby improving the computational efficiency. In addition, in response to the problem of multi-scale feature extraction, a strategy of feature fusion is introduced to implement multi-scale convolution operations to enhance the ability of the linear attention mechanism based on the Gaussian error linear unit activation function to extract local feature information, which can enhance the model's perception of features of different scales. At the hardware level, based on the obtained multi-scale fusion feature map, a corresponding optimizer is configured for the model to improve computational efficiency, determine the final target image segmentation model, and use the target image segmentation model to implement automated image segmentation tasks. In this way, the above two innovative mechanisms are combined to effectively integrate modules based on a preset deep learning model architecture into a high-precision image segmentation model, achieving a dual improvement in the performance and computational efficiency of the image segmentation model.
[0044] In addition, the present application provides an image segmentation device, a program product, and a medium, which correspond to the above-mentioned image segmentation method and have the same effect as above. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0046] Figure 1 A flow chart of an image segmentation method disclosed in this application;
[0047] Figure 2 A schematic diagram of a linear attention mechanism based on a Gaussian error linear unit activation function disclosed in this application;
[0048] Figure 3 A fusion module of a linear attention mechanism based on a Gaussian error linear unit activation function and multi-scale features disclosed in this application;
[0049] Figure 4 A schematic diagram of an image segmentation model of a multi-scale attention mechanism module based on a Gaussian error linear unit activation function disclosed in this application;
[0050] Figure 5 A flowchart of a specific image segmentation method disclosed in this application;
[0051] Figure 6 This is a schematic diagram of the structure of an image segmentation device disclosed in this application;
[0052] Figure 7 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION
[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0054] Image segmentation is an important technology in the field of computer vision. By dividing an image into several regions with specific semantics, it has been widely used in many scenarios such as medical image analysis, autonomous driving, remote sensing image processing, and security monitoring. Traditional image segmentation methods rely on manually designed features and rules, which often perform poorly when faced with complex scenes and diverse targets. With the rapid development of deep learning technology, image segmentation methods based on convolutional neural networks (CNNs), such as fully convolutional networks (FCNs), U-Net (Convolutional Networks for Biomedical Image Segmentation), and Mask R-CNN, have significantly improved the accuracy and robustness of image segmentation. In recent years, in the field of image processing, applications based on preset deep learning model architectures have achieved remarkable results, and their excellent global modeling capabilities have shown great potential in image segmentation tasks. However, existing high-precision image segmentation schemes have problems with high computational complexity and low efficiency in multi-scale feature extraction and global receptive field capture. To this end, the present application provides an image segmentation solution that can achieve a dual improvement in image segmentation model performance and computational efficiency.
[0055] The embodiment of the present invention discloses an image segmentation method, see Figure 1 As shown, the method includes:
[0056] Step S11: Construct a basic image segmentation model based on a preset convolutional neural network.
[0057] It is understandable that image segmentation based on convolutional neural networks can improve the accuracy and robustness of image segmentation. Therefore, in an embodiment of the present application, a basic image segmentation model based on a preset convolutional neural network is first constructed. Before constructing the basic image segmentation model, it is necessary to collect and annotate a data set for image segmentation. Specifically, a data set for image segmentation is collected, and data preprocessing and data enhancement are performed on the image data in the data set; wherein the data preprocessing is used to standardize the image data, including adjusting the image size, normalizing pixel values, etc.; in addition, data enhancement operations are also performed, and data enhancement is used to perform any one or more combinations of random cropping, rotation, flipping, and color jittering on the image data to increase the diversity of the data and the robustness of the model.
[0058] Step S12: extracting features from a data set used for image segmentation using a multi-scale convolution operation in the basic image segmentation model to obtain features of different scales.
[0059] At present, multi-scale feature extraction methods usually rely on convolution operations of larger kernels. Although this method can capture information of different scales, it has high computational requirements and low efficiency, and it is difficult to achieve fast calculation without special hardware adaptation. In the embodiment of the present application, for the problem of multi-scale feature extraction, a multiple (multi-scale) convolution feature fusion strategy is introduced to extract features of different scales through multi-scale convolution operations to enhance the ability to extract local feature information.
[0060] Specifically, the sequence data corresponding to each feature vector in the data set used for image segmentation is determined; the sequence data is mapped to obtain the corresponding query matrix, key matrix and value matrix; and the query matrix, the key matrix and the value matrix are respectively extracted using a multi-scale convolution operation in the basic image segmentation model to obtain features of different scales.
[0061] In the embodiment of the present application, each sample in the data set can be represented as a feature vector, and these feature vectors together constitute the basis of the data set. When performing a multi-scale convolution operation, the corresponding serialized sequence data is determined according to the feature vector in the data set. Further, due to the subsequent need to enhance the expression ability of multi-scale features based on the attention mechanism, the local attention mechanism or the hierarchical attention mechanism is used to significantly reduce the consumption of computing resources while ensuring the global receptive field of the model. In the attention mechanism, the query matrix Q, the key matrix K and the value matrix V implement a specific form of self-attention mechanism, which work together in the attention allocation process of the model to help the model better understand and process the key information in the input data. Therefore, the query matrix, the key matrix and the value matrix are subjected to multi-scale deep convolution for feature extraction to obtain features of different scales.
[0062] It can be understood that the multi-scale convolution operation realizes mixed convolution by mixing convolution kernels of different sizes in one convolution operation, thereby capturing different scale features of the image and improving the representation ability and robustness of the model. For example, Q, K, and V are subjected to feature extraction using multi-scale (1x1, 3x3, 5x5) deep convolutions, followed by splicing and linear dimensional transformation, to finally achieve the fusion of multi-scale features.
[0063] Step S13: construct a linear attention mechanism based on a preset deep learning model architecture, and use a Gaussian error linear unit activation function as a constructor of the linear attention mechanism, and then fuse the features through the linear attention mechanism to obtain a multi-scale fused feature map.
[0064] In a feasible implementation, the preset deep learning model architecture can be a Transformer architecture. Image segmentation based on the Transformer architecture can provide many advantages for image segmentation tasks through its unique self-attention mechanism and parallel design. The following embodiments are all described by taking the Transformer architecture as an example, and no further description will be given later.
[0065] Specifically, most Transformer-based image segmentation models use the Softmax mechanism to capture the global receptive field. Although it can effectively capture long-range dependencies, its computational complexity is as high as the square of the input sequence length, resulting in slow computation and huge resource consumption when processing large-size images. Therefore, for example, a linear attention mechanism is constructed based on Transformer to achieve the requirements of the global receptive field, so that the derived linear Transformer has O(N) time and memory complexity, which greatly reduces the computational complexity and improves the computational efficiency.
[0066] Furthermore, in order to reduce computational complexity, the present application embodiment uses the GeLU (Gaussian Error Linear Unit) activation function as the constructor of the linear attention mechanism. GeLU is not only relatively low in computational complexity, but also can better capture nonlinear features, thereby effectively reducing computational complexity and resource consumption while improving model performance.
[0067] After selecting GeLU as the constructor of linear attention, Figure 2 The figure shows an exemplary linear attention diagram based on GeLU. For the input sequence, it is mapped to obtain the query matrix Q, key matrix K and value matrix V respectively; through the linear attention mechanism based on GeLU, column summation, matrix multiplication and other operations are performed in the calculation process, and finally normalized to obtain the corresponding output result.
[0068] Compared with nonlinear attention functions such as Softmax, GeLU is still relatively weak in extracting local feature information. Therefore, in order to effectively overcome this problem, in the embodiment of the present application, the features of different scales obtained by multi-scale convolution operations are fused through a linear attention mechanism to enhance the ability of the GeLU-based linear attention mechanism to extract local feature information.
[0069] Specifically, the query matrix, the key matrix and the value matrix after feature extraction are calculated using the calculation formula of the linear attention mechanism based on the Gaussian error linear unit activation function, and the output matrices obtained after the calculation are sequentially matrix-connected and linearly transformed to obtain the multi-scale fusion feature map.
[0070] like Figure 3 The figure shows a schematic diagram of the fusion module of the GeLU-based linear attention mechanism and multi-scale features. After the input sequence is linearly transformed, Q, K, and V are subjected to feature extraction using multi-scale (1x1, 3x3, 5x5) deep convolutions, and then matrix connection and linear transformation are performed through the GeLU-based linear attention mechanism to finally achieve the fusion of multi-scale features and obtain the corresponding output results. It can be seen that the GeLU attention mechanism is combined with a multi-scale feature information extraction and splicing fusion mechanism on its basis. This mechanism effectively enhances the relatively insufficient problem of the linear attention mechanism in obtaining local feature information, and complements the linear attention mechanism in this aspect through multi-scale small-size convolutions and the final splicing fusion.
[0071] In addition, multiple attention distributions can be processed in parallel through a multi-head attention mechanism, thereby more comprehensively capturing multiple associations in the sequence and further enhancing the model's attention capture and expression capabilities. Specifically, a multi-scale fusion feature map based on the parallel output of the multi-head attention mechanism is obtained, and the multi-scale fusion feature map output in parallel is sequentially matrix-connected and linearly transformed to obtain the target output; accordingly, the target output is upsampled and concatenated, and the corresponding loss function and optimizer are configured for the basic image segmentation model to obtain the target image segmentation model.
[0072] Step S14: performing upsampling and splicing operations on the multi-scale fusion feature map, and configuring a corresponding loss function and optimizer for the basic image segmentation model to obtain a target image segmentation model, so as to perform image segmentation using the target image segmentation model.
[0073] In the embodiment of the present application, a multi-scale feature pyramid is introduced to enable efficient processing and fusion of multi-scale information. It forms a feature pyramid structure through top-down and bottom-up connections. The top-down part uses high-level feature maps to provide rich semantic information, while the bottom-up part restores the details of low-resolution feature maps by upsampling layer by layer. This structure enables the extraction and fusion of feature information at different scales, thereby improving the robustness and detection accuracy of the model for targets of different sizes.
[0074] Specifically, the multi-scale fusion feature map is upsampled using a transposed convolution operation to restore the feature map to the resolution of the original input image. And in the upsampling process, the features of the current layer are spliced with the features of the corresponding coding layer. That is, in each upsampling stage, the features of the current layer are spliced with the features of the corresponding coding layer to retain high-resolution feature information. In this way, by fusing shallow and deep features, the detailed expression of the segmentation result is enhanced.
[0075] Furthermore, the corresponding loss function and optimizer are configured for the basic image segmentation model. A suitable loss function is selected to guide the optimization of the model; the optimizer is configured and a suitable learning rate scheduling strategy is configured for it to ensure that the model can converge effectively. In addition, the gradient clipping technology is used to eliminate the gradient explosion, and finally the target image segmentation model is trained.
[0076] It should be pointed out that, in the process of finally determining the target image segmentation model, it also includes: setting the batch size and training rounds for the basic image segmentation model in the training process; performing row model training on the basic image segmentation model based on the batch size and the training rounds, and verifying the current basic image segmentation model using the verification set after each training round, so as to perform parameter tuning on the basic image segmentation model according to the verification results.
[0077] Row model training refers to dividing the parameters or data of the model into multiple parts in a distributed computing environment, and processing them in parallel on different computing nodes to improve training efficiency and scalability. In the embodiment of the present application, by setting a suitable batch size and training rounds, verification is performed after each training round, the performance of the model on the verification set is evaluated, and the hyperparameters are adjusted according to the verification results to perform model tuning. It is understandable that the setting parameters of the batch size and training rounds can be adjusted accordingly based on experience to adapt to different application scenarios.
[0078] like Figure 4 The figure shows an example of embedding a basic image segmentation model after fusing a linear attention mechanism based on GeLU and multi-scale features. In this model, a fusion module of a linear attention mechanism based on GeLU and multi-scale features is embedded in the basic image segmentation model. Compared with the Transformer attention module based on Softmax, this module can greatly reduce the computational complexity and improve the computational efficiency, while ensuring the retention of multi-scale feature information and the performance of the image segmentation model. The final model output is obtained through a series of model operation modules, including a convolution module, a fusion module of a linear attention mechanism based on GeLU and multi-scale features, and a residual antecedent superposition operation.
[0079] The present application provides an image segmentation method, comprising: constructing a basic image segmentation model based on a preset convolutional neural network; using a multi-scale convolution operation in the basic image segmentation model to extract features of a data set used for image segmentation to obtain features of different scales; constructing a linear attention mechanism based on a preset deep learning model architecture, and using a Gaussian error linear unit activation function as a constructor of the linear attention mechanism, and then fusing the features through the linear attention mechanism to obtain a multi-scale fused feature map; upsampling and splicing the multi-scale fused feature map, and configuring a corresponding loss function and optimizer for the basic image segmentation model to obtain a target image segmentation model, so as to use the target image segmentation model to perform image segmentation.
[0080] The beneficial technical effects of the present application are as follows: at the algorithm level, by introducing a linear attention mechanism based on a Gaussian error linear unit activation function, the computational complexity is greatly reduced in the realization of the global receptive field, while ensuring the global receptive field of the model, the consumption of computing resources is significantly reduced, thereby improving the computational efficiency. In addition, in response to the problem of multi-scale feature extraction, a strategy of feature fusion is introduced to implement multi-scale convolution operations to enhance the ability of the linear attention mechanism based on the Gaussian error linear unit activation function to extract local feature information, which can enhance the model's perception of features of different scales. At the hardware level, based on the obtained multi-scale fusion feature map, a corresponding optimizer is configured for the model to improve computational efficiency, determine the final target image segmentation model, and use the target image segmentation model to implement automated image segmentation tasks. In this way, the above two innovative mechanisms are combined to effectively integrate modules based on a preset deep learning model architecture into a high-precision image segmentation model, achieving a dual improvement in the performance and computational efficiency of the image segmentation model.
[0081] In this embodiment, the implementation of the linear attention mechanism based on the Gaussian error linear unit activation function is further described. Similarly, the preset deep learning model architecture is a Transformer architecture as an example for description.
[0082] First, let’s explain Transformer and attention mechanism:
[0083] set up is a sequence of N feature vectors of dimension F. Transformer is a function T: , consisting of L Transformer layers The composition is as follows: ; Among them, the function Each feature is transformed independently one by one, usually through a two-layer Feed Forward Networks (FFN). Is the self-attention function (Self-Attention). Self-attention function The weighted average of the feature representations at each position is calculated, with the weights proportional to the similarity between these representations. Formally, the input sequence x is passed through three matrices , , ,here( ) are projected to the corresponding representations Q, K, and V. The output of all positions The calculation is as follows:
[0084] ;
[0085] In the above formula, the Softmax function is applied row by row. Q, K, and V are called "query", "key", and "value" respectively. A specific form of self-attention mechanism is implemented in this formula, called Softmax attention, where the similarity score is an exponential function of the dot product between Q and K. Assuming that the i-th row of the matrix returns a vector, the generalized attention equation for any similarity function can be written as follows:
[0086] ;
[0087] here Represents the similarity function. If the similarity function is replaced by , then this formula is consistent with the output of all the above positions The calculation formulas are equivalent.
[0088] Furthermore, linear attention is proposed based on this attention mechanism:
[0089] It should be noted that the output of all the above positions The definition of the attention mechanism in the calculation formula is universal, so it can be used to define several other attention implementations, such as polynomial attention. In order to define an attention function using the calculation formula of the generalized attention equation for any similarity function above, it is necessary to The only constraint imposed is its non-negativity. That is, any kernel function satisfies .
[0090] Given a feature representation The kernel function converts the output of all the above positions The calculation formula is rewritten as follows:
[0091] ;
[0092] This can then be further simplified by exploiting the associativity of matrix multiplication:
[0093] ;
[0094] The above formula is easier to understand when the numerator is written in vectorized form, as shown below:
[0095] ;
[0096] It can be seen that the output from all the above positions As can be seen from the calculation formula, the computational cost of Softmax attention increases with O(N 2) changes, where N is the sequence length. The same is true for memory requirements, because the full attention matrix must be stored to calculate the gradients of the query, key, and value. In contrast, the linear Transformer derived from the calculation formula further simplified by the associativity of matrix multiplication in the embodiment of the present application has a time and memory complexity of O(N). Only need to calculate and once, and reuse them in each query.
[0097] Secondly, from the above description, we can understand that the computational complexity of Softmax to calculate attention is O(N 2 ). In the embodiment of the present application, it is replaced by a linear attention with a computational complexity of only O(N) to achieve the requirement of global receptive field. In constructing the similarity function of linear attention There are several options, and this embodiment is described for GeLU:
[0098] The basic calculation formula of GeLU is: ;
[0099] in, is the cumulative distribution function of the standard normal distribution, which can be approximated as:
[0100] ;
[0101] The computational complexity and memory complexity of GeLU are both O(1). The advantage of GeLU is its smoothness, which can better handle inputs less than zero and provide more delicate activations. In some tasks, using GeLU can improve the performance of the model, especially in natural language processing tasks. In addition, GeLU gradually approaches a linear activation function, which makes training more stable in some cases. Therefore, GeLU is selected as the constructor of linear attention in this embodiment.
[0102] Furthermore, a linear attention mechanism based on the Gaussian error linear unit activation function is constructed.
[0103] Let the similarity function be , then the calculation formula of the generalized attention equation of any similarity function disclosed in the above content can be rewritten as:
[0104] ;
[0105] Then, we can exploit the associativity of matrix multiplication to reduce the computational complexity and memory usage from quadratic to linear without changing its functionality:
[0106] ;
[0107] Where Q is the query matrix, K is the key matrix, V is the value matrix, T is the transpose operation, and i and j are the row and column numbers of the matrix. and Once, they can then be reused in each query, requiring only O(N) computational cost and O(N) memory. In addition, GeLU linear attention does not involve hardware-unfriendly operations like Softmax. Designing dedicated accelerators and optimizing storage access patterns can also be more efficient on hardware and improve computational efficiency.
[0108] It can be seen that in order to further optimize the computational performance of the attention mechanism, this embodiment introduces the GELU activation function, which replaces the traditional Softmax. The computational complexity of the GELU activation function is a constant level O(1), compared to the O(N) complexity of Softmax. 2 ) complexity, significantly reducing the computational complexity and accelerating the attention calculation. On the other hand, it can maintain high computational efficiency when processing large-size images while ensuring the improvement of model performance.
[0109] The present application embodiment discloses a specific image segmentation method, see Figure 5 As shown, the method includes:
[0110] Step S21: Initialize a preset convolutional neural network, remove the last fully connected layer in the preset convolutional neural network, and then select a corresponding target version type for the preset convolutional neural network to obtain the basic image segmentation model.
[0111] It is understandable that image segmentation based on convolutional neural networks can improve the accuracy and robustness of image segmentation. Therefore, in the embodiment of the present application, a basic image segmentation model based on a preset convolutional neural network is first constructed. Specifically, the backbone network (such as EfficientNet) is initialized. The pre-trained EfficientNet model is loaded and its last fully connected layer is removed for image segmentation tasks. According to the specific task requirements, a suitable EfficientNet version (such as EfficientNet-B0 to EfficientNet-B7) is selected.
[0112] It should be pointed out that data preprocessing and data enhancement are required before this. It is necessary to collect and annotate the data set for image segmentation, and standardize the image data, including adjusting the image size, normalizing pixel values, etc. In addition, data enhancement operations such as random cropping, rotation, flipping, and color jittering are also required to increase the diversity of the data and the robustness of the model.
[0113] Step S22: extracting features from a data set used for image segmentation using a multi-scale convolution operation in the basic image segmentation model to obtain features of different scales.
[0114] Step S23: construct a linear attention mechanism based on a preset deep learning model architecture, and use a Gaussian error linear unit activation function as a constructor of the linear attention mechanism, and then fuse the features through the linear attention mechanism to obtain a multi-scale fused feature map.
[0115] Step S24: upsampling and splicing operations are performed on the multi-scale fusion feature map, and a corresponding loss function and optimizer are configured for the basic image segmentation model to obtain a target image segmentation model, so as to perform image segmentation using the target image segmentation model.
[0116] In the embodiment of the present application, a GeLU-based multi-scale attention module is embedded in each module after skipping several stages. The multi-scale attention module extracts features of different scales through multi-scale convolution operations, and uses the GeLU activation function for nonlinear transformation to enhance the feature expression capability. The introduction of this module can improve the model's perception of features of different scales. Then upsampling and splicing operations are performed, and loss functions and optimizers are configured. The target image segmentation model is finally obtained through row model training.
[0117] Among them, for more specific processing procedures of the above steps S22 to S24, reference can be made to the corresponding contents disclosed in the above embodiments, which will not be repeated here.
[0118] Step S25: Use the target image segmentation model to perform verification on a test set, and use a preset evaluation index to evaluate the target image segmentation model.
[0119] In an embodiment of the present application, the target image segmentation model finally obtained is evaluated on the test set. Specifically, the segmentation result of the image segmentation of the test set using the target image segmentation model is obtained, and the segmentation result is visualized and analyzed according to the preset evaluation index to verify the segmentation effect of the target image segmentation model.
[0120] Step S26: Determine the target application scenario, and export the target image segmentation model into a target deployment format that can be deployed in the target application scenario; deploy the target image segmentation model in the target application scenario according to the target deployment format, so as to perform image segmentation using the target image segmentation model based on the target deployment format.
[0121] Finally, export the trained model into a format suitable for deployment, deploy the model in actual application scenarios, and implement automated image segmentation tasks.
[0122] In addition, through the embodiments of the present application, it is also possible to adapt and optimize a wider series of high-precision dense prediction models and tasks similar to image segmentation tasks, so as to achieve a dual improvement in model performance and computing efficiency.
[0123] The beneficial technical effects of the present application are as follows: at the algorithm level, by introducing a linear attention mechanism based on a Gaussian error linear unit activation function, the computational complexity is greatly reduced in the realization of the global receptive field, while ensuring the global receptive field of the model, the consumption of computing resources is significantly reduced, thereby improving the computational efficiency. In addition, in response to the problem of multi-scale feature extraction, a strategy of feature fusion is introduced to implement multi-scale convolution operations to enhance the ability of the linear attention mechanism based on the Gaussian error linear unit activation function to extract local feature information, which can enhance the model's perception of features of different scales. At the hardware level, based on the obtained multi-scale fusion feature map, a corresponding optimizer is configured for the model to improve computational efficiency, determine the final target image segmentation model, and use the target image segmentation model to implement automated image segmentation tasks. In this way, the above two innovative mechanisms are combined to effectively integrate modules based on a preset deep learning model architecture into a high-precision image segmentation model, achieving a dual improvement in the performance and computational efficiency of the image segmentation model.
[0124] Correspondingly, the present application also discloses an image segmentation device, see Figure 6 As shown, the device comprises:
[0125] A basic model building module 11 is used to build a basic image segmentation model based on a preset convolutional neural network;
[0126] A multi-scale feature extraction module 12, used to extract features from a data set used for image segmentation using a multi-scale convolution operation in the basic image segmentation model to obtain features of different scales;
[0127] An attention mechanism construction module 13 is used to construct a linear attention mechanism based on a preset deep learning model architecture, and use a Gaussian error linear unit activation function as a constructor of the linear attention mechanism, and then perform feature fusion on the features through the linear attention mechanism to obtain a multi-scale fusion feature map;
[0128] The image segmentation module 14 is used to perform upsampling and splicing operations on the multi-scale fusion feature map, and configure a corresponding loss function and optimizer for the basic image segmentation model to obtain a target image segmentation model, so as to use the target image segmentation model to perform image segmentation.
[0129] Among them, for more specific working processes of the above-mentioned modules, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.
[0130] It can be seen that the above scheme of this embodiment includes: constructing a basic image segmentation model based on a preset convolutional neural network; using a multi-scale convolution operation in the basic image segmentation model to extract features of a data set used for image segmentation to obtain features of different scales; constructing a linear attention mechanism based on a preset deep learning model architecture, and using a Gaussian error linear unit activation function as a constructor of the linear attention mechanism, and then fusing the features through the linear attention mechanism to obtain a multi-scale fused feature map; upsampling and splicing the multi-scale fused feature map, and configuring a corresponding loss function and optimizer for the basic image segmentation model to obtain a target image segmentation model, so as to use the target image segmentation model to perform image segmentation.
[0131] The beneficial technical effects of the present application are as follows: at the algorithm level, by introducing a linear attention mechanism based on a Gaussian error linear unit activation function, the computational complexity is greatly reduced in the realization of the global receptive field, while ensuring the global receptive field of the model, the consumption of computing resources is significantly reduced, thereby improving the computational efficiency. In addition, in response to the problem of multi-scale feature extraction, a strategy of feature fusion is introduced to implement multi-scale convolution operations to enhance the ability of the linear attention mechanism based on the Gaussian error linear unit activation function to extract local feature information, which can enhance the model's perception of features of different scales. At the hardware level, based on the obtained multi-scale fusion feature map, a corresponding optimizer is configured for the model to improve computational efficiency, determine the final target image segmentation model, and use the target image segmentation model to implement automated image segmentation tasks. In this way, the above two innovative mechanisms are combined to effectively integrate modules based on a preset deep learning model architecture into a high-precision image segmentation model, achieving a dual improvement in the performance and computational efficiency of the image segmentation model.
[0132] In a specific implementation, the image segmentation device further includes:
[0133] The data set collection module is used to collect data sets for image segmentation and perform data preprocessing and data enhancement on the image data in the data set; wherein the data preprocessing is used to standardize the image data; and the data enhancement is used to perform any one or a combination of random cropping, rotation, flipping and color jittering on the image data.
[0134] In a specific implementation, the basic model building module 11 is specifically used to:
[0135] Initialize a preset convolutional neural network, remove the last fully connected layer in the preset convolutional neural network, and then select a corresponding target version type for the preset convolutional neural network to obtain the basic image segmentation model.
[0136] In a specific implementation, the multi-scale feature extraction module 12 is specifically used to:
[0137] Determine the sequence data corresponding to each feature vector in the data set used for image segmentation;
[0138] Mapping the sequence data to obtain corresponding query matrix, key matrix and value matrix;
[0139] In the basic image segmentation model, multi-scale convolution operations are used to extract features from the query matrix, the key matrix and the value matrix respectively to obtain features of different scales.
[0140] In a specific implementation, the attention mechanism construction module 13 is specifically used to:
[0141] The query matrix, the key matrix and the value matrix after feature extraction are calculated using the calculation formula of the linear attention mechanism based on the Gaussian error linear unit activation function, and the output matrices obtained after the calculation are sequentially matrix-connected and linearly transformed to obtain the multi-scale fusion feature map.
[0142] In a specific implementation, the image segmentation device further includes:
[0143] The multi-head attention mechanism execution module is used to obtain the multi-scale fusion feature map based on the parallel output of the multi-head attention mechanism, and perform matrix connection and linear transformation on the parallel output multi-scale fusion feature map in sequence to obtain the target output;
[0144] Accordingly, the image segmentation module 14 is specifically used for:
[0145] The target output is upsampled and concatenated, and a corresponding loss function and optimizer are configured for the basic image segmentation model to obtain a target image segmentation model.
[0146] In a specific implementation, the image segmentation module 14 is specifically used to:
[0147] The corresponding loss function and optimizer are configured for the basic image segmentation model, and the gradient clipping technology is used to eliminate the gradient explosion to obtain the target image segmentation model.
[0148] In a specific implementation, the image segmentation module 14 is specifically used to:
[0149] The multi-scale fusion feature map is up-sampled using a transposed convolution operation, and during the up-sampling process, the features of the current layer are concatenated with the features of the corresponding encoding layer.
[0150] In a specific implementation, the image segmentation module 14 further includes:
[0151] A row model training module is used to set the batch size and training rounds for the basic image segmentation model during the training process; based on the batch size and the training rounds, the basic image segmentation model is trained on the row model, and after each training round, the current basic image segmentation model is verified using a verification set so as to tune the parameters of the basic image segmentation model according to the verification results.
[0152] In a specific implementation, the image segmentation device further includes:
[0153] The verification module is used to use the target image segmentation model to perform verification on a test set and to evaluate the target image segmentation model using a preset evaluation index.
[0154] In a specific implementation, the verification module is specifically used to:
[0155] Obtain the segmentation results of the image segmentation of the test set using the target image segmentation model, and perform visual analysis on the segmentation results according to the preset evaluation index to verify the segmentation effect of the target image segmentation model.
[0156] In a specific implementation, the image segmentation device further includes:
[0157] A format export module, used to determine a target application scenario and export the target image segmentation model into a target deployment format that can be deployed in the target application scenario;
[0158] A deployment module is used to deploy the target image segmentation model in the target application scenario according to the target deployment format, so as to perform image segmentation using the target image segmentation model based on the target deployment format.
[0159] Furthermore, the present application also discloses an electronic device. Figure 7 This is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the content in the diagram cannot be considered as any limitation on the scope of use of the present application.
[0160] Figure 7The present invention provides a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the image segmentation method disclosed in any of the above embodiments. In addition, the electronic device 20 in this embodiment may specifically be a computer.
[0161] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.
[0162] In addition, the memory 22 as a carrier for resource storage may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222 and data 223, etc. The data 223 may include various data. The storage method may be temporary storage or permanent storage.
[0163] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the image segmentation method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.
[0164] Furthermore, the embodiment of the present application also discloses a computer-readable storage medium, wherein the computer-readable storage medium mentioned here includes a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a magnetic disk or an optical disk, or any other form of storage medium known in the technical field. Wherein, when the computer program is executed by the processor, the aforementioned image segmentation method is implemented. For the specific steps of the method, reference can be made to the corresponding contents disclosed in the aforementioned embodiment, and no further description will be given here.
[0165] Furthermore, an embodiment of the present application also provides a computer program product, including a computer program / instruction, which implements any one of the above-mentioned image segmentation methods when executed by a processor.
[0166] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0167] The steps of the image segmentation method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0168] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0169] The above is a detailed introduction to an image segmentation method, device, program product and medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for a general technician in this field, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. An image segmentation method, characterized in that: include: Build a basic image segmentation model based on a preset convolutional neural network; In the basic image segmentation model, a multi-scale convolution operation is used to extract features from a data set used for image segmentation to obtain features of different scales; Constructing a linear attention mechanism based on a preset deep learning model architecture, and using a Gaussian error linear unit activation function as a constructor of the linear attention mechanism, and then fusing the features through the linear attention mechanism to obtain a multi-scale fused feature map; Performing upsampling and splicing operations on the multi-scale fusion feature map, and configuring a corresponding loss function and optimizer for the basic image segmentation model to obtain a target image segmentation model, so as to perform image segmentation using the target image segmentation model; The method of extracting features from a dataset used for image segmentation using a multi-scale convolution operation in the basic image segmentation model to obtain features of different scales includes: Determine the sequence data corresponding to each feature vector in the data set used for image segmentation; Mapping the sequence data to obtain corresponding query matrix, key matrix and value matrix; Performing feature extraction on the query matrix, the key matrix, and the value matrix respectively using a multi-scale convolution operation in the basic image segmentation model to obtain features of different scales; The process of constructing a linear attention mechanism based on a preset deep learning model architecture and using a Gaussian error linear unit activation function as a constructor of the linear attention mechanism includes: Let the similarity function be , the corresponding generalized attention equation is rewritten as: ; Use the associativity of matrix multiplication to reduce the computational complexity and memory usage from quadratic to linear: ; Wherein, Q is the query matrix, K is the key matrix, V is the value matrix, T is the transpose operation, and i and j are the row and column numbers of the matrix.
2. The image segmentation method according to claim 1, wherein: Before constructing the basic image segmentation model based on the preset convolutional neural network, the method further includes: Collect a dataset for image segmentation and perform data preprocessing and data enhancement on the image data in the dataset; wherein the data preprocessing is used to standardize the image data; and the data enhancement is used to perform any one or more combinations of random cropping, rotation, flipping, and color jittering on the image data.
3. The image segmentation method according to claim 1, wherein: The construction of a basic image segmentation model based on a preset convolutional neural network includes: Initialize a preset convolutional neural network, remove the last fully connected layer in the preset convolutional neural network, and then select a corresponding target version type for the preset convolutional neural network to obtain the basic image segmentation model.
4. The image segmentation method according to claim 1, wherein: The feature fusing of the features through the linear attention mechanism to obtain a multi-scale fused feature map includes: The query matrix, the key matrix and the value matrix after feature extraction are calculated using the calculation formula of the linear attention mechanism based on the Gaussian error linear unit activation function, and the output matrices obtained after the calculation are sequentially matrix-connected and linearly transformed to obtain the multi-scale fusion feature map.
5. The image segmentation method according to claim 1, wherein: Also includes: Obtain multi-scale fusion feature maps based on the parallel output of the multi-head attention mechanism, and perform matrix connection and linear transformation on the parallel output multi-scale fusion feature maps in sequence to obtain the target output; Accordingly, performing upsampling and splicing operations on the multi-scale fusion feature map, and configuring a corresponding loss function and optimizer for the basic image segmentation model to obtain a target image segmentation model, includes: The target output is upsampled and concatenated, and a corresponding loss function and optimizer are configured for the basic image segmentation model to obtain a target image segmentation model.
6. The image segmentation method according to claim 1, wherein: The configuring a corresponding loss function and an optimizer for the basic image segmentation model to obtain a target image segmentation model includes: The corresponding loss function and optimizer are configured for the basic image segmentation model, and the gradient clipping technology is used to eliminate the gradient explosion to obtain the target image segmentation model.
7. The image segmentation method according to claim 1, wherein: The upsampling and splicing operations on the multi-scale fusion feature map include: The multi-scale fusion feature map is up-sampled using a transposed convolution operation, and during the up-sampling process, the features of the current layer are concatenated with the features of the corresponding encoding layer.
8. The image segmentation method according to claim 1, wherein: The process of performing upsampling and splicing operations on the multi-scale fusion feature map and configuring a corresponding loss function and optimizer for the basic image segmentation model to obtain a target image segmentation model also includes: Setting the batch size and training rounds for the basic image segmentation model during training; The basic image segmentation model is trained based on the batch size and the training rounds, and the current basic image segmentation model is verified using a validation set after each training round, so as to tune the parameters of the basic image segmentation model according to the verification results.
9. The image segmentation method according to claim 1, wherein: After performing upsampling and splicing operations on the multi-scale fusion feature map and configuring a corresponding loss function and optimizer for the basic image segmentation model to obtain a target image segmentation model, the method further includes: The target image segmentation model is used to perform verification on a test set, and the target image segmentation model is evaluated using preset evaluation indicators.
10. The image segmentation method according to claim 9, characterized in that: The method of verifying the target image segmentation model on a test set and evaluating the target image segmentation model using a preset evaluation index includes: Obtain the segmentation results of the test set using the target image segmentation model, and perform a visual analysis on the segmentation results according to the preset evaluation index to verify the segmentation effect of the target image segmentation model.
11. The image segmentation method according to any one of claims 1 to 10, characterized in that: After performing upsampling and splicing operations on the multi-scale fusion feature map and configuring a corresponding loss function and optimizer for the basic image segmentation model to obtain a target image segmentation model, so as to perform image segmentation using the target image segmentation model, the method further includes: Determining a target application scenario, and exporting the target image segmentation model into a target deployment format that can be deployed in the target application scenario; The target image segmentation model is deployed in the target application scenario according to the target deployment format, so as to perform image segmentation using the target image segmentation model based on the target deployment format.
12. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to load and execute the computer program to implement the image segmentation method according to any one of claims 1 to 11.
13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the image segmentation method according to any one of claims 1 to 11 are implemented.
14. A computer-readable storage medium, characterized in that Used to store a computer program; wherein when the computer program is executed by a processor, the image segmentation method according to any one of claims 1 to 11 is implemented.
