Image segmentation method and device, terminal equipment and storage medium
By freezing the parameters of the feature extraction module and improving it to a LoRA branch, the problem of slow training speed of image segmentation models was solved, achieving fast and accurate image segmentation results.
Patent Information
- Application Number
- CN202510780634.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-11
- Publication Date
- 2025-10-31
AI Technical Summary
Existing image segmentation models converge slowly during training and fail to reach their optimal state, resulting in inaccurate segmentation.
The parameters of the feature extraction module are frozen, the decoding module is trained using a small number of samples, and the feature extraction module is improved to have a LoRA branch. The parameters are adjusted by low-rank matrix factorization to simplify the training process.
This improves the training speed and accuracy of image segmentation models, enabling fast and accurate image segmentation.
Smart Images
Figure CN120876844A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of image segmentation technology, and in particular relates to an image segmentation method, apparatus, terminal device and storage medium. Background Technology
[0002] Image segmentation is the technique and process of dividing an image into several specific regions with unique properties and extracting the target of interest. For example, during robot operation, the robot needs to quickly collect data and rapidly perform image segmentation on the collected data to obtain the required data.
[0003] Currently, image segmentation models are used to segment images, but these models need to be trained before use to ensure accurate output. However, the convergence speed of current image segmentation models is not fast enough during training, resulting in suboptimal models and inaccurate segmentation. Therefore, how to train image segmentation models accurately and efficiently is a problem that needs to be solved. Summary of the Invention
[0004] This application provides an image segmentation method, apparatus, terminal device, and storage medium, which can quickly and accurately train an image segmentation model, enabling the trained image segmentation model to segment images more accurately.
[0005] In a first aspect, embodiments of this application provide an image segmentation method, including:
[0006] Obtain a first sample set and a first image segmentation model, wherein the first image segmentation model includes a feature extraction module and a data decoding module, and the output data of the feature extraction module is the input data of the data decoding module;
[0007] Freeze the parameters in the feature extraction module of the first image segmentation model, train the first image segmentation model using the first sample set, update the parameters in the data decoding module of the first image segmentation model, and obtain the second image segmentation model;
[0008] Obtain a second sample set and a third image segmentation model, wherein the number of samples in the second sample set is greater than the number of samples in the first sample set, and the third image segmentation model is a model obtained by constructing the feature extraction module in the second image segmentation model into a feature extraction module with a LoRA branch;
[0009] The third image segmentation model is trained using the second sample set to obtain the target image segmentation model;
[0010] The target image to be segmented is segmented using the target image segmentation model to obtain a segmented image.
[0011] In one possible implementation of the first aspect, after freezing the parameters in the feature extraction module of the first image segmentation model, training the first image segmentation model using a first sample set, updating the parameters in the data decoding module of the first image segmentation model, and obtaining the second image segmentation model, the method further includes:
[0012] Obtain a third sample set, wherein the number of samples in the third sample set is less than the number of samples in the second sample set;
[0013] The second image segmentation model is trained using the third sample set, and the parameters of the feature extraction module and the data decoding module in the second image segmentation model are updated to obtain the fourth image segmentation model.
[0014] Accordingly, the third image segmentation model is the model obtained by constructing the feature extraction module in the fourth image segmentation model into a feature extraction module with a LoRA branch.
[0015] In one possible implementation of the first aspect, the feature extraction module is a DINO network module.
[0016] In one possible implementation of the first aspect, the DINO network consists of multiple attention mechanisms, and at least one of the attention mechanisms in the DINO network module of the third image segmentation model has a LoRA branch.
[0017] In one possible implementation of the first aspect, the step of segmenting the target image to be segmented using the target image segmentation model to obtain a segmented image includes:
[0018] Obtain the target image to be segmented;
[0019] The target image is input into the trained target image segmentation model, and the DINO network module in the target image segmentation model is used to extract features from the target image to obtain the initial feature data of the target image. The DINO network module in the target image segmentation model has a LoRA branch.
[0020] The initial feature data are decoded using the data decoding module in the target image segmentation model to obtain a segmented image of the target image.
[0021] In one possible implementation of the first aspect, the step of decoding each of the initial feature data using the data decoding module in the target image segmentation model to obtain a segmented image of the target image includes:
[0022] The data decoding module in the target image segmentation model is used to reconstruct the features of each initial feature data to obtain the first candidate data corresponding to each initial feature data, and each first candidate data has the same format.
[0023] The first candidate data is sampled to obtain the second candidate data corresponding to each of the first candidate data.
[0024] Feature fusion is performed on each of the second candidate data to obtain the target feature data;
[0025] A segmented image of the target image is generated based on the target feature data.
[0026] In one possible implementation of the first aspect, training the third image segmentation model using the second sample set to obtain the target image segmentation model includes:
[0027] The original parameters of the feature extraction module in the third image segmentation model are frozen. The third image segmentation model is trained using the second sample set. The parameters in the LoRA branch and the parameters of the data decoding module in the third image segmentation model are updated to obtain the target image segmentation model.
[0028] Secondly, embodiments of this application provide an image segmentation apparatus, comprising:
[0029] The first sample acquisition module is used to acquire a first sample set and a first image segmentation model, wherein the first image segmentation model includes a feature extraction module and a data decoding module, and the output data of the feature extraction module is the input data of the data decoding module;
[0030] The first training module is used to freeze the parameters in the feature extraction module of the first image segmentation model, train the first image segmentation model using the first sample set, update the parameters in the data decoding module of the first image segmentation model, and obtain the second image segmentation model.
[0031] The second sample acquisition module is used to acquire a second sample set and a third image segmentation model, wherein the number of samples in the second sample set is greater than the number of samples in the first sample set, and the third image segmentation model is a model obtained by constructing the feature extraction module in the second image segmentation model into a feature extraction module with a LoRA branch;
[0032] The second training module is used to train the third image segmentation model using the second sample set to obtain the target image segmentation model.
[0033] The image segmentation module is used to segment the target image to be segmented using the target image segmentation model to obtain a segmented image.
[0034] Thirdly, embodiments of this application provide a terminal device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image segmentation method described in any one of the first aspects above.
[0035] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image segmentation method described in any one of the first aspects above.
[0036] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the image segmentation method described in any one of the first aspects.
[0037] The beneficial effects of the first aspect of this application compared to the prior art are as follows: This application first obtains a first sample set, and uses the first sample set to train a first image segmentation model to update the parameters of the data decoding module in the first image segmentation model. However, the parameters of the feature extraction module in the first image segmentation module are not updated, resulting in a second image segmentation model. Since the feature extraction module is more difficult to train than the data decoding module, updating the parameters in the data decoding module first in this application reduces the amount of parameter updates, making the training of the first image segmentation model simpler and faster. Furthermore, it allows the data decoding module to have better initialization parameters, facilitating subsequent training of the model.
[0038] After obtaining the second image segmentation model, it is improved by constructing a feature extraction module with a LoRA branch, resulting in a third image segmentation model. The third image segmentation model is trained using the second sample set to obtain the target image segmentation model. Finally, the trained target image segmentation model is used to segment the target image to be segmented, yielding the segmented image. The inclusion of a LoRA branch in the third image segmentation model accelerates its training. Since the parameters of the data decoding module have already been updated and optimized, training the third image segmentation model further improves its overall training speed, reduces training difficulty, and enables rapid convergence, leading to a quickly accurate target image segmentation model.
[0039] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description
[0040] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0041] Figure 1 This is a structural diagram of an image segmentation model provided in an embodiment of this application;
[0042] Figure 2 This is a schematic flowchart of an image segmentation method provided in an embodiment of this application;
[0043] Figure 3 This is a structural diagram of a LoRA branch provided in an embodiment of this application;
[0044] Figure 4 This is a flowchart illustrating a method for training an image segmentation model using a third sample set, according to an embodiment of this application.
[0045] Figure 5 This is a schematic diagram of the structure of an image segmentation apparatus provided in an embodiment of this application;
[0046] Figure 6 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation
[0047] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.
[0048] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0049] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.
[0050] Image segmentation is the process of separating a desired image from a collected image. In robot interaction scenarios, due to the complexity of the segmentation environment and the numerous requirements, robots need to quickly collect data and segment it rapidly.
[0051] Current image segmentation methods have poor generalization ability, cannot be applied to all scenarios, or are not accurate enough in segmenting images. Moreover, when using current image segmentation models for image segmentation, the convergence speed is often not fast enough or converges locally too early, failing to reach the optimal state, making the training of image segmentation models difficult and inefficient.
[0052] Based on this, this application proposes an image segmentation model, which includes a feature extraction module and a data decoding module. When training the image segmentation model, the parameters of the feature extraction module, which is more difficult to train, can be frozen first. The image segmentation model is then trained using a small number of samples, and the data decoding module in the image segmentation model is updated.
[0053] Then, the feature extraction module in the image segmentation model is modified to have a LoRA branch. The image segmentation model with the LoRA branch is trained using samples, and the parameters in the LoRA branch and the data decoding module are updated. Since LoRA adjusts the model parameters through low-rank matrix factorization, training the image segmentation model with the LoRA branch is simpler and faster than training the original image segmentation model. Moreover, when training the image segmentation model with a small number of samples, the image segmentation model can converge quickly.
[0054] To address the issue of poor generalization in image segmentation models, the inventors discovered that self-supervised visual transformer models (DINO networks, such as the second-generation DINO network) exhibit better generalization. The DINO network is a visual transformer model based on self-supervised learning, used for learning visual feature representations in unlabeled data. Therefore, the inventors designed an image segmentation model based on the DINO network. Specifically, the feature extraction module utilizes the DINO network for feature extraction, making the image segmentation model applicable to any scenario and enabling more accurate image segmentation.
[0055] Specifically, such as Figure 1 The diagram shows the structure of the image segmentation model. The image segmentation model includes a feature extraction module 1 and a data decoding module 2.
[0056] Feature extraction module 1 uses a DINO network. After extracting features from the input target image, the feature extraction module outputs four feature data points, which are: Figure 1 The Transformer blocks 1-4 are used in this implementation. In one approach, the DINO network consists of 12 Transformers (attention mechanism), outputting feature data at layers 3, 6, 9, and 12. Specifically, layer 3 outputs shallow features, also known as local features, such as edge and texture features; layer 12 outputs deep features, also known as global features, such as object categories. For example, each Transformer block outputs 1×1024×768 feature data.
[0057] The data decoding module 2 includes a feature recombination unit, a sampling unit, a feature fusion unit, and a result output unit.
[0058] The feature reconstructing unit primarily transforms the feature data output from feature extraction module 1 into a unified format suitable for subsequent processing. The feature reconstructing unit comprises four sub-units (Feature Recon1-Feature Recon4). Each feature data output from feature extraction module 1 is input into a corresponding sub-unit for feature reconstructing. Specifically, the feature data output from block 1 is input to Feature Recon1; the feature data output from block 2 is input to Feature Recon2; the feature data output from block 3 is input to Feature Recon3; and the feature data output from block 4 is input to Feature Recon4. Each feature reconstructing sub-unit includes a Reshape layer and a Project layer. The Project layer includes a first convolutional layer and a second convolutional layer. The Reshape layer receives the feature data output from feature extraction module 1 and performs spatial dimension transformation on the feature data, making it more suitable for convolution operations. For example, it transforms 1x1024x768 feature data into 1x768x32x32 feature data. The feature data, after being transformed by the Reshape layer, is input to the first convolutional layer. The first convolutional layer extracts features from the data in ascending order of depth, mapping the number of channels to a preset system hierarchy. For example, it extracts features according to {96, 192, 384, 768} (block1→96, block2→192, block3→384, block4→768), resulting in channel-transformed feature data. The feature data from the first convolutional layer is then input to the second convolutional layer, which adjusts the feature data to a uniform format. For example, Feature Recon1 processes the feature data output from block1 through the first convolutional layer to obtain 1x96x32x32 feature data, and then processes this 1x96x32x32 feature data through the second convolutional layer to obtain 1x128x32x32 feature data. Feature Recon4 processes the feature data output from block4 through the first convolutional layer to obtain 1x768x32x32 feature data. The 1x768x32x32 feature data is then processed through the second convolutional layer to obtain 1x128x32x32 feature data. Feature Recon2 outputs 1×128×32×32 feature data; Feature Recon3 outputs 1×128×64×64 feature data.
[0059] The sampling unit comprises three sampling sub-units. The first sampling sub-unit upsamples the feature data output by Feature Recon1; specifically, it performs a deconvolution on the feature data output by Feature Recon1, upsampling by a factor of 4, transforming the 1x128x32x32 feature data into 1x128x128x128 feature data. The second sampling sub-unit upsamples the feature data output by Feature Recon2; specifically, it performs a deconvolution on the feature data output by Feature Recon2, upsampling by a factor of 2, transforming the 1x128x32x32 feature data into 1x128x64x64 feature data. The third sampling sub-unit downsamples the feature data output by Feature Recon4; specifically, it performs a downsampling on the feature data output by Feature Recon4, transforming the 1x128x32x32 feature data into 1x128x16x16 feature data. No processing is performed on the feature data output by Feature Recon3.
[0060] The feature fusion unit comprises four fusion sub-units. The first fusion sub-unit receives the feature data output from the third sampling sub-unit and transmits it to the second fusion sub-unit. The second fusion sub-unit receives the feature data output from Feature Recon3 and fuses it with the feature data transmitted from the first fusion sub-unit; it then transmits the fused feature data to the third fusion sub-unit. The third fusion sub-unit receives the feature data output from the second sampling sub-unit and fuses it with the feature data transmitted from the second sampling sub-unit; it then transmits the fused feature data to the fourth fusion sub-unit. The fourth fusion sub-unit receives the feature data output from the first sampling sub-unit and fuses it with the feature data output from the third fusion sub-unit; it then transmits the fused feature data to the result output unit.
[0061] Each fusion subunit includes a fusion layer (Conv Unit), an upsampling layer (upsample), and a dimension-preserving layer (Project). The fusion layer consists of two convolutional layers, and the dimension-preserving layer consists of one convolutional layer. The fusion layer is used to fuse the received feature data. The upsampling layer is used to upsample the fused feature data. The dimension-preserving layer is used to preserve the dimensions of the upsampled feature data.
[0062] The output unit (Seg Head) consists of a convolutional layer that outputs the final image segmentation result, which may include the segmented image and the category of the segmented image.
[0063] The loss function of an image segmentation model can be composed of cross-entropy loss (CE_Loss) and damage function (DICE_Loss), with the total loss being CE_Loss + DICE_loss.
[0064] The following combination Figure 1 The image segmentation method of the embodiments of this application will be described in detail.
[0065] Figure 2 A schematic flowchart of the image segmentation method provided in this application is shown, with reference to... Figure 2 The optimization method is described in detail below:
[0066] S101, Obtain a first sample set and a first image segmentation model, wherein the first image segmentation model includes a feature extraction module and a data decoding module, and the output data of the feature extraction module is the input data of the data decoding module.
[0067] In this embodiment, the first sample set can be obtained from the total sample set. For example, 20% of the samples in the total sample set can be used as the first sample set. The total sample set can be generated based on the actual captured images.
[0068] The first image segmentation model is the initial model, which is a model that has not undergone any training. The first image segmentation model can be as follows: Figure 1 The model in the text.
[0069] S102, freeze the parameters in the feature extraction module of the first image segmentation model, train the first image segmentation model using the first sample set, update the parameters in the data decoding module of the first image segmentation model, and obtain the second image segmentation model.
[0070] In this embodiment, when training the first image segmentation model using the first sample set, after each training iteration, the data decoding module is updated based on the output of the first image segmentation model, while the parameters in the feature extraction module are not updated. This keeps the parameters in the feature extraction module unchanged, achieving the goal of training the data decoding module first and ensuring its better initialization. Furthermore, if the feature extraction module and the data decoding module are trained simultaneously, the first image segmentation model will be difficult and inefficient to train, and the trained model will not achieve optimal results. Therefore, using a small number of samples to train the data decoding module in the first image segmentation model simplifies the training process and facilitates subsequent training.
[0071] S103, obtain the second sample set and the third image segmentation model. The number of samples in the second sample set is greater than that in the first sample set. The third image segmentation model is obtained by constructing the feature extraction module in the second image segmentation model into a feature extraction module with a LoRA branch.
[0072] In this embodiment, the second sample set can be obtained from the total sample set. For example, 80% of the samples in the total sample set are used as the second sample set, or 60% of the samples in the total sample set are used as the second sample set.
[0073] Improve the second image segmentation model by constructing the feature extraction module into a feature extraction module with a LoRA branch. For example, the feature extraction module includes an attention mechanism, and the attention mechanism is constructed into an attention mechanism with a LoRA branch. If the feature extraction module is a DINO network module and there are 12 attention mechanisms in the DINO network module, at least one attention mechanism is provided with a LoRA branch.
[0074] Specifically, referring to Figure 3 the LoRA transformation mechanism diagram shown, LoRA injects two low-rank matrices A (dimension reduction) and B (dimension increase) into linear layers (such as fully connected layers, attention modules) by freezing the original model parameters. The middle layer dimension is r, and the product of A and B, ΔW = AB, approximates the parameter update amount. The original weight W remains unchanged during training, and ΔW is added to W during inference. When in use, the dimension of the trainable layer is the same as that of the pre-trained model layer, which is d. First, the dimension d is reduced to r through a fully connected layer, and then mapped back to the d dimension from r through a fully connected layer, where r << d, and r is the rank of the matrix. In this way, the matrix calculation changes from d * d to d * r + r * d, and the number of parameters is reduced significantly.
[0075] When the feature extraction module is a DINO network module, perform LoRA improvement on QKV in the attention mechanism in the DINO network. Here, Q is Query, K is Key, and V is Value, which are the core components of the self-attention mechanism.
[0076] As an example, for Q, Figure 3 the left branch in [[ ]] is the original frozen network, that is, the original network. The right branch introduces the A and B parameters for learning. The fully connected layer parameters of A are (d, r). The fully connected parameters of B are (r, d). The parameters of the right branch are learnable. Finally, the output h = Wx + ΔWx = Wx + BAx, where x is the input feature vector (feature data).
[0077] S104, the third image segmentation model is trained using the second sample set to obtain the target image segmentation model.
[0078] In this embodiment, after obtaining the third image segmentation model with the LoRA branch, the third image segmentation model is trained using the second sample set. Specifically, the original parameters of the feature extraction module in the third image segmentation model are frozen, the third image segmentation model is trained using the second sample set, and the parameters in the LoRA branch and the parameters of the data decoding module in the third image segmentation model are updated to obtain the target image segmentation model.
[0079] Since the parameters in the LoRA branch are low-rank parameters, training the parameters in the LoRA branch is simpler and faster than training the original parameters in the feature extraction module. Therefore, training a third image segmentation model with a LoRA branch can make training faster and simpler.
[0080] S105, the target image to be segmented is segmented using the target image segmentation model to obtain a segmented image.
[0081] In this embodiment, the segmented images output by the target image segmentation model can also be labeled with the category of the segmented images.
[0082] In this embodiment, after acquiring the initial image captured by the camera, image processing can be performed on the initial image to obtain the target image. Image processing may include normalization, resizing, and horizontal mirroring. Normalization involves scaling pixel values to a fixed range.
[0083] In this application, a first sample set is first obtained, and the first image segmentation model is trained using the first sample set to update the parameters of the data decoding module in the first image segmentation model. However, the parameters of the feature extraction module in the first image segmentation module are not updated, resulting in a second image segmentation model. Since the feature extraction module is more difficult to train than the data decoding module, this application updates the parameters in the data decoding module first. On the one hand, this reduces the amount of parameter updates, making the training of the first image segmentation model simpler and faster; on the other hand, it allows the data decoding module to have better initial parameters, facilitating subsequent training of the model.
[0084] After obtaining the second image segmentation model, it is improved by constructing a feature extraction module with a LoRA branch, resulting in a third image segmentation model. The third image segmentation model is trained using the second sample set to obtain the target image segmentation model. Finally, the trained target image segmentation model is used to segment the target image to be segmented, yielding the segmented image. The inclusion of a LoRA branch in the third image segmentation model accelerates its training. Since the parameters of the data decoding module have already been updated and optimized, training the third image segmentation model further improves its overall training speed, reduces training difficulty, and enables rapid convergence, leading to a quickly accurate target image segmentation model.
[0085] During the training of the image segmentation model, since the parameters in the feature extraction module are frozen, to avoid inaccurate feature extraction due to unchanged parameters, after training the first image segmentation model using the first sample set, a small number of samples can be used to train the entire second image segmentation model. This updates the parameters of the feature extraction module and the data decoding module in the second image segmentation model, optimizing the parameters in the feature extraction module and improving the accuracy of the image segmentation model. Furthermore, because the number of samples used in training the second image segmentation model is small, training it will not take much time.
[0086] Specifically, such as Figure 4 As shown, after step S102 above, the method of this application may further include:
[0087] S201, Obtain a third sample set, wherein the number of samples in the third sample set is less than the number of samples in the second sample set.
[0088] In this real-time example, the third sample set can be obtained from the total sample set, for example, 20% of the samples in the total sample set can be used as the third sample set.
[0089] S202, the second image segmentation model is trained using the third sample set, and the parameters of the feature extraction module and the data decoding module in the second image segmentation model are updated to obtain the fourth image segmentation model.
[0090] In this embodiment, when training the second image segmentation model using the third sample set, the parameters in the entire image segmentation model are updated according to the output results of the second image segmentation model at each training session, so that both the feature extraction module and the data decoding module in the second image segmentation model can be trained, thereby improving the accuracy of the entire model.
[0091] Accordingly, after the second and fourth image segmentation models, the feature extraction module in the fourth image segmentation model is constructed into a feature extraction module with LoRA branch to obtain the third image segmentation model. Then, the third image segmentation model is trained to make the target image segmentation model more accurate.
[0092] After obtaining the trained target image segmentation model, the target image is segmented using the target image segmentation model.
[0093] Specifically, methods for segmenting target images using target image segmentation models include:
[0094] S301, Obtain the target image to be segmented.
[0095] S302, the target image is input into the trained target image segmentation model, and the DINO network module in the target image segmentation model is used to extract features from the target image to obtain the initial feature data of the target image. The DINO network module in the target image segmentation model has a LoRA branch.
[0096] S303, the data decoding module in the target image segmentation model is used to decode each of the initial feature data to obtain the segmented image of the target image.
[0097] Specifically, refer to Figure 1 The data decoding module's process of decoding the initial feature data includes:
[0098] The initial feature data are recombined using the data decoding module in the target image segmentation model to obtain first candidate data corresponding to each initial feature data, and each first candidate data has the same format; the first candidate data is sampled to obtain second candidate data corresponding to each first candidate data; the second candidate data is fused to obtain target feature data; and a segmented image of the target image is generated based on the target feature data.
[0099] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0100] Corresponding to the image segmentation method described in the above embodiments, Figure 5 A structural block diagram of an image segmentation apparatus provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.
[0101] Reference Figure 5 The device 300 may include: a first sample acquisition module 310, a first training module 320, a second sample acquisition module 330, a second training module 340, and an image segmentation module 350.
[0102] The first sample acquisition module 310 is used to acquire a first sample set and a first image segmentation model. The first image segmentation model includes a feature extraction module and a data decoding module. The output data of the feature extraction module is the input data of the data decoding module.
[0103] The first training module 320 is used to freeze the parameters in the feature extraction module of the first image segmentation model, train the first image segmentation model using the first sample set, update the parameters in the data decoding module of the first image segmentation model, and obtain the second image segmentation model.
[0104] The second sample acquisition module 330 is used to acquire a second sample set and a third image segmentation model, wherein the number of samples in the second sample set is greater than the number of samples in the first sample set, and the third image segmentation model is a model obtained by constructing the feature extraction module in the second image segmentation model into a feature extraction module with a LoRA branch.
[0105] The second training module 340 is used to train the third image segmentation model using the second sample set to obtain the target image segmentation model.
[0106] The image segmentation module 350 is used to segment the target image to be segmented using the target image segmentation model to obtain a segmented image.
[0107] In one possible implementation, the following components are also connected to the first training module 320:
[0108] The third sample acquisition module acquires a third sample set, wherein the number of samples in the third sample set is less than the number of samples in the second sample set;
[0109] The third training module uses the third sample set to train the second image segmentation model, updates the parameters of the feature extraction module and the data decoding module in the second image segmentation model, and obtains the fourth image segmentation model.
[0110] Accordingly, the third image segmentation model is the model obtained by constructing the feature extraction module in the fourth image segmentation model into a feature extraction module with a LoRA branch.
[0111] In one possible implementation, the feature extraction module is a DINO network module.
[0112] In one possible implementation, the DINO network consists of multiple attention mechanisms, and at least one of the attention mechanisms in the DINO network module of the third image segmentation model has a LoRA branch.
[0113] In one possible implementation, the image segmentation module 350 can specifically be used for:
[0114] Obtain the target image to be segmented;
[0115] The target image is input into the trained target image segmentation model, and the DINO network module in the target image segmentation model is used to extract features from the target image to obtain the initial feature data of the target image. The DINO network module in the target image segmentation model has a LoRA branch.
[0116] The initial feature data are decoded using the data decoding module in the target image segmentation model to obtain a segmented image of the target image.
[0117] In one possible implementation, the image segmentation module 350 can specifically be used for:
[0118] The data decoding module in the target image segmentation model is used to reconstruct the features of each initial feature data to obtain the first candidate data corresponding to each initial feature data, and each first candidate data has the same format.
[0119] The first candidate data is sampled to obtain the second candidate data corresponding to each of the first candidate data.
[0120] Feature fusion is performed on each of the second candidate data to obtain the target feature data;
[0121] A segmented image of the target image is generated based on the target feature data.
[0122] In one possible implementation, the second training module 340 can specifically be used for:
[0123] The original parameters of the feature extraction module in the third image segmentation model are frozen. The third image segmentation model is trained using the second sample set. The parameters in the LoRA branch and the parameters of the data decoding module in the third image segmentation model are updated to obtain the target image segmentation model.
[0124] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.
[0125] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0126] This application also provides a terminal device, see [link to relevant documentation] Figure 6 The terminal device 400 may include: at least one processor 410, a memory 420, and a computer program stored in the memory 420 and executable on the at least one processor 410. When the processor 410 executes the computer program, it implements the steps in any of the above method embodiments, for example... Figure 2 Steps S101 to S105 in the illustrated embodiment. Alternatively, when the processor 410 executes the computer program, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 5 The functions of the first sample acquisition module 310 to the image segmentation module 350 are shown.
[0127] For example, a computer program may be divided into one or more modules / units, one or more of which are stored in memory 420 and executed by processor 410 to complete this application. The one or more modules / units may be a series of computer program segments capable of performing a specific function, which are used to describe the execution process of the computer program in terminal device 400.
[0128] Those skilled in the art will understand that Figure 6 This is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, buses, etc.
[0129] The processor 410 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.
[0130] The memory 420 can be an internal storage unit of the terminal device or an external storage device, such as a plug-in hard drive, a smart media card (SMC), a secure digital card (SD), or a flash card. The memory 420 is used to store the computer program and other programs and data required by the terminal device. The memory 420 can also be used to temporarily store data that has been output or will be output.
[0131] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0132] The image segmentation method provided in this application can be applied to terminal devices such as computers, tablets, laptops, netbooks, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of terminal device.
[0133] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0134] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0135] In the embodiments provided in this application, it should be understood that the disclosed terminal devices, apparatuses, and methods can be implemented in other ways. For example, the terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, apparatuses, or units, and may be electrical, mechanical, or other forms.
[0136] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0137] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0138] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by one or more processors, it can implement the steps of the various method embodiments described above.
[0139] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by one or more processors, it can implement the steps of the various method embodiments described above.
[0140] Similarly, as a computer program product, when the computer program product is run on a terminal device, it enables the terminal device to implement the steps in the above-described method embodiments.
[0141] The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.
[0142] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. An image segmentation method, characterized in that, include: Obtain a first sample set and a first image segmentation model, wherein the first image segmentation model includes a feature extraction module and a data decoding module, and the output data of the feature extraction module is the input data of the data decoding module; Freeze the parameters in the feature extraction module of the first image segmentation model, train the first image segmentation model using the first sample set, update the parameters in the data decoding module of the first image segmentation model, and obtain the second image segmentation model; Obtain a second sample set and a third image segmentation model, wherein the number of samples in the second sample set is greater than the number of samples in the first sample set, and the third image segmentation model is a model obtained by constructing the feature extraction module in the second image segmentation model into a feature extraction module with a LoRA branch; The third image segmentation model is trained using the second sample set to obtain the target image segmentation model; The target image to be segmented is segmented using the target image segmentation model to obtain a segmented image.
2. The image segmentation method as described in claim 1, characterized in that, After freezing the parameters in the feature extraction module of the first image segmentation model, training the first image segmentation model using the first sample set, updating the parameters in the data decoding module of the first image segmentation model, and obtaining the second image segmentation model, the method further includes: Obtain a third sample set, wherein the number of samples in the third sample set is less than the number of samples in the second sample set; The second image segmentation model is trained using the third sample set, and the parameters of the feature extraction module and the data decoding module in the second image segmentation model are updated to obtain the fourth image segmentation model. Accordingly, the third image segmentation model is the model obtained by constructing the feature extraction module in the fourth image segmentation model into a feature extraction module with a LoRA branch.
3. The image segmentation method as described in claim 1, characterized in that, The feature extraction module is a DINO network module.
4. The image segmentation method as described in claim 3, characterized in that, The DINO network consists of multiple attention mechanisms, and at least one of the attention mechanisms in the DINO network module of the third image segmentation model has a LoRA branch.
5. The image segmentation method as described in claim 4, characterized in that, The step of segmenting the target image to be segmented using the target image segmentation model to obtain a segmented image includes: Obtain the target image to be segmented; The target image is input into the trained target image segmentation model, and the DINO network module in the target image segmentation model is used to extract features from the target image to obtain the initial feature data of the target image. The DINO network module in the target image segmentation model has a LoRA branch. The initial feature data are decoded using the data decoding module in the target image segmentation model to obtain a segmented image of the target image.
6. The image segmentation method as described in claim 5, characterized in that, The step of decoding each of the initial feature data using the data decoding module in the target image segmentation model to obtain a segmented image of the target image includes: The data decoding module in the target image segmentation model is used to reconstruct the features of each initial feature data to obtain the first candidate data corresponding to each initial feature data, and each first candidate data has the same format. The first candidate data is sampled to obtain the second candidate data corresponding to each of the first candidate data. Feature fusion is performed on each of the second candidate data to obtain the target feature data; A segmented image of the target image is generated based on the target feature data.
7. The image segmentation method according to any one of claims 1 to 6, characterized in that, The step of training the third image segmentation model using the second sample set to obtain the target image segmentation model includes: The original parameters of the feature extraction module in the third image segmentation model are frozen. The third image segmentation model is trained using the second sample set. The parameters in the LoRA branch and the parameters of the data decoding module in the third image segmentation model are updated to obtain the target image segmentation model.
8. An image segmentation apparatus, characterized in that, include: The first sample acquisition module is used to acquire a first sample set and a first image segmentation model, wherein the first image segmentation model includes a feature extraction module and a data decoding module, and the output data of the feature extraction module is the input data of the data decoding module; The first training module is used to freeze the parameters in the feature extraction module of the first image segmentation model, train the first image segmentation model using the first sample set, update the parameters in the data decoding module of the first image segmentation model, and obtain the second image segmentation model. The second sample acquisition module is used to acquire a second sample set and a third image segmentation model, wherein the number of samples in the second sample set is greater than the number of samples in the first sample set, and the third image segmentation model is a model obtained by constructing the feature extraction module in the second image segmentation model into a feature extraction module with a LoRA branch; The second training module is used to train the third image segmentation model using the second sample set to obtain the target image segmentation model. The image segmentation module is used to segment the target image to be segmented using the target image segmentation model to obtain a segmented image.
9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image segmentation method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the image segmentation method as described in any one of claims 1 to 7.