Sample image set determination method, apparatus, device, and computer medium

Abstract

The application provides a sample image set determination method and device, equipment and a computer medium. The method comprises the following steps: labeling a first region occupied by a first key object in a first to-be-processed image to obtain a first labeled image corresponding to the first to-be-processed image; cutting the first labeled image based on a preset cutting mode to obtain a cut image; obtaining a second key object corresponding to the first key object from the cut image; and determining a sample image set used for training a target segmentation model based on a plurality of preset background images and the second key object, wherein the target segmentation model is used for labeling a second region occupied by a second key object in a second to-be-processed image, and can improve the segmentation accuracy of a segmentation model trained based on sample data.

Description

Technical Field

[0001] This application belongs to the field of image recognition technology, and in particular relates to a method, apparatus, device and computer medium for determining a sample image set. Background Technology

[0002] In VR headset applications, there are many scenarios that require semantic segmentation of images. For example, in video conferencing, semantic segmentation algorithms can separate people from the background, enabling background replacement. In virtual reality scenarios, semantic segmentation algorithms can be used to separate arms from the background, allowing users to interact with virtual objects using their hands, enhancing the user experience.

[0003] Current advanced semantic segmentation algorithms utilize neural networks. Achieving good segmentation results and accuracy requires collecting large amounts of data and performing corresponding mask annotations to obtain sample data. This sample data is then used to segment key objects in the image, forming a segmentation model. This annotated sample data is then used for training and learning. To accelerate the annotation process, a common workflow involves pre-annotating this data using a large semantic segmentation model, followed by manual adjustments to areas with poor pre-annotation. Related technologies also employ interactive annotation models, such as the Segment Anything model, to accelerate pre-annotation. This involves manually providing multiple positive and negative sample points, ultimately adjusting the mask provided by the Segment Anything model to the desired shape and position to obtain sample data. However, in these technologies, the sample data used to train the segmentation model often suffers from poor edge accuracy in key regions, leading to low segmentation accuracy of the model trained on this generated sample data. Summary of the Invention

[0004] This application provides an implementation scheme different from related technologies to solve the technical problem in related technologies where the segmentation accuracy of the segmentation model trained based on the generated sample data is low due to the poor accuracy of the edges of key regions in the sample data used to train the segmentation model.

[0005] In a first aspect, this application provides a method for determining a sample image set, comprising: labeling a first region occupied by a first key object in a first image to be processed to obtain a first labeled image corresponding to the first image to be processed; cropping the first labeled image based on a preset cropping method to obtain a cropped image; extracting a second key object corresponding to the first key object from the cropped image; and determining a sample image set for training a target segmentation model based on multiple preset background images and the second key object, wherein the target segmentation model is used to label a second region occupied by a second key object in a second image to be processed.

[0006] Secondly, this application provides a sample image set determination apparatus, comprising:

[0007] An annotation unit is used to annotate the first region occupied by the first key object in the first image to be processed, so as to obtain the first annotated image corresponding to the first image to be processed.

[0008] The cropping unit is used to crop the first annotated image based on a preset cropping method to obtain a cropped image;

[0009] The acquisition unit is used to acquire the second key object corresponding to the first key object from the cropped image;

[0010] The determining unit is used to determine a set of sample images for training a target segmentation model based on multiple preset background images and the second key object. The target segmentation model is used to label the second region occupied by the second key object in the second image to be processed.

[0011] Thirdly, this application provides an electronic device, comprising:

[0012] Processor; and

[0013] Memory for storing the executable instructions of the processor;

[0014] The processor is configured to execute the first aspect or any of the possible implementations of the first aspect by executing the executable instructions.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods in the first aspect or any of the possible implementations of the first aspect.

[0016] The present application provides a scheme for annotating a first region occupied by a first key object in a first image to be processed, thereby obtaining a first annotated image corresponding to the first image to be processed; cropping the first annotated image based on a preset cropping method, thereby obtaining a cropped image; extracting a second key object corresponding to the first key object from the cropped image; and determining a sample image set for training a target segmentation model based on multiple preset background images and the second key object. The target segmentation model is used to annotate a second region occupied by a second key object in a second image to be processed. This scheme can use a preset cropping method to remove image edges that are easily misjudged, thereby improving the effectiveness and accuracy of the determined sample data, and further improving the segmentation accuracy of the segmentation model trained based on the sample data. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0018] Figure 1a A schematic flowchart illustrating a method for determining a sample image set according to an embodiment of this application;

[0019] Figure 1b A schematic diagram of a first image to be processed provided in an embodiment of this application;

[0020] Figure 1c This is a schematic diagram of the structure of an initial interactive annotation model provided in an embodiment of this application;

[0021] Figure 1d This is a schematic diagram of the structure of a target interactive annotation model provided in an embodiment of this application;

[0022] Figure 1e A schematic flowchart illustrating a method for determining a sample image set according to an embodiment of this application;

[0023] Figure 2 This is a schematic diagram of the structure of a sample image set determination device provided in an embodiment of this application;

[0024] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] The embodiments of this application are described in detail below, with examples of these embodiments illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0026] The terms "first" and "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the present application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] First, some terms used in the embodiments of this application will be explained below to facilitate understanding by those skilled in the art.

[0028] Semantic segmentation assigns each pixel in an image to a corresponding category. In object detection, bounding boxes are traditionally used to label and predict objects in an image. Semantic segmentation focuses on dividing an image into regions belonging to different semantic categories. Unlike object detection, semantic segmentation can identify and understand the content of each pixel in an image: its semantic region labeling and prediction are pixel-level.

[0029] A mask is an element that obscures or covers part of an image, while displaying a specific area of ​​the image. It is like a window. A mask exists as a separate layer and is usually positioned with one layer covering the other.

[0030] The design principles of the MobileSAM model mainly include the following aspects:

[0031] Lightweight Model: To adapt to the resource constraints of mobile devices, the MobileSAM model adopts a lightweight neural network architecture, reducing the model size through pruning, quantization, and other compression techniques to make it suitable for deployment on mobile devices.

[0032] High performance: Despite optimizations, the MobileSAM model still delivers segmentation accuracy comparable to the original SAM model. This is thanks to efficient feature extraction, cross-modal attention modules, and decoder design.

[0033] Cross-platform compatibility: The MobileSAM model can run on multiple mobile operating systems (such as Android and iOS) and supports a wide range of device types. This is thanks to the model's design and optimization, which makes it cross-platform compatible.

[0034] Knowledge distillation (KD) is a classic model compression method. Its core idea is to improve the performance of a student model without changing its structure by guiding a lightweight student model to "imitate" a more performant and structurally complex teacher model (or an ensemble of multiple models).

[0035] ViT, short for Vision Transformer, is a vision processing model based on the Transformer architecture. Traditional computer vision tasks typically use convolutional neural networks (CNNs) to extract features from images. ViT aims to apply the Transformer model to computer vision tasks, capturing long-range dependencies in images through a global attention mechanism.

[0036] The ViT-H model is a visual transformer model that has demonstrated powerful performance on image classification tasks. In particular, when trained on large-scale datasets, the ViT-H model can achieve very high accuracy.

[0037] TinyViT is a novel small transformer that transfers knowledge from large models to small models through knowledge distillation, leveraging the advantages of large-scale datasets. This approach achieves performance comparable to or even surpassing that of large models while maintaining high efficiency, particularly excelling in image classification and transfer learning for downstream tasks. During pre-training, a fast and scalable distillation strategy is implemented by storing sparse soft labels and data augmentation information, addressing the efficiency and cost issues of traditional methods.

[0038] The SAM model is an open-source image segmentation model that has achieved a new breakthrough in the field of computer vision, capable of segmenting any object in any image without any annotations.

[0039] Image embedding in machine learning is a method that transforms image data into continuous, low-dimensional vector representations. These vector representations are typically used for subsequent machine learning tasks such as classification, clustering, and retrieval.

[0040] Model fine-tuning involves initializing your model weights using a pre-trained model obtained from large datasets, thereby improving accuracy.

[0041] In VR headset applications, there are many scenarios that require semantic segmentation of images. For example, in video conferencing, semantic segmentation algorithms can separate people from the background, enabling background replacement. In virtual reality scenarios, semantic segmentation algorithms can be used to separate arms from the background, allowing users to interact with virtual objects using their hands, enhancing the user experience.

[0042] Current advanced semantic segmentation algorithms utilize neural networks. Achieving good segmentation results and accuracy requires collecting large amounts of data and performing corresponding mask annotations to obtain sample data. This sample data is then used to segment key objects in the image, forming a segmentation model. This annotated sample data is then used for training and learning. To accelerate the annotation process, a common workflow involves pre-annotating this data using a large semantic segmentation model, followed by manual adjustments to areas with poor pre-annotation. Related technologies also employ interactive annotation models, such as the Segment Anything model, to accelerate pre-annotation. This involves manually providing multiple positive and negative sample points, ultimately adjusting the mask provided by the Segment Anything model to the desired shape and position to obtain sample data. However, in these technologies, the sample data used to train the segmentation model often suffers from poor edge accuracy in key regions, leading to low segmentation accuracy of the model trained on this generated sample data.

[0043] Furthermore, to obtain a sample dataset for training the segmentation model and quickly develop a more robust model, simply collecting and labeling data is insufficient; data augmentation techniques are needed to enrich the diversity of the sample data. Existing data augmentation methods include adjusting image color and contrast, and horizontal and vertical flipping. However, these methods are not entirely suitable for image data acquired by VR glasses.

[0044] Currently, achieving good segmentation results in VR scenarios requires collecting a large amount of sample data using VR glasses, followed by algorithmic pre-annotation and meticulous manual annotation. This leads to the following problems:

[0045] 1) VR glasses often use wide-angle cameras, resulting in significant distortion. Existing semantic segmentation models, however, are based on data captured by ordinary RGB cameras and do not exhibit this distortion. Therefore, directly using existing semantic segmentation models for pre-annotation leads to almost unusable results. Most of the pre-annotated results require manual re-annotation, completely failing to improve annotation efficiency. Therefore, this solution proposes an efficient annotation process for VR scene data, enabling iterative optimization between the semantic segmentation model and the data used for pre-annotation. On the other hand, while "de-distorting" the images captured by VR glasses and then annotating them with existing semantic segmentation models yields better results than directly annotating distorted images, de-distortion degrades image quality. In actual VR glasses usage scenarios, segmentation algorithms also use distorted images as input. Therefore, annotating on de-distorted images results in information loss, hindering the performance improvement of semantic segmentation algorithms in VR applications.

[0046] 2) The typical annotation process uses interactive annotation, where positive / negative sample points are manually provided to adjust the mask to the desired shape. The large semantic segmentation model Segment Anything can be used for interactive annotation; however, Segment Anything has a large number of parameters, requiring fine-tuning to adapt it to VR scene data annotation. However, its large number of parameters makes fine-tuning the entire model quite difficult.

[0047] 3) To cover more usage scenarios, it's necessary to collect and annotate arm images from different scenes and backgrounds for model training. However, a larger data volume means greater annotation pressure, significantly increasing annotation costs and hindering rapid performance iteration of the segmentation algorithm. Furthermore, the data augmentation methods used in traditional semantic segmentation model training are not entirely applicable to segmentation algorithm training in VR scenarios. For example, in arm segmentation tasks, horizontally or vertically flipping or rotating images will cause inconsistencies between the training data distribution and the data distribution used in practice, leading to model performance degradation in real-world applications.

[0048] Therefore, this application proposes an efficient annotation process for VR scene data, enabling the semantic segmentation model and data used for pre-annotation to iteratively optimize each other. An efficient fine-tuning and deployment scheme is also proposed.

[0049] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0050] The solution in this application is applicable to VR scenarios. The sample image set determined by the solution in this application can be used to train a segmentation model that can label key regions in the image to be processed. The segmentation model can label key regions by outputting images with masks. Specifically, the image region within the mask is the key region.

[0051] Figure 1a A flowchart illustrating a method for determining a sample image set, provided as an exemplary embodiment of this application, is applicable to any electronic device. The method includes at least the following steps S201-S204:

[0052] S201. Mark the first region occupied by the first key object in the first image to be processed to obtain the first marked image corresponding to the first image to be processed.

[0053] Optionally, the first image to be processed may include a first key object and background information other than the first key object. The first key object in the first image to be processed refers to the object whose posture, action, movement behavior, or movement trajectory needs to be considered.

[0054] The first region refers to the image area occupied by the first key object.

[0055] In a VR scenario, the aforementioned first key object can refer to user limb objects such as the user's arm or fingers. The first image to be processed can refer to an image taken from a first-person perspective in the VR scenario; for example, the first image to be processed can be seen in [reference needed]. Figure 1b As shown.

[0056] The aforementioned annotation of the first region can be achieved by generating a mask that covers the first key object to annotate the first region.

[0057] S202. Based on a preset cropping method, the first annotated image is cropped to obtain a cropped image;

[0058] In the aforementioned S202, the first annotated image is cropped based on a preset cropping method to obtain a cropped image. This can refer to cropping the first annotated image based on a preset cropping area and / or cropping shape to obtain a cropped image.

[0059] Optionally, the aforementioned cutting area and / or cutting shape can be set by relevant personnel, and the specific shape can be a circle, an ellipse, etc.

[0060] If there is a dark background at the edge of the image after the first annotation, the image area that belongs to the first key object will blend into the dark background, and edge annotation errors are likely to occur in the image after the first annotation. The area that is prone to annotation errors can be cropped out by a preset cropping method.

[0061] The introduction of the preset cropping method in this application can remove dark edges in the first image to be processed, thereby improving the accuracy of the target segmentation model when the second key object corresponding to the first key object is used as a sample image. The second key object corresponding to the first key object refers to the first key object in the cropped image.

[0062] S203. Obtain the second key object corresponding to the first key object from the cropped image;

[0063] Specifically, the second key object can be extracted from the cropped image using image cutout technology.

[0064] S204. Based on multiple preset background images and the second key object, a sample image set is determined for training a target segmentation model. The target segmentation model is used to label the second region occupied by the second key object in the second image to be processed.

[0065] In some optional embodiments of this application, the annotation of the first region occupied by the first key object in the first image to be processed in the aforementioned S201 includes:

[0066] In response to the number of processed images reaching a preset threshold, the first region occupied by the first key object in the first image to be processed is labeled using the initial segmentation model corresponding to the target segmentation model. Here, the target segmentation model is the trained initial segmentation model. The aforementioned preset threshold can be set by relevant personnel.

[0067] In some optional embodiments of this application, the aforementioned step S201, which involves marking the first region occupied by the first key object in the first image to be processed, further includes:

[0068] In response to the fact that the number of processed images has not reached a preset threshold, the first region occupied by the first key object in the first image to be processed is annotated using the target interactive annotation model.

[0069] The target interactive annotation model can annotate the first region occupied by the first key object in the first image to be processed through interaction with humans. When the mask annotated by the target interactive annotation model is inaccurate, the mask can be adjusted based on the positive and negative sample points annotated by humans.

[0070] In some optional embodiments of this application, the initial interactive annotation model includes: an initial image encoder, an image embedding unit, and an initial image decoder. The method further includes the following S1-S3:

[0071] S1. Obtain the second sample image set;

[0072] Optionally, the second sample image set may be partially or entirely consistent with the first sample image set, or it may be completely different.

[0073] S2. The initial interactive annotation model is trained based on the second sample image set to obtain a trained initial interactive annotation model, wherein the parameters of the initial image decoder are adjustable parameters during the training process of the initial interactive annotation model.

[0074] S3. Determine the target interactive annotation model based on the initial interactive annotation model that has been trained.

[0075] In some optional embodiments of this application, the target interaction annotation model includes: a target image encoder, the image embedding unit, and a target image decoder. In the aforementioned S3, determining the target interaction annotation model based on the trained initial interaction annotation model includes: replacing the trained initial image encoder in the trained initial interaction annotation model with the target image encoder to obtain the target interaction annotation model. The initial image encoder in the initial interaction annotation model before training is an encoder obtained by distilling the target image encoder.

[0076] Optionally, the aforementioned initial interactive annotation model is the segmentanything (SAM) model, which currently boasts the largest number of parameters and the best performance. However, the SAM model has not been trained on specific VR tasks (such as arm segmentation) and cannot be directly used in the annotation scenario of this solution. Therefore, a method for fine-tuning the SAM model was proposed. Since the SAM model has a large number of parameters, directly fine-tuning the entire network would lead to a decrease in overall performance and a long training process. This solution uses the Mobilesam architecture for fine-tuning, and during the fine-tuning process, only the parameters of the initial image decoder are changed, while other parts are frozen.

[0077] For details, please refer to Figure 1c As shown, during the training of the initial interactive annotation model, Figure 1c The parameters in the initial image decoder are adjustable. The aforementioned target image encoder can refer to the SAM encoder ViT-H, and the aforementioned initial image encoder can refer to the encoder Tiny ViT, which has fewer parameters obtained by distilling ViT-H using features. This ensures efficient inference speed. Furthermore... Figure 1d A schematic diagram of the target interactive annotation model.

[0078] In this approach, the initial interactive annotation model is trained by setting only the parameters of the initial image decoder to be adjustable. This allows training to be completed without requiring high-performance graphics card resources, and the training speed is relatively fast and the training efficiency is high.

[0079] In some optional embodiments of this application, in S204, based on multiple preset background images and the second key object, a sample image set for training the target segmentation model is determined, including:

[0080] For each of the plurality of preset background images;

[0081] The second key object is fused with the preset background image to obtain a sample image corresponding to the preset background image, thereby obtaining a sample image set for training the target segmentation model.

[0082] In some optional embodiments of this application, in S2042, the second key object is fused with the preset background image to obtain a sample image corresponding to the preset background image, including:

[0083] The second key object is overlaid onto the preset background image to obtain a sample image corresponding to the preset background image.

[0084] In some optional embodiments of this application, in S2042, the second key object is overlaid onto the preset background image to obtain a sample image corresponding to the preset background image, including:

[0085] Replace the image information in the area to be covered by the second key object in the preset background image with the second key object to obtain the sample image corresponding to the preset background image.

[0086] The aforementioned preset background images can be different. To address the issue of data diversity, this technical solution collects data under different backgrounds and then fuses this background data with the second key object to achieve data augmentation. Unlike other semantic segmentation tasks, VR scene segmentation tasks are from a first-person perspective. The categories to be segmented are mostly arms in the first-person view, other people and objects in the background, etc., which have no semantic connection with the background. Secondly, the number of categories to be segmented is small, often only two or three, and the task's objective is to separate them from the background. These two properties are necessary for changing the background. Other semantic segmentation tasks do not possess this property. For example, in autonomous driving segmentation tasks, the car category to be segmented is only meaningful if it is on the road. If, after changing the background, the car is no longer on the road, then the data is unrealistic, and the trained model's performance is poor. Another example is the classic semantic segmentation dataset ADE20k, which requires segmenting almost all objects in the scene. In this case, there is no clear distinction between foreground and background, and background changing is not possible.

[0087] In some optional embodiments of this application, the above method further includes performing line image enhancement on the first image to be processed, including color transformation, brightness transformation, etc.

[0088] Optionally, the aforementioned preset background image can be an image captured in a VR scene, or it can be an image that has undergone image enhancement processing.

[0089] The aforementioned initial image encoder and initial image decoder can be deployed on different devices. The initial image encoder can be deployed on a machine with a dedicated GPU, while the image embedding unit and initial image decoder can be deployed on another device. This ensures that the data processing speed of both the initial image encoder and initial image decoder is maintained. Using ViT-H directly as the encoder has two advantages: First, TinyViT is derived from the features of ViT-H through feature distillation. Since the features of these two encoders are similar, even if TinyViT is used as the encoder during fine-tuning, the larger ViT-H can be used directly during deployment. Second, ViT-H, as an encoder with a larger number of parameters, can provide better features to the decoder, improving the overall network segmentation performance. Furthermore, for each image, the encoder only infers once. Each subsequent interactive process of providing positive and negative sample points infers the mask decoder based on the obtained features (without needing to pass through the encoder again). Therefore, in practical use, using ViT-H as the encoder will not significantly impact the overall annotation speed.

[0090] Optionally, after obtaining the first sample image set, the method further includes: training the initial segmentation model based on the first sample image set to obtain a target segmentation model. After obtaining the target segmentation model, it can be used as a new initial segmentation model to process subsequent images to be processed.

[0091] Optionally, when the mask of the first image to be processed after the initial segmentation model needs to be adjusted, the mask can be further adjusted based on the target interactive annotation model.

[0092] Optionally, the aforementioned second image to be processed may refer to the next first image to be processed, or other images.

[0093] Optionally, the aforementioned first image to be processed can be an image from the first sample image set.

[0094] To further illustrate the scheme of this application, the following is combined with... Figure 1e Further explanation of the scheme in this application:

[0095] Determine whether the current image to be processed (such as the first image to be processed) is the image in the initial annotation stage. If so, annotate the image to be processed using the target interactive annotation model to obtain the annotated image. If not, annotate the image to be processed using the initial segmentation model and determine whether the image after annotation by the initial segmentation model needs mask adjustment. If so, further adjust the mask using the target interactive annotation model and perform edge cropping on the adjusted image. If not, directly perform edge cropping on the image after annotation by the initial segmentation model.

[0096] After edge cropping, the cropped image can be used for second key object matting and fused with various pure background images to obtain the first sample image set.

[0097] In the aforementioned interactive image annotation model, manual annotation only requires fine-tuning poorly labeled edges, eliminating the need for re-annotating a mask. This significantly improves annotation efficiency. Furthermore, the introduction of a pre-defined background image in this application rapidly enhances data richness, enabling the pre-annotation model to achieve accurate results even under complex backgrounds. Experiments have shown that the annotation efficiency using this method increases from 5 images / person / hour to 9.5 images / person / hour, nearly doubling the annotation speed.

[0098] The present application provides a scheme for annotating a first region occupied by a first key object in a first image to be processed, thereby obtaining a first annotated image corresponding to the first image to be processed; cropping the first annotated image based on a preset cropping method, thereby obtaining a cropped image; extracting a second key object corresponding to the first key object from the cropped image; and determining a sample image set for training a target segmentation model based on multiple preset background images and the second key object. The target segmentation model is used to annotate a second region occupied by a second key object in a second image to be processed. This scheme can use a preset cropping method to remove image edges that are easily misjudged, thereby improving the effectiveness and accuracy of the determined sample data, and further improving the segmentation accuracy of the segmentation model trained based on the sample data.

[0099] Figure 2 A schematic diagram of a data processing apparatus provided for an exemplary embodiment of this application, wherein the apparatus includes:

[0100] The annotation unit 21 is used to annotate the first region occupied by the first key object in the first image to be processed, so as to obtain the first annotated image corresponding to the first image to be processed.

[0101] The cropping unit 22 is used to crop the first annotated image based on a preset cropping method to obtain a cropped image;

[0102] The acquisition unit 23 is used to acquire the second key object corresponding to the first key object from the cropped image;

[0103] The determining unit 24 is used to determine a set of sample images for training a target segmentation model based on multiple preset background images and the second key object. The target segmentation model is used to label the second region occupied by the second key object in the second image to be processed.

[0104] In some optional embodiments of this application, when the aforementioned device is used to annotate the first region occupied by the first key object in the first image to be processed, it is specifically used for:

[0105] In response to the fact that the number of processed images has reached a preset threshold, the first region occupied by the first key object in the first image to be processed is labeled using the initial segmentation model corresponding to the target segmentation model.

[0106] In some optional embodiments of this application, when the aforementioned device is used to annotate the first region occupied by the first key object in the first image to be processed, it is specifically used for:

[0107] In response to the fact that the number of processed images has not reached a preset threshold, the first region occupied by the first key object in the first image to be processed is annotated using the target interactive annotation model.

[0108] In some optional embodiments of this application, the initial interactive annotation model includes: an initial image encoder, an image embedding unit, and an initial image decoder. The apparatus is further used for:

[0109] Obtain the second set of sample images;

[0110] The initial interactive annotation model is trained based on the second sample image set to obtain a trained initial interactive annotation model. During the training of the initial interactive annotation model, the parameters of the initial image decoder are adjustable parameters.

[0111] The target interactive annotation model is determined based on the initial interactive annotation model that has been trained.

[0112] In some optional embodiments of this application, the target interaction annotation model includes: a target image encoder, the image embedding unit, and a target image decoder. When the aforementioned device is used to determine the target interaction annotation model based on the trained initial interaction annotation model, it is specifically used to: replace the trained initial image encoder in the trained initial interaction annotation model with the target image encoder to obtain the target interaction annotation model, wherein, before the initial interaction annotation model is trained, the initial image encoder in the initial interaction annotation model is an encoder obtained by distilling the target image encoder.

[0113] In some optional embodiments of this application, when the aforementioned apparatus is used to determine a sample image set for training a target segmentation model based on multiple preset background images and the second key object, it is specifically used for:

[0114] For each of the plurality of preset background images;

[0115] The second key object is fused with the preset background image to obtain a sample image corresponding to the preset background image, thereby obtaining a sample image set for training the target segmentation model.

[0116] In some optional embodiments of this application, when the aforementioned device is used to fuse the second key object with the preset background image to obtain a sample image corresponding to the preset background image, it is specifically used for:

[0117] The second key object is overlaid onto the preset background image to obtain a sample image corresponding to the preset background image.

[0118] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, they will not be repeated here. Specifically, the device can execute the above method embodiments, and the foregoing and other operations and / or functions of each module in the device correspond to the corresponding processes in the various methods in the above method embodiments, which will not be repeated here for the sake of brevity.

[0119] The apparatus of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method applied in this application embodiment can be directly manifested as execution by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0120] Figure 3 This is a schematic block diagram of an electronic device provided in an embodiment of this application. The electronic device may include:

[0121] The system includes a memory 301 and a processor 302. The memory 301 stores computer programs and transfers the program code to the processor 302. In other words, the processor 302 can retrieve and run the computer programs from the memory 301 to implement the methods described in the embodiments of this application.

[0122] For example, the processor 302 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0123] In some embodiments of this application, the processor 302 may include, but is not limited to:

[0124] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0125] In some embodiments of this application, the memory 301 includes, but is not limited to:

[0126] Volatile memory and / or non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM).

[0127] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 301 and executed by the processor 302 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0128] like Figure 3 As shown, the electronic device may further include:

[0129] Transceiver 303, which can be connected to processor 302 or memory 301.

[0130] The processor 302 can control the transceiver 303 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 303 may include a transmitter and a receiver. The transceiver 303 may further include antennas, and the number of antennas may be one or more.

[0131] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0132] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0133] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0134] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments claimed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0136] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0137] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for determining a sample image set, characterized in that, include: The first region occupied by the first key object in the first image to be processed is labeled to obtain the first labeled image corresponding to the first image to be processed; Based on a preset cropping method, the first annotated image is cropped to obtain a cropped image; The second key object corresponding to the first key object is obtained from the cropped image; Based on multiple preset background images and the second key object, a sample image set is determined for training a target segmentation model. The target segmentation model is used to label the second region occupied by the second key object in the second image to be processed.

2. The method according to claim 1, characterized in that, The first region occupied by the first key object in the first image to be processed is annotated, including: In response to the fact that the number of processed images has reached a preset threshold, the first region occupied by the first key object in the first image to be processed is labeled using the initial segmentation model corresponding to the target segmentation model.

3. The method according to claim 1, characterized in that, The annotation of the first region occupied by the first key object in the first image to be processed also includes: In response to the fact that the number of processed images has not reached a preset threshold, the first region occupied by the first key object in the first image to be processed is annotated using the target interactive annotation model.

4. The method according to claim 3, characterized in that, The initial interactive annotation model includes: an initial image encoder, an image embedding unit, and an initial image decoder. The method further includes: Obtain the second set of sample images; The initial interactive annotation model is trained based on the second sample image set to obtain a trained initial interactive annotation model. During the training of the initial interactive annotation model, the parameters of the initial image decoder are adjustable parameters. The target interactive annotation model is determined based on the initial interactive annotation model that has been trained.

5. The method according to claim 4, characterized in that, The target interaction annotation model includes: a target image encoder, the image embedding unit, and a target image decoder. Determining the target interaction annotation model based on the trained initial interaction annotation model includes: replacing the trained initial image encoder in the trained initial interaction annotation model with the target image encoder to obtain the target interaction annotation model. Before the initial interaction annotation model is trained, the initial image encoder in the initial interaction annotation model is an encoder obtained by distilling the target image encoder.

6. The method according to claim 1, characterized in that, Based on multiple preset background images and the second key object, a sample image set is determined for training the target segmentation model, including: For each of the plurality of preset background images; The second key object is fused with the preset background image to obtain a sample image corresponding to the preset background image, thereby obtaining a sample image set for training the target segmentation model.

7. The method according to claim 6, characterized in that, The second key object is fused with the preset background image to obtain a sample image corresponding to the preset background image, including: The second key object is overlaid onto the preset background image to obtain a sample image corresponding to the preset background image.

8. A sample image set determination device, characterized in that, include: An annotation unit is used to annotate the first region occupied by the first key object in the first image to be processed, so as to obtain the first annotated image corresponding to the first image to be processed. The cropping unit is used to crop the first annotated image based on a preset cropping method to obtain a cropped image; The acquisition unit is used to acquire the second key object corresponding to the first key object from the cropped image; The determining unit is used to determine a set of sample images for training a target segmentation model based on multiple preset background images and the second key object. The target segmentation model is used to label the second region occupied by the second key object in the second image to be processed.

9. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1-7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

Images