Method and device for extracting target object in image, equipment and medium

Abstract

The invention discloses a method, a device and equipment for extracting a target object in an image and a medium, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a to-be-recognized image; receiving a description statement input by a user for a target object, and translating the description statement into instruction information through a translation node; inputting the to-be-recognized image and the instruction information into an image recognition model, so that the image recognition model recognizes a target object from the to-be-recognized image according to the instruction information; acquiring contour information of the target object; generating a mask image of the target object according to the contour information; and performing superposition processing on the mask image and the to-be-recognized image to extract image data of the target object from the to-be-recognized image. According to the technical scheme, efficient and accurate cutout processing can be realized, the cutout speed and quality are improved, dependence on color distinguishing in the image is not needed, and better robustness is achieved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to a method, device, equipment and medium for extracting a target object in an image. Background Art

[0002] With the rapid development of science and technology, people's image processing technology is also constantly improving, and cutout is one of the more commonly used techniques.

[0003] In some scenes, it is necessary to cut out the target person, target object, etc. in the image, which requires the use of cutout technology. Commonly used methods include manual cutout and automatic image recognition cutout. However, manual cutout has the problem of low efficiency and low accuracy. Automatic image recognition cutout needs to be able to accurately identify the image, which requires a large difference between the foreground and background colors of the cutout target to be able to accurately cut out. Moreover, with the current increasing demand for images, it is often necessary to use partial areas such as the arms of people and the corners of objects as cutout objects, which makes the automatic image recognition cutout method completely lack the basis for distinguishing partial areas as cutout objects. Therefore, how to better adapt to various cutout needs and accurately extract target objects in images is a technical problem that needs to be solved in this field. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, equipment and medium for extracting target objects from an image, so as to extract the target objects from the image according to the actual needs of the user, realize efficient and accurate cutout processing, improve the cutout speed and quality, and do not need to rely on the distinction of colors in the image, and have better robustness.

[0005] In a first aspect, an embodiment of the present application provides a method for extracting a target object in an image, the method comprising:

[0006] Obtain an image to be recognized;

[0007] Receiving a description sentence input by a user for a target object, and translating the description sentence into instruction information through a translation node;

[0008] Inputting the image to be recognized and the instruction information into an image recognition model, so that the image recognition model recognizes the target object from the image to be recognized according to the instruction information;

[0009] Acquire contour information of the target object;

[0010] generating a mask image of the target object according to the contour information;

[0011] The mask image is superimposed with the image to be identified to extract image data of the target object from the image to be identified.

[0012] Furthermore, before translating the description statement into instruction information through the translation node, the method further includes:

[0013] Obtaining instruction statement types supported by the image recognition model;

[0014] Accordingly, the description statement is translated into instruction information by a translation node, including:

[0015] According to the instruction statement type, the description statement is translated into instruction information matching the instruction statement type.

[0016] Further, obtaining the contour information of the target object includes:

[0017] Generate a contour extraction instruction, and draw the contour information of the target object through the image recognition model.

[0018] Furthermore, generating a mask image of the target object according to the contour information includes:

[0019] The contour information is provided to ComfyUI, so that ComfyUI generates a mask image with black pixels and white pixels based on the contour information; wherein the black pixels correspond to unconcerned parts, and the white pixels correspond to target objects.

[0020] Furthermore, the image recognition model is a YOLO model;

[0021] Accordingly, the image to be identified and the instruction information are input into an image recognition model, so that the image recognition model identifies the target object from the image to be identified according to the instruction information, including:

[0022] The image to be identified and the instruction information are input into the YOLO model, so that the YOLO model performs target object detection on the image to be identified under the guidance of the instruction information to obtain the target object in the image to be identified.

[0023] Furthermore, the image recognition model is a SAM large model;

[0024] Inputting the image to be recognized and the instruction information into an image recognition model, so that the image recognition model recognizes the target object from the image to be recognized according to the instruction information, comprises:

[0025] The image to be identified and the instruction information are input into the SAM large model, so that the SAM large model, under the guidance of the instruction information, performs target object identification on the image to be identified to obtain the target object in the image to be identified.

[0026] Further, the target object is a designated part of a complete body;

[0027] The translation node is a translation node of ComfyUI.

[0028] In a second aspect, an embodiment of the present application provides a device for extracting a target object in an image, the device comprising:

[0029] An image acquisition module, used for acquiring an image to be identified;

[0030] An instruction information generating module, used for receiving a description sentence inputted by a user for a target object, and translating the description sentence into instruction information through a translation node;

[0031] A target object recognition module, used for inputting the image to be recognized and the instruction information into an image recognition model, so that the image recognition model recognizes the target object from the image to be recognized according to the instruction information;

[0032] A contour information acquisition module, used to acquire contour information of the target object;

[0033] A mask image generating module, used for generating a mask image of the target object according to the contour information;

[0034] The extraction module is used to perform superposition processing on the mask image and the image to be identified, so as to extract the image data of the target object from the image to be identified.

[0035] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0036] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0037] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0038] In an embodiment of the present application, an image to be identified is obtained; a description statement input by a user for a target object is received, and the description statement is translated into instruction information through a translation node; the image to be identified and the instruction information are input into an image recognition model, so that the image recognition model can identify the target object from the image to be identified according to the instruction information; contour information of the target object is obtained; a mask image of the target object is generated according to the contour information; and the mask image is superimposed with the image to be identified to extract the image data of the target object from the image to be identified. The technical solution provided by the present application can realize efficient and accurate cutout processing, improve the cutout speed and quality, and does not need to rely on the distinction of colors in the image, and has better robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flowchart of a method for extracting a target object in an image provided in Example 1 of the present application;

[0040] Figure 2 is a flow chart of a method for extracting a target object in an image provided in Example 2 of the present application;

[0041] Figure 3 is a schematic diagram of a target object extraction process in an image provided by Embodiment 2 of the present application;

[0042] Figure 4 is a schematic diagram of the structure of a device for extracting a target object in an image provided in Embodiment 3 of the present application;

[0043] Figure 5 It is a schematic diagram of the structure of the electronic device provided in Example 4 of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all of the content is shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0046] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0047] In conjunction with the accompanying drawings, the following detailed description is given of the method, device, equipment and medium for extracting target objects in images provided by the embodiments of the present application through specific embodiments and their application scenarios.

[0048] Embodiment 1

[0049] Figure 1 It is a flowchart of a method for extracting a target object in an image provided in Example 1 of the present application.

[0050] like Figure 1 As shown, the specific steps include:

[0051] S101, obtaining an image to be recognized;

[0052] First, the present application is applicable to the scenario of cutting out a part of the target area in the image. Based on the above scenario, it can be understood that the execution subject of the present application can be a terminal device with image processing capabilities, such as a desktop computer, a laptop computer, a tablet computer, a smart phone, and an intelligent auxiliary device.

[0053] The image to be identified can be an image captured by an electronic device or read from a storage device. In addition, the image to be identified can also be a drawn image, such as a frame of an animation. The acquisition method can be to obtain image data during or after the drawing process by connecting to the drawing software.

[0054] S102, receiving a description sentence input by a user for a target object, and translating the description sentence into instruction information through a translation node;

[0055] After the image to be identified is acquired, the image to be identified can be displayed on the display screen of the electronic device. The user can determine the target object that he needs to cut out in the displayed image to be identified, and describe it through language. The user can enter a description sentence of the target object for the electronic device to determine the object that the user actually needs to cut out. For example, the user can enter "Atom Boy", "Atom Boy's arm", "Atom Boy's head", "the person standing in the picture", "the arm of the person standing in the picture", etc. Among them, the target objects corresponding to "Atom Boy" and "the person standing in the picture" are equivalent to a whole, and the target objects corresponding to "Atom Boy's arm", "Atom Boy's head", and "the arm of the person standing in the picture" are equivalent to a partial area of ​​a whole.

[0056] In this solution, the description statement can be a natural sentence, and the description statement needs to be translated into an image processing instruction that can be executed by the electronic device, so translation is required. Here, the BERT model can be used for translation. The BERT model is a language representation model. BERT stands for Bidirectional Encoder Representations from Transformers. The BERT model includes an Embedding module, a Transformer module, and a pre-fine-tuning module. Embedding module: It mainly consists of three types of Embeddings, including Token Embeddings for representing input words or subwords, Positional Embeddings for representing the position information of words in sentences, and Segment Embeddings for distinguishing different sentence sequences. Transformer module: BERT uses a multi-layer stacked Transformer Encoder as its main structure. Each layer includes a self-attention mechanism and a feedforward neural network. Through the multi-layer Transformer structure, BERT can capture long-distance dependencies and produce high-quality representations. Pre-fine-tuning module: After pre-training, BERT can adapt to specific NLP tasks through fine-tuning of specific tasks. For example, for text classification tasks, the last linear layer can be replaced by a classification layer; for sequence labeling tasks, a CRF layer can be used, etc.

[0057] In addition, as a bidirectional encoding model based on Transformer, the BERT model is connected through the TransformerEncoder block, which can simultaneously consider the contextual information on both the left and right sides of the word, thereby capturing richer and more complex semantic information than earlier models. In the pre-training stage, the BERT model uses a large amount of text data to learn rich semantic and syntactic knowledge, and then fine-tunes it to adapt to various NLP tasks. This approach not only saves a lot of computing resources, but also continuously updates and optimizes the performance of the model.

[0058] It can be understood that the instruction information may be an instruction for performing cutout, which may be provided to an algorithm responsible for cutout to perform a cutout operation on a target object.

[0059] S103, inputting the image to be recognized and the instruction information into an image recognition model, so that the image recognition model recognizes a target object from the image to be recognized according to the instruction information;

[0060] In this solution, after obtaining the instruction information, the image to be identified and the instruction information can be input into the image recognition model. In this way, the image recognition model can determine the target object in the image to be identified based on the instruction information after performing content recognition on the image to be identified. For example, the target object is an arm of a person, a drawing on a flower pot, and so on.

[0061] Here, the image recognition model may be a YOLO model, which may be used to detect a target object in a picture. In addition, the image recognition model may also use other models, such as a Segment Anything (SAM) model, to perform semantic segmentation on the picture.

[0062] S104, obtaining contour information of the target object;

[0063] Among them, after determining the target object, an image recognition model or other means can be used to generate a contour based on the determined target object for subsequent cutout use. It can be understood that the contour information may include pixel position information of each point on the contour line. At the same time, the contour can be closed or open, for example, it is also possible that the boundary with the image to be identified can form a closed area.

[0064] S105, generating a mask image of the target object according to the contour information;

[0065] Contour information can be information that reflects the position of the contour. After obtaining the contour information, a mask image can be generated. Mask image refers to a set of binary images used to describe the characteristics, position or other information of certain parts of the original image in image processing. In layman's terms, it is to use the mask image to mark the area in the original image that we need to pay attention to, without considering the content of the entire image. The mask image is usually composed of black and white pixels, where black represents the part that is not concerned and white represents the part that is concerned.

[0066] S106: Perform superposition processing on the mask image and the image to be identified, so as to extract image data of the target object from the image to be identified.

[0067] This solution can use the mask image and the image to be identified for superposition processing, that is, to obtain which pixels in the image to be processed correspond to the parts of the mask image that need to be processed, and which pixels correspond to the parts of the mask image that do not need to be processed, so that the image data of the target object can be extracted based on the superposition.

[0068] The superposition operation of the two needs to be aligned, so that the correspondence between the pixels can be ensured. After the superposition, the target position of the cutout can be determined based on the position of the white pixel in the mask image, so as to achieve the cutout of the target object.

[0069] The technical solution provided in this embodiment is as follows: acquiring an image to be identified; receiving a description statement input by a user for a target object, and translating the description statement into instruction information through a translation node; inputting the image to be identified and the instruction information into an image recognition model, so that the image recognition model can identify the target object from the image to be identified according to the instruction information; acquiring contour information of the target object; generating a mask image of the target object according to the contour information; and superimposing the mask image with the image to be identified to extract the image data of the target object from the image to be identified. The technical solution provided in this application can realize efficient and accurate cutout processing, improve the cutout speed and quality, and does not need to rely on the distinction of colors in the image, and has better robustness.

[0070] In one embodiment, optionally, the target object is a specified part of a complete body; and the translation node is a translation node of ComfyUI.

[0071] Among them, the specified parts of the complete subject, such as the head, arms, torso and legs of a cartoon character. The extraction of the specified parts of the complete subject can be used as the basis for subsequent animation generation. For example, in the animation or game industry, each part of the constructed character can be distinguished to extract the image, so that when the subsequent animation is generated and the dynamic effect is drawn, the image of the extracted specified part can be used as a resource, which can improve the efficiency of image generation and reduce the workload of the staff.

[0072] In this solution, the translation node can be a translation node of ComfyUI. ComfyUI is a node-based graphical user interface (GUI) specially designed for StableDiffusion. It enables users to build complex image generation workflows by linking different blocks (called nodes). ComfyUI has the following characteristics: It provides extremely high degrees of freedom and flexibility, and supports a high degree of customization and workflow reuse. It has low requirements for system configuration and can speed up the generation of original images. In the node-based interface, nodes can include various tasks, such as loading checkpoint models, input prompts, specifying samplers, etc. The configuration and connection method of the nodes directly affect the final output results. Nodes are divided into official native nodes and custom nodes developed by users. Compared with the traditional StableDiffusion WebUI, the advantages of ComfyUI are:

[0073] The video memory requirement is low, so more users can use it easily without having to worry too much about hardware configuration issues.

[0074] Fast startup and operation speed significantly improves the generation speed, saving time and computing resources.

[0075] The generation freedom is high, and the workflow can be customized and adjusted according to needs to achieve more diverse generation effects.

[0076] Environments and models can be shared between ComfyUI and WebUI, making it easy to conduct various experiments and creations.

[0077] The workflow can be customized and shared. When a problem occurs, the specific location of the error is clear at a glance, making it easy to troubleshoot and solve. When the generated image is dragged in, ComfyUI will automatically restore the entire workflow and select the appropriate model for the user, greatly improving work efficiency.

[0078] This technical solution, by extracting the specified parts of the complete object, can facilitate subsequent processing such as animation production and dynamic effect display. At the same time, using the translation node of ComfyUI, natural language can be converted into instruction information that can be understood by the image recognition model, and accurate extraction of the specified parts can be achieved, so that the extracted target object is more in line with the actual needs of the user, and there is no need to use coded processing, which improves the convenience of use for staff and has strong robustness.

[0079] Based on the above technical solutions, optionally, before translating the description statement into instruction information through a translation node, the method also includes: obtaining the instruction statement type supported by the image recognition model; accordingly, translating the description statement into instruction information through a translation node includes: according to the instruction statement type, translating the description statement into instruction information matching the instruction statement type.

[0080] In this scheme, before translating through the translation node, it is necessary to determine the instruction statement type supported by the image recognition model. For example, if the image recognition model supports the instruction statement type of the first format, and the translation obtains the instruction statement type of the second format, then even if the image recognition model obtains the instruction information, it is impossible to clearly know how to execute the instruction information, and even an error may occur. Therefore, it is necessary to determine the instruction statement type supported by the image recognition model before translation. Specifically, the image recognition model can be determined before translation, such as a YOLO model, or a SAM large model, or other artificial intelligence models. After obtaining the image recognition model, the instruction statement type supported by the model can be determined. For example, the YOLO model supports the instruction statement type of the first format, and the SAM large model supports the instruction statement type of the second format. In the process of translating the translation node, it can be determined according to the image recognition model which format to use for translation.

[0081] Through such a setting, the present scheme can ensure that the instruction information obtained through the translation node is the instruction information that can be understood by the subsequent image recognition model, thereby avoiding the situation where the instruction information cannot be understood and the image recognition model cannot determine the target object after identifying the image content. This can improve the internal coordination of the implementation process of the present scheme and improve the image processing capability of the scheme provided by the present scheme.

[0082] On the basis of the above technical solutions, optionally, obtaining the contour information of the target object includes: generating a contour extraction instruction, and drawing the contour information of the target object through the image recognition model.

[0083] In this solution, after determining the target object, its contour information needs to be determined to provide a data basis for subsequent cutouts. Therefore, after acquiring the target object, a corresponding contour extraction instruction needs to be generated to draw the contour information of the target object through an image recognition model. It should be noted that the extraction of contour information is completed by the image recognition model. After the image recognition model recognizes the target object according to the instruction information, it generates the contour information of the target object according to the received contour extraction instruction.

[0084] Through such a setting, the present scheme can entrust the task of generating contour information to the image recognition model, so that after identifying the target object, the image recognition model can directly complete the operation of generating its contour information without the need for data transmission. After the contour information is generated, it can be verified based on the range of the identified target object to ensure the accuracy of the generated contour information.

[0085] On the basis of the above technical solutions, optionally, the image recognition model is a YOLO model;

[0086] Accordingly, the image to be identified and the instruction information are input into an image recognition model, so that the image recognition model identifies the target object from the image to be identified according to the instruction information, including:

[0087] The image to be identified and the instruction information are input into the YOLO model, so that the YOLO model performs target object detection on the image to be identified under the guidance of the instruction information to obtain the target object in the image to be identified.

[0088] The image recognition model uses the YOLO model. The YOLO (You Only Look Once) model is an object detection algorithm based on a single neural network. The YOLO model has the following characteristics:

[0089] Fast speed: Able to process images in real time and achieve fast target detection.

[0090] End-to-end training: The object detection problem is formulated as a regression problem, using a convolutional neural network to predict bounding boxes and class probabilities directly from the input image.

[0091] The basic process of the YOLO model execution algorithm is as follows:

[0092] Resize the input image to a fixed size (such as 448×448);

[0093] Process the image through convolutional neural network;

[0094] The final detection result is obtained by using non-maximum suppression processing;

[0095] In specific implementation, YOLO divides the image into S×S grids. Each cell is responsible for detecting targets whose center points fall within the grid. Each cell predicts B bounding boxes and the confidence of these bounding boxes. The confidence includes two aspects: one is the possibility that the bounding box contains the target, and the other is the accuracy of the bounding box, which is defined as the intersection over union (IoU) between the predicted box and the actual box. The size and position of the bounding box are represented by 4 values ​​(x, y, w, h), where (x, y) is the center coordinate of the bounding box (the offset value relative to the coordinate point of the upper left corner of each cell, and the unit is relative to the cell size), and w and h are the width and height of the bounding box (the ratio of the width to the height of the entire image). Each bounding box actually contains 5 elements (x, y, w, h, c), the first 4 represent the size and position of the bounding box, and the last value is the confidence. In addition, each cell also gives the predicted C category probability values, indicating the probability that the target of the bounding box predicted by the cell belongs to each category.

[0096] The network structure of the YOLO algorithm usually draws on other excellent network structures, such as GoogleNet, etc. For example, YOLO v3 uses a feature extractor architecture called Darknet-53, which contains 53 convolutional layers, each followed by a batch normalization layer and a Leaky ReLU activation function, and uses a convolutional layer with a stride of 2 to downsample the feature map.

[0097] The training process of the YOLO algorithm involves using a pre-trained classification network and then fine-tuning it for the detection task. The loss function usually contains parts such as the coordinate prediction error of the bounding box and the confidence prediction error.

[0098] Different versions of the YOLO algorithm may differ in network structure, performance, etc., but the overall ideas and basic principles are similar. These improved versions are designed to improve detection accuracy, speed or adapt to different application scenarios.

[0099] By using the YOLO model, this solution can achieve accurate detection of target objects from the image to be identified. It not only has a fast detection speed, but also meets the actual needs of users. For example, after the staff inputs a certain natural sentence, after translation, they can obtain instruction information that the YOLO model can understand, and under the guidance of the instruction information, the target object detection is performed on the image to be identified to obtain the target object in the image to be identified. In this way, the code writing burden of the staff can be greatly reduced and the recognition effect of the target object can be improved.

[0100] On the basis of the above technical solutions, optionally, the image recognition model is a SAM large model;

[0101] Inputting the image to be recognized and the instruction information into an image recognition model, so that the image recognition model recognizes the target object from the image to be recognized according to the instruction information, comprises:

[0102] The image to be identified and the instruction information are input into the SAM large model, so that the SAM large model, under the guidance of the instruction information, performs target object identification on the image to be identified to obtain the target object in the image to be identified.

[0103] Among them, the SAM (Segment Anything Model) model is an advanced image segmentation model. It can segment any object in a photo or video image with one click, and can be transferred to other tasks with zero samples.

[0104] The design of the SAM large model is based on three main components:

[0105] Promptable segmentation tasks: After training on millions of images and over 1 billion masks, it can return valid segmentation masks for any prompt. The prompt is the segmentation task, which can be foreground / background points, rough boxes, masks, clicks, text, etc. Any information indicating what to segment in the image, which is also used as the pre-training target of the model.

[0106] Model architecture: consists of three components working together.

[0107] Image encoder: used to generate one-time image embeddings. For example, masked autoencoders (MAE) and pre-trained visual transformers (ViT) can be applied before the prompt model to extract features from the input image and generate image embeddings.

[0108] Hint Encoder: Encodes background points, masks, bounding boxes, or text into embedding vectors in real time. It considers both sparse (points, boxes, text) and dense (masks) sets of hints. Points and boxes are represented by positional encodings and learned embeddings are added for each hint type; free-form text hints are represented by CLIP's off-the-shelf text encoder; dense hints (such as masks) are embedded via convolutions and element-wise summed over image embeddings.

[0109] Mask Decoder: Predicts a segmentation mask based on the embeddings of the image and hint encoders. It maps the image embedding, hint embedding, and output token to the mask, all of which are updated by the decoder block, which uses immediate self-attention and cross-attention in both directions (from hint to image embedding and back). The masks are annotated and used to update the model weights, which augments the dataset and allows the model to learn and improve over time, making it efficient and flexible.

[0110] Datasets supporting tasks and models: SegmentAnything 10 billion mask (SA-1B) dataset, the largest annotated segmentation dataset to date, designed for the development and evaluation of advanced segmentation models. The dataset covers a wide range of fields, objects, and scenes, with diversity; contains more than one billion high-quality annotated images, with a large amount of data; and is carefully annotated with high-quality masks, with high accuracy and detail.

[0111] The SAM large model has the following features and advantages:

[0112] You can segment objects by simply clicking or interactively selecting points to include or exclude from the object, or you can create segmentations by drawing a bounding box or segmentation region using the Polygon tool, which snaps to objects.

[0113] When uncertainty is encountered in identifying an object, multiple valid masks can be generated.

[0114] All objects present in the image can be automatically identified and masks generated.

[0115] After pre-computing image embeddings, segmentation masks are immediately provided for any cue, enabling real-time interaction with the model.

[0116] It supports multiple prompts, such as mouse clicks, picture frames, etc., and can also split everything with one click.

[0117] Its prompts can be flexibly integrated with other systems and support output method expansion.

[0118] With zero-shot generalization capabilities, it has learned the general concept of objects and is able to segment unfamiliar objects and images without additional training.

[0119] The open source of SAM has promoted the development of image segmentation and computer vision related fields. It has broad application potential in data annotation, lowering the threshold for algorithm application, and various downstream tasks, such as medical image segmentation, video segmentation, image restoration, 3D tasks, image generation, remote sensing tasks, moving object detection, OCR tasks, etc. For example, in medical imaging, it can be used to extract clinically relevant information and help radiologists enhance analysis; in the field of autonomous driving, it can reduce the difficulty of semantic segmentation, improve algorithm recognition capabilities and be applied to data annotation, etc.

[0120] This solution can improve the accuracy of target object recognition by using the SAM large model to identify the target object in the image to be recognized under the guidance of the instruction information. At the same time, the SAM large model can support the recognition of target objects based on the instruction information, which can improve the flexibility of target object recognition.

[0121] Embodiment 2

[0122] Figure 2 : is a flow chart of a method for extracting a target object in an image provided in Example 2 of the present application. This solution makes a better improvement on the above-mentioned embodiment, and the specific improvement is: generating a mask image of the target object according to the contour information, including: providing the contour information to ComfyUI, so that ComfyUI generates a mask image with black pixels and white pixels based on the contour information; wherein the black pixels correspond to the unconcerned parts, and the white pixels correspond to the target object.

[0123] like Figure 2 As shown, the specific steps include:

[0124] S201, obtaining an image to be recognized;

[0125] S202, receiving a description sentence input by a user for a target object, and translating the description sentence into instruction information through a translation node;

[0126] S203, inputting the image to be recognized and the instruction information into an image recognition model, so that the image recognition model recognizes a target object from the image to be recognized according to the instruction information;

[0127] S204, obtaining contour information of the target object;

[0128] S205, providing the contour information to ComfyUI, so that ComfyUI generates a mask image with black pixels and white pixels based on the contour information; wherein the black pixels correspond to the unconcerned parts, and the white pixels correspond to the target objects;

[0129] ComfyUI can generate a mask image based on the contour information after receiving the contour information. Specifically, it can be understood that the mask image can be generated based on the position information of each pixel in the contour information and the information of the target object and the information of the uninterested part corresponding to the two sides of the line formed by the pixel points.

[0130] S206: Perform superposition processing on the mask image and the image to be identified, so as to extract image data of the target object from the image to be identified.

[0131] In this solution, after the mask image is generated, the mixed and superimposed relationship of the images can be used to superimpose and mix the black and white mask with the image to be identified, and the masked part after superposition can be extracted, thereby realizing the extraction of the specified object.

[0132] Through such a setting, this embodiment can improve the accuracy of mask image generation, thereby improving the accuracy of target object extraction, and the generation of the mask image can be completed based on a certain node of ComfyUI, thereby improving the efficiency of mask image generation.

[0133] Figure 3 Schematic diagram of the target object extraction process in the image provided by the second embodiment of the present application. In a specific embodiment, the present solution may adopt the following steps:

[0134] Step 1: The user's description is automatically converted into a language that YOLO can recognize through the ComfyUI translation node;

[0135] Step 2: Input the converted language and image to YOLO recognition, and let YOLO find objects that match the description in the image;

[0136] Step 3: YOLO will outline the matching object, and then use ComfyUI to convert the outline into a black and white mask image;

[0137] Step 4. Finally, use the mixed and superimposed relationship of the image to superimpose and mix the black and white mask with the original image, and extract the masked part after superposition, so as to achieve the extraction of the specified object.

[0138] In an exemplary scenario, for example, a staff member needs to cut out a cartoon character and separate its limbs for subsequent animation. This technology can be used to cut out multiple elements such as the head, hands, feet, and body concurrently, and each thread is cut out separately, thus achieving a one-click cutout and layered effect.

[0139] This solution can achieve accurate extraction of objects in the picture through semantic segmentation. It can also accurately identify multiple objects in the picture, breaking away from the limitations of manual object identification. Using this technical solution for identification can improve work efficiency and empower mass production.

[0140] Embodiment 3

[0141] Figure 4 It is a structural schematic diagram of a device for extracting a target object in an image provided in Example 3 of the present application.

[0142] like Figure 4 As shown, the device comprises:

[0143] An image acquisition module 401 is used to acquire an image to be recognized;

[0144] The instruction information generating module 402 is used to receive a description sentence input by a user for a target object, and translate the description sentence into instruction information through a translation node;

[0145] A target object recognition module 403, used for inputting the image to be recognized and the instruction information into an image recognition model, so that the image recognition model recognizes the target object from the image to be recognized according to the instruction information;

[0146] The contour information acquisition module 404 is used to acquire the contour information of the target object;

[0147] A mask image generating module 405, configured to generate a mask image of the target object according to the contour information;

[0148] The extraction module 406 is used to perform superposition processing on the mask image and the image to be identified, so as to extract the image data of the target object from the image to be identified.

[0149] The technical solution provided in this embodiment is as follows: acquiring an image to be identified; receiving a description statement input by a user for a target object, and translating the description statement into instruction information through a translation node; inputting the image to be identified and the instruction information into an image recognition model, so that the image recognition model can identify the target object from the image to be identified according to the instruction information; acquiring contour information of the target object; generating a mask image of the target object according to the contour information; and superimposing the mask image with the image to be identified to extract the image data of the target object from the image to be identified. The technical solution provided in this application can realize efficient and accurate cutout processing, improve the cutout speed and quality, and does not need to rely on the distinction of colors in the image, and has better robustness.

[0150] The device for extracting the target object in the image in the embodiment of the present application can be a device, or a component, integrated circuit or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0151] The device for extracting the target object in the image in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0152] The device for extracting target objects in images provided in the embodiments of the present application can implement each process implemented in the above method embodiments, and will not be described again here to avoid repetition.

[0153] Embodiment 4

[0154] Figure 5 Schematic diagram of the structure of the electronic device provided in the fourth embodiment of the present application. Figure 5 As shown, an embodiment of the present application also provides an electronic device 500, including a processor 501, a memory 502, and a program or instruction stored in the memory 502 and executable on the processor 501. When the program or instruction is executed by the processor 501, each process of the embodiment of the device for extracting the target object in the above-mentioned image is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0155] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0156] Embodiment 5

[0157] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned embodiment of the device for extracting the target object in the image is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0158] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0159] Embodiment 6

[0160] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned embodiment of the device for extracting target objects in the image, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0161] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0162] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0163] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0164] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0165] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A method for extracting a target object in an image, characterized in that: The method comprises: Obtain an image to be recognized; Receiving a description sentence input by a user for a target object, and translating the description sentence into instruction information through a translation node; Inputting the image to be recognized and the instruction information into an image recognition model, so that the image recognition model recognizes the target object from the image to be recognized according to the instruction information; Acquire contour information of the target object; generating a mask image of the target object according to the contour information; The mask image is superimposed with the image to be identified to extract image data of the target object from the image to be identified.

2. The method for extracting a target object in an image according to claim 1, characterized in that: Before translating the description statement into instruction information through the translation node, the method further includes: Obtaining instruction statement types supported by the image recognition model; Accordingly, the description statement is translated into instruction information by a translation node, including: According to the instruction statement type, the description statement is translated into instruction information matching the instruction statement type.

3. The method for extracting a target object in an image according to claim 1, characterized in that: Acquiring contour information of the target object includes: Generate a contour extraction instruction, and draw the contour information of the target object through the image recognition model.

4. The method for extracting a target object in an image according to claim 1, characterized in that: Generating a mask image of the target object according to the contour information includes: The contour information is provided to ComfyUI, so that ComfyUI generates a mask image with black pixels and white pixels based on the contour information; wherein the black pixels correspond to unconcerned parts, and the white pixels correspond to target objects.

5. The method for extracting a target object in an image according to claim 1, characterized in that: The image recognition model is a YOLO model; Accordingly, the image to be identified and the instruction information are input into an image recognition model, so that the image recognition model identifies the target object from the image to be identified according to the instruction information, including: The image to be identified and the instruction information are input into the YOLO model, so that the YOLO model performs target object detection on the image to be identified under the guidance of the instruction information to obtain the target object in the image to be identified.

6. The method for extracting a target object in an image according to claim 1, characterized in that: The image recognition model is a SAM large model; Inputting the image to be recognized and the instruction information into an image recognition model, so that the image recognition model recognizes the target object from the image to be recognized according to the instruction information, comprises: The image to be identified and the instruction information are input into the SAM large model, so that the SAM large model, under the guidance of the instruction information, performs target object identification on the image to be identified to obtain the target object in the image to be identified.

7. The method for extracting a target object in an image according to any one of claims 1 to 6, characterized in that: The target object is a designated part of a complete subject; The translation node is a translation node of ComfyUI.

8. A device for extracting a target object in an image, characterized in that: The device comprises: An image acquisition module, used for acquiring an image to be identified; An instruction information generating module, used for receiving a description sentence inputted by a user for a target object, and translating the description sentence into instruction information through a translation node; A target object recognition module, used for inputting the image to be recognized and the instruction information into an image recognition model, so that the image recognition model recognizes the target object from the image to be recognized according to the instruction information; A contour information acquisition module, used to acquire contour information of the target object; A mask image generating module, used for generating a mask image of the target object according to the contour information; The extraction module is used to perform superposition processing on the mask image and the image to be identified, so as to extract the image data of the target object from the image to be identified.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method for extracting a target object in an image as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method for extracting a target object in an image according to any one of claims 1 to 7 are implemented.

Images