Image processing method and device, electronic equipment, and storage medium

Abstract

The embodiment of the present disclosure relates to the technical field of image processing, and relates to an image processing method and device, electronic equipment and a computer readable storage medium. The image processing method comprises: acquiring a to-be-processed image; performing subject detection on the to-be-processed image to acquire a detection frame of a subject object contained in the to-be-processed image, a text category label corresponding to the detection frame and a category confidence; determining a target text category label according to the category confidence and a user selection operation acting on the text category label; and performing cutout processing on the to-be-processed image in combination with the target text category label, the detection frame and the to-be-processed image to acquire a target subject object corresponding to the user selection operation. The technical solution in the embodiment of the present disclosure can improve the accuracy and authenticity of image cutout.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to an image processing method and apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In the process of image processing, image matting can be performed to separate the subject and background, thereby enabling image processing of both the subject and the background.

[0003] In related technologies, for images containing multiple subjects, single-subject matting is generally achievable, but the effect and accuracy are poor when matting multiple subjects. Alternatively, relying solely on corpus information from multimodal subject segmentation algorithms and on user-defined subject objects in the image may lead to problems such as incorrect subject identification, resulting in poor accuracy. Summary of the Invention

[0004] The purpose of this disclosure is to provide an image processing method and apparatus, electronic device, and computer-readable storage medium, thereby overcoming, at least to some extent, the problem of limited image matting due to the limitations and defects of related technologies.

[0005] According to a first aspect of this disclosure, an image processing method is provided, comprising: acquiring an image to be processed; performing subject detection on the image to be processed, acquiring a detection box containing a subject object in the image to be processed, a text category label corresponding to the detection box, and a category confidence score; determining a target text category label based on the category confidence score and a user selection operation applied to the text category label; and performing image matting on the image to be processed by combining the target text category label, the detection box, and the image to be processed to acquire a target subject object corresponding to the user selection operation.

[0006] According to a second aspect of this disclosure, an image processing apparatus is provided, comprising: an image acquisition module for acquiring an image to be processed; a subject detection module for performing subject detection on the image to be processed, acquiring a detection box, text category label, and category confidence score of the subject object contained in the image to be processed; a label determination module for determining a target text category label based on the category confidence score and a user selection operation applied to the text category label; and an image matting module for performing image matting processing on the image to be processed by combining the target text category label, the detection box, and the image to be processed, to obtain the target subject object corresponding to the user selection operation.

[0007] According to a third aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method described in any of the preceding methods and possible implementations thereof by executing the executable instructions.

[0008] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the method described in any of the preceding claims and possible implementations thereof.

[0009] In the technical solution provided in this disclosure, on the one hand, a target text category label can be determined based on the user's selection operation by combining the category confidence of the text category label. Then, by combining the target text category label, the detection box, and the image to be processed, the target subject object can be determined by extracting from the image to be processed. Since the number of target subject objects obtained by extraction can be determined according to the user's selection operation, it can avoid the limitation of related technologies that can only extract... Figure 1 The limitations of not being able to specify a single subject or define the target image for cutout have increased the scope and diversity of applications, enabling interactive cutout and improving user experience. On the other hand, by incorporating category confidence into the cutout process, the quality and accuracy of the obtained target subject can be improved. Furthermore, by combining data from multiple modalities, such as target text category labels, detection boxes, and the image to be processed, to obtain the specified target subject, the accuracy and specificity of the cutout can be enhanced.

[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0012] Figure 1 A schematic diagram illustrates an application scenario where the image processing method of the present disclosure embodiments can be applied.

[0013] Figure 2 The schematic diagram illustrates a flowchart of an image processing method according to an embodiment of the present disclosure.

[0014] Figure 3 This illustration shows a schematic diagram of the process of subject detection using a target detection model in an embodiment of this disclosure.

[0015] Figures 4A-4C This illustration schematically shows a diagram of determining the detection box, text category label, and category confidence level in an embodiment of this disclosure.

[0016] Figure 5The schematic diagram illustrates the process of determining the target subject object in an embodiment of this disclosure.

[0017] Figure 6 The illustration shows a flowchart of image matting using a segmentation model in an embodiment of this disclosure.

[0018] Figures 7A-7B This illustration shows a schematic diagram of image matting using one or more input data in an embodiment of this disclosure.

[0019] Figure 8 This diagram illustrates the image matting results in an embodiment of the present disclosure.

[0020] Figure 9 This schematically illustrates the overall process of interactive image matting according to an embodiment of the present disclosure.

[0021] Figure 10 A schematic block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown.

[0022] Figure 11 A block diagram of an electronic device according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0023] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0024] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0025] In order to solve the technical problems in the related technologies, this disclosure provides an image processing method that can be applied to the process of cutting out an image to obtain a subject object. Then, various types of image processing can be performed on the cut-out subject object, such as editing the subject object or performing operations such as changing the background. Figure 1 A schematic diagram of a system architecture for an image processing method and apparatus applicable to embodiments of the present disclosure is shown.

[0026] like Figure 1 As shown, image 101 to be processed can be obtained. The image to be processed can be a color image. The image to be processed can be a stored image or an image captured by the terminal's camera; there is no specific limitation here, as long as a color image can be obtained.

[0027] In this embodiment of the disclosure, the image to be processed can be subjected to subject detection to obtain information such as detection boxes containing the subject object, text category labels, and category confidence scores. Next, the target text category label can be determined by combining the category confidence scores of the sub-images corresponding to each detection box and the user selection operation on the text category label. By combining the target text category label, the detection boxes, and the image to be processed, the image to be processed is subjected to image matting to obtain the target subject object corresponding to the user selection operation from the image to be processed.

[0028] It should be noted that the image processing method provided in this embodiment can be executed by a terminal. The terminal can be a smart device with image processing capabilities, such as a smartphone, computer, tablet, smart speaker, smartwatch, in-vehicle device, wearable device, monitoring device, etc. The image processing device can also be located in the terminal. Alternatively, the image processing method can also be executed by a server; no specific limitations are made here.

[0029] Figure 2 The flowchart of image processing is illustrated in the diagram. (Reference) Figure 2 As shown, the main steps include:

[0030] In step S210, the image to be processed is acquired;

[0031] In step S220, subject detection is performed on the image to be processed to obtain the detection bounding box, text category label, and category confidence of the subject objects contained in the image to be processed.

[0032] In step S230, the target text category label is determined based on the category confidence level and the user selection operation applied to the text category label;

[0033] In step S240, the target text category label, detection box, and image to be processed are combined to perform image matting on the image to be processed, thereby obtaining the target subject object corresponding to the user's selected operation.

[0034] In this embodiment of the disclosure, the image to be processed can first be acquired, and subject detection can be performed on the image to be processed to obtain the detection box of each subject object contained in the image to be processed. At the same time, the text category label and category confidence of the subject object can also be acquired.

[0035] Next, the text category label and category confidence of the sub-image corresponding to the detection box can be determined. Combined with the category confidence and the user selection operation on the text category label, the target text category label can be determined. An interactive general subject segmentation method can be constructed from multiple dimensions such as the target text category label, the detection box and the image to be processed, so as to achieve subject matting of the specified object and obtain the target subject object.

[0036] The technical solution in this disclosure, on the one hand, can combine the category confidence of the text category label to determine a target text category label based on the user's selection operation. Then, combining the target text category label, the detection box, and the image to be processed, the target subject object is extracted from the image to be processed. Since the number of target subject objects obtained by extraction can be determined according to the user's selection operation, it can avoid the limitation of related technologies that can only extract... Figure 1 The limitations of not being able to specify a single subject or define the target image for cutout have increased the scope and diversity of applications, enabling interactive cutout and improving user experience. On the other hand, by incorporating category confidence into the cutout process, the quality and accuracy of the obtained target subject can be improved. Furthermore, by combining data from multiple modalities, such as target text category labels, detection boxes, and the image to be processed, to obtain the specified target subject, the accuracy and specificity of the cutout can be enhanced.

[0037] Next, refer to Figure 2 The diagram illustrates each step of the image processing method.

[0038] In step S210, the image to be processed is acquired.

[0039] In this embodiment, the image to be processed can be of various types, such as RGB color images, grayscale images, or black and white images. The image to be processed can contain at least one subject object. The at least one subject object can be of various types, such as a person, an animal, or a building. Each image to be processed can contain the same type of subject object or different types of subject objects, depending on the specific needs. The image to be processed can be an image from the network or an image sent from another client, an image captured by a terminal's camera, or an image from a video, as long as it contains at least one subject object.

[0040] Next, in step S220, subject detection is performed on the image to be processed, and the detection boxes of the main objects contained in the image to be processed, the text category labels corresponding to the detection boxes, and the category confidence scores are obtained.

[0041] In this embodiment, subject detection is used to detect subject objects contained in an image, thereby obtaining subject information of the subject objects. The number of subject objects may include one or more, specifically depending on the subject information, including the position and category information of the subject objects. Specifically, subject detection can be performed on the image to be processed to determine the detection boxes, text category labels, and category confidence scores of the subject objects contained in the image. The detection boxes can be represented by the coordinate pixel information of the detection boxes, and the coordinate pixel information of each detection box may include at least one. For example, the coordinate pixel information of the detection boxes may include the coordinates of the upper left corner and the lower right corner, etc. The text category label is used to represent the category of the subject object and can be represented by text information. The text category label can be various forms of text information, such as Chinese text or English text, etc. The category confidence score can be the probability that the subject object belongs to the text category label. The value range of the category confidence score can be [0,1), and the higher the probability that the subject object belongs to the text category label, the greater its category confidence score. Furthermore, each subject object can correspond to a detection box and can correspond to a text category label and a category confidence score.

[0042] Subject detection on an image to be processed yields multiple sub-images from which the main subject is separated. The number of sub-images is determined by the number of main subjects. A sub-image can be a portion of the image to be processed, and each sub-image may include one main subject, as well as some other main subjects. For example, sub-image 1 may include a portion of a person and a house. Next, the text category labels of the main subjects contained in the sub-images and the category confidence of these labels can be determined. Specifically, the main subject with the largest area in a sub-image can be identified as the main subject of that sub-image, and its corresponding text category label and category confidence can be determined. For example, sub-image 1 may include a portion of a person and a house. By comparison, it can be seen that sub-image 1 contains a person as the main subject, the text category label is "person," and the category confidence can be 80%, etc.

[0043] In some embodiments, a target detection algorithm can be used to detect the subject object and obtain its bounding box, text category label, and category confidence score. The target detection algorithm can be an open-source lightweight target detection algorithm, such as the YOLOX_tiny algorithm, or other algorithms. There are no specific limitations here, as long as the bounding box, text category label, and category confidence score can be obtained.

[0044] YOLOX_tiny is an open-source, lightweight object detection algorithm. For example, YOLOX_tiny can be YOLOv3-YOLOv5 or a standard YOLOX model; this example uses YOLOv5. YOLOv5 mainly consists of four parts: the input layer, the backbone network (Backbone), the Neck layer, and the prediction layer (Prediction). The backbone network is a convolutional neural network that aggregates and forms image features at different fine-grained levels. The Neck layer includes a series of network layers that mix and combine image features, passing these features to the prediction layer for fusing features from different layers to detect the target. The prediction layer is used to predict the image features, generating bounding boxes and predicting the class.

[0045] refer to Figure 3 As shown in the diagram, based on the specific structure of the object detection model, the feature vector is first obtained by performing a convolution operation on the image to be processed through the backbone network Backone. The feature vector is then fused through the Neck layer to obtain an intermediate feature vector, which is then transmitted to the prediction layer to improve the comprehensiveness and accuracy of the feature vector. Next, the intermediate feature vector is fitted through the prediction layer to obtain the output result, which can be the detection box, text category label, and category confidence of each subject object contained in the image to be processed.

[0046] For example, refer to Figure 4A The image to be processed shown includes bounding boxes for the main objects obtained by subject detection. These bounding boxes can be represented by pixel coordinates, specifically the top-left and top-right pixel coordinates. For example, the bounding box coordinates for main object 1 are (x0, y0) and (x1, y1), and the bounding box coordinates for main object 2 are (x2, y2) and (x3, y3). Figure 4B To perform subject detection on the image to be processed, the text category label and category confidence score of subject object 1 are obtained. For example, the text category label of subject object 1 is "person", and the category confidence score is 86%. Figure 4C To perform subject detection on the image to be processed, the text category label and category confidence score of subject object 2 are obtained. For example, the text category label of subject object 2 is "dog" and the category confidence score is 97%.

[0047] In this embodiment of the disclosure, the YOLOv5 algorithm can quickly detect all subject objects in the image to be processed, and output the coordinate pixel information of the detection box of each subject object, the text category label, and the category confidence score, thereby improving the accuracy of detection.

[0048] Next, refer to Figure 2 As shown, in step S230, the target text category label is determined based on the category confidence level and the user selection operation applied to the text category label.

[0049] In this embodiment of the disclosure, the target text category label can be determined by combining category confidence and user selection operations applied to one or more text category labels. The number of target text category labels can be one or more. Furthermore, the one or more subject objects corresponding to the selected target text category label can be processed by image cutout, thereby obtaining the target subject object by combining the user selection operation, achieving interactive image cutout.

[0050] The target subject object can include one or more subject objects, specifically determined by the number of target text category tags selected by the user's selection operation. For example, when the user's selection operation applies to multiple text category tags, the number of target subject objects can be multiple. When the user's selection operation applies to a single text category tag, the number of target subject objects can be one; no specific limitation is made here.

[0051] In some embodiments, the specific steps for determining the target text category label, combining the category confidence score and the user selection operation on the text category label, may include: determining at least one sub-image based on the detection box; determining candidate sub-images from the at least one sub-image based on the category confidence score of each sub-image; and further determining the target text category label to be extracted from the range of text category labels corresponding to the candidate sub-images based on the user selection operation.

[0052] First, after obtaining the detection bounding box of the main object, sub-images for separating the main object can be determined based on the detection bounding box. Each sub-image contains only one main object, and may also include background images, etc. Candidate sub-images refer to the sub-images containing the main object that can be cut out; that is, they represent the range of the main object that can be cut out. Candidate sub-images can be a portion of at least one sub-image or all sub-images, depending on the category confidence.

[0053] For example, candidate sub-images can be selected by comparing the category confidence score with a confidence threshold. Specifically, different methods can be used to determine candidate sub-images based on the comparison results. If the category confidence score of a sub-image is greater than or equal to the confidence threshold, the sub-image can be identified as a candidate sub-image to facilitate subsequent image matting. Furthermore, the text category label corresponding to this sub-image can be displayed on the user interface for user selection. If the category confidence score is greater than the confidence threshold, the target text category label can be determined by combining the user's selection of the text category label corresponding to the candidate sub-image, and the image to be processed can be matted within the candidate sub-image range to obtain the corresponding target subject object.

[0054] For example, a target text category label can be determined based on a user selection action. This target text category label can be one or more text category labels whose category confidence is greater than a confidence threshold, specifically determined by the user's selection action on these text category labels. The user selection action can include one or more of touch operations and voice-triggered operations.

[0055] If the comparison result shows that the category confidence score of the text category label of a sub-image is less than the confidence threshold, then the sub-image cannot be identified as a candidate sub-image. This means the sub-image cannot be used in the subsequent image cutout process, and its text category label will not be displayed on the user interface to avoid affecting the cutout process. Interactive image cutout is only performed based on the text category label displayed on the user interface. By not displaying the text category labels of sub-images with a category confidence score less than the confidence threshold, invalid cutout can be avoided, improving the accuracy of the cutout.

[0056] Alternatively, text category labels with a category confidence score below a confidence threshold can be displayed on the user interface. In this case, candidate sub-images are determined based on the target method, and the target text category label is determined based on the default method or user response. The target method can either exclude sub-images from the candidate list or include them as candidate sub-images, depending on the specific requirements. Different target methods will result in different methods for obtaining the target text category label.

[0057] For example, if sub-images with a category confidence score lower than the confidence threshold are not considered as candidate sub-images, the target text category label can be determined using the default method. Specifically, if the user selects a text category label with a category confidence score lower than the confidence threshold, it can be excluded from the target text category label, and the subject object corresponding to that text category label will not be output, i.e., image matting will not be performed.

[0058] If sub-images with a category confidence score below a confidence threshold are considered candidate sub-images, the target text category label can be determined through user response actions. Specifically, a prompt message can be provided on the user interface to remind the user that the category confidence score of the sub-image is low, and the user response action determines whether to determine the text category label of the sub-image as the target text category label. The user response action can be a selection action of the user on the prompt message. The prompt message can be, for example, "Continue" or "Cancel". When the user response action is a click on the "Continue" control, the text category labels with a category confidence score below the confidence threshold can be cut out; when the user response action is a click on the "Cancel" control, the cutting out of the text category labels with a category confidence score below the confidence threshold can be stopped.

[0059] For example, if all text category labels are displayed on the user interface, and the user selects text category label 1, but the category confidence of the sub-image corresponding to text category label 1 is less than the confidence threshold, then the sub-image corresponding to text category label 1 will not be cut out. Alternatively, if the user selects text category label 1, but the category confidence of the sub-image corresponding to text category label 1 is less than the confidence threshold, a prompt message can be displayed around the text category label. This prompt message includes two controls: "Continue" or "Cancel." If the user selects the "Continue" control, then the sub-image corresponding to text category label 1 will be cut out.

[0060] In this embodiment of the disclosure, candidate sub-images are determined by comparing the category confidence of the text category label with the confidence threshold, and the target text category label is determined based on the user's selection operation of the text category label of the candidate sub-image. This avoids the misoperation caused by the user possibly selecting the text category label with a low category confidence, and improves the accuracy and effectiveness of image matting.

[0061] In this embodiment of the disclosure, by combining the comparison results of category confidence and confidence threshold, different methods are used to determine whether all text category labels are identified as target text category labels for image matting. This can improve the accuracy of identifying target text category labels and the accuracy of identifying matting objects, avoid the problem of poor image quality caused by overall image matting, expand the application scope, realize personalized image matting, and improve the comprehensiveness and richness of image matting.

[0062] In some embodiments, one or more initial subject objects can be determined from the image to be processed using detection boxes, i.e., preliminary image matting can be performed using detection boxes. For example, all sub-images corresponding to the detection boxes can be determined as initial subject objects. Further, one or more of the initial subject objects can be used as target subject objects based on determined target text category labels.

[0063] The specific steps for matting the subject object corresponding to any text category label can include: after determining the target text category label through user selection, performing matting processing using a segmentation model to obtain the target subject object corresponding to the target text category label. For example, the target text category label, detection boxes, and one or more of the image to be processed can be used as input to the segmentation model. The segmentation model processes the input to obtain the target subject object corresponding to the target text category label from the image to be processed. The target text category label can be one or more text labels, specifically determined from multiple text category labels displayed on the user interface based on user selection. Alternatively, the target text category label, pixel information of the image to be processed, and detection boxes can be used as input to the segmentation model. The segmentation model performs feature extraction to obtain the segmentation result. The segmentation result can be the target subject object corresponding to the target text category label.

[0064] In some embodiments, the segmentation model can be SAM (Segment Anything Model), or any other type of segmentation model. Here, we will use SAM as an example. The segmentation model takes one or more of the multimodal data as input and segments out the main object specified in the text. The SAM model establishes a large-scale image segmentation model that can accept text prompts and achieves generalization ability based on training on massive amounts of data.

[0065] Based on this, when performing image cutout on each target text category label, the target text category label, the detection box, and the pixel information in the image to be processed can be combined to perform image cutout processing on the image to be processed, thereby obtaining the target subject object corresponding to the user's selection operation. (Reference) Figure 5 As shown, the main steps include:

[0066] In step S510, the image to be processed is encoded to obtain image features;

[0067] In step S520, the corresponding encoded features are obtained by combining the target text category label, the detection box, and the pixel information of the image to be processed;

[0068] In step S530, based on image features, the encoded features are decoded to obtain the corresponding decoding results, and the target subject object of each target text category label is determined according to the decoding results.

[0069] In this embodiment of the disclosure, reference is made to Figure 6 As shown, the first step is to extract the encoded features of the image to be processed. For example, the image to be processed can be encoded using a first encoder to obtain image features. The first encoder can be an image encoder, such as a Transformer encoder or a Vision Transformer model. Next, one or more of the following can be transmitted as input data to a second encoder: the target text category label, the detection box, and the pixel information of the image to be processed, to extract the encoded features of each input data, thereby obtaining one or more encoded features. Here, the detection box can be the top-left and bottom-right corners of the detection box, the pixel information of the image to be processed can be the pixel position of each pixel in the image to be processed, and the target text category label can be text data. The second encoder can be a cue encoder, used to encode multimodal data. Each input data can correspond to one encoded feature.

[0070] It should be noted that data from only one modality can be used as input data, or data from multiple modalities can be used simultaneously. For example, any one of the target text category label, the detection box, and the pixel information of the image to be processed can be used as input data and encoded by the second encoder to obtain the corresponding encoded features. Alternatively, any two or three of the target text category label, the detection box, and the pixel information of the image to be processed can be simultaneously used as input data to the second encoder for encoding, obtaining their respective corresponding encoded features.

[0071] After obtaining the encoded features, a decoder can be used to decode each encoded feature based on the image features, yielding a decoding result for each input data point. Each decoding result corresponds to a confidence score, which represents the accuracy of the decoding result. The confidence score is positively correlated with accuracy; that is, the higher the confidence score, the higher the accuracy.

[0072] After obtaining one or more decoding results, the target subject object corresponding to the target text category label of the user's selected operation can be determined from the one or more decoding results based on the confidence scores corresponding to the one or more decoding results. For example, the decoding result with the highest confidence score can be determined as the target subject object corresponding to the target text category label.

[0073] In some embodiments, reference Figure 7A As shown, the target text category label can be human or dog. When the input data only contains the target text category label, the decoding result corresponding to each target text category label can be obtained. Furthermore, the decoding result corresponding to each target text category label can be directly used as the target subject object corresponding to that target text category label.

[0074] refer to Figure 7B As shown, the target text category label can be "dog". When the input data includes the image to be processed, the detection box, and the target text category label, the decoding results for each input data corresponding to each target text category label can be obtained, namely decoding result 1, decoding result 2, and decoding result 3. Since there are multiple decoding results corresponding to each target text category label, the decoding result with the highest confidence score for each target text category label can be directly taken as the target subject object corresponding to that target text category label. For example, for the target text category label "dog", decoding result 2 has the highest confidence score, so decoding result 2 can be taken as the target subject object corresponding to the target text category label "dog".

[0075] It should be noted that, to improve accuracy, the segmentation model can be trained to improve its performance. For example, the model parameters can be adjusted using a dataset to minimize the loss function, thereby achieving further training and optimization.

[0076] In some embodiments, by inputting the target text category label, the image to be processed, and one or more items in the detection bounding box into the segmentation model, the target subject object corresponding to the target text category label can be obtained. Furthermore, different text labels input into the MTTR will yield different subject segmentation results. For example, refer to... Figure 8 As shown, inputting "person" yields the segmentation result of a human image, meaning the target subject is a person; inputting "dog" yields the segmentation result of a dog, meaning the target subject is a dog; inputting both "person" and "dog" yields the segmentation result of both a person and a dog, meaning the target subjects are both a person and a dog.

[0077] In some embodiments, if multiple bounding boxes, text category labels, and category confidence scores are obtained through an object detection model, for example, the text category label for object 1 is "person" with a category confidence score of 80%; the text category label for object 2 is "dog" with a category confidence score of 90%; and the text category label for object 3 is "vehicle" with a category confidence score of 50%, and the category confidence score is 75%, by comparing the category confidence score of each text category label with a confidence threshold, it can be determined that the category confidence scores of object 1 and object 2 are greater than the confidence threshold, while the category confidence score of object 3 is less than the confidence threshold. Since its category confidence score is less than the confidence threshold, the sub-images corresponding to object 1 and object 2 can be used as candidate sub-images, thereby extracting the target object within the range of the candidate sub-images. Simultaneously, the text category labels corresponding to object 1 and object 2 can be displayed on the user interface for the user to select. When a user selection operation is detected, the target object can be determined based on the object corresponding to the target text category label affected by the user selection operation. It's important to note that by only displaying subject objects with a category confidence score greater than the confidence threshold on the user interface, accidental selection of subjects that cannot be cut out is avoided, thus improving accuracy. Furthermore, by combining user selection actions, the system can cut out specific subject objects, avoiding the limitations of only being able to cut out the entire image or not being able to specify the subject, increasing the scope and diversity of applications, and improving the user experience.

[0078] In this embodiment, when the category confidence of the text category label is greater than a confidence threshold, the target text category label can be determined based on the user's selection operation. Then, the target subject object can be determined from the sub-images of the image to be processed based on the target text category label. Since the number of target subject objects obtained by image matting can be determined based on the user's selection operation, it avoids the limitation of only being able to matte in related technologies. Figure 1 The limitations of individual subject objects are addressed. However, since the user can freely select the text category label to be cut out, the target subject object can be determined according to actual needs. This allows for specified cutout based on requirements and user preferences, increasing the range of cutout options. Furthermore, after determining the target text category label, segmentation using a segmentation model based on the multimodal data represented by the image to be processed and the target text category label improves the accuracy of subject segmentation.

[0079] In some embodiments, the target subject can be selectively extracted from the image to be processed, and the target subject can be further composited, for example, to create emoticons; or the target subject can be composited with reference materials, thereby realizing various types of image creation, etc.

[0080] Figure 9 The flowchart for image matting is illustrated in the image below. (See reference) Figure 9 As shown, the main steps include:

[0081] In step S902, the image to be processed is input.

[0082] In step S904, the object detection algorithm is used to detect the subject in the image to be processed.

[0083] In step S906, the detection box of the main object, the text label category, and the category confidence are obtained.

[0084] In step S908, a sub-image is obtained based on the detection box.

[0085] In step S910, it is determined whether the category confidence of the text category label of the sub-image is greater than the confidence threshold. If yes, proceed to step S912. If no, proceed to step S920.

[0086] In step S912, text category labels are displayed on the user interface.

[0087] In step S914, the target text category label is determined through a user selection operation. The target text label can be, for example, "person" and / or "dog".

[0088] In step S916, the target text category label, text box, and image to be processed are input into the segmentation model.

[0089] In step S918, the target object corresponding to the target text category label is obtained by image extraction.

[0090] In step S920, the text category label is not displayed.

[0091] In this embodiment, an interactive and selectable multi-subject automated image matting method is achieved through user selection of text category labels, increasing versatility and better aligning with user preferences. The accuracy of the matted subject objects is improved through object detection algorithms and segmentation models. Furthermore, since the target subject object corresponding to the target text category label can be obtained through detection boxes and target text category labels, and since selection can be made based on confidence scores from multiple dimensions of decoding results, both comprehensiveness and accuracy of the target subject object are enhanced.

[0092] This disclosure provides an image processing apparatus, with reference to... Figure 10 As shown, the image processing device 1000 may include: an image acquisition module 1001, a subject detection module 1002, a label determination module 1003, and an image matting module 1004, wherein:

[0093] Image acquisition module 1001 is used to acquire the image to be processed;

[0094] The subject detection module 1002 is used to perform subject detection on the image to be processed, and to obtain the detection box, text category label and category confidence of the subject object contained in the image to be processed;

[0095] The label determination module 1003 is used to determine the target text category label based on the category confidence level and the user selection operation performed on the text category label;

[0096] The image matting module 1004 is used to combine the target text category label, the detection box, and the image to be processed to perform matting processing on the image to be processed, and obtain the target subject object corresponding to the user selection operation.

[0097] In one exemplary embodiment of this disclosure, the image matting module is configured to perform: determining an initial subject object from the image to be processed using a detection box; and determining a target subject object in the initial subject object based on a target text category label.

[0098] In one exemplary embodiment of this disclosure, the number of target text category labels is at least one; the image matting module includes: an encoding module for encoding the image to be processed to obtain image features; a feature extraction module for combining the target text category labels, detection boxes, and pixel information in the image to be processed to obtain corresponding encoded features; and a decoding module for decoding the encoded features based on the image features to obtain corresponding decoding results, and determining the target subject object based on the decoding results.

[0099] In one exemplary embodiment of this disclosure, the number of decoding results is one or more; the decoding module includes: a first target subject object determination module, configured to determine the decoding result with the highest confidence score as the target subject object for each target text category label when the number of decoding results is multiple; and a second target subject object determination module, configured to determine the decoding result as the target subject object for each target text category label when the number of decoding results is one.

[0100] In one exemplary embodiment of this disclosure, the subject detection module is configured to perform: extracting features from the image to be processed to obtain a feature vector, and performing a convolution operation on the feature vector to obtain an intermediate feature vector; predicting the intermediate feature vector to obtain the detection box, text category label, and category confidence of the subject object contained in the image to be processed.

[0101] In one exemplary embodiment of this disclosure, the label determination module includes: a confidence determination module, used to determine the text category label of the sub-image corresponding to the detection box and the category confidence of the sub-image; and a label selection module, used to determine the target text category label based on the category confidence of the sub-image and the user selection operation.

[0102] In one exemplary embodiment of this disclosure, the label selection module is configured to: determine the comparison result of the category confidence score and the confidence score threshold of the text category label of the sub-image; if the comparison result is that the category confidence score is greater than the confidence score threshold, select the sub-image as a candidate sub-image, and determine the target text category label based on the user selection operation among the text category labels corresponding to the candidate sub-image; if the comparison result is that the category confidence score is less than the confidence score threshold, determine the candidate sub-image according to the target method, and determine the target text category label based on the default method or the user response operation among the text category labels corresponding to the candidate sub-image.

[0103] It should be noted that the specific details of each part of the above-mentioned image processing apparatus have been described in detail in the implementation of the image processing method section. For any undisclosed details, please refer to the implementation of the method section, and therefore will not be repeated here.

[0104] Exemplary embodiments of this disclosure also provide an electronic device. This electronic device may be the terminal described above. Generally, the electronic device may include a processor and a memory, the memory being used to store executable instructions of the processor, and the processor being configured to perform the image processing method described above by executing the executable instructions.

[0105] The following is based on Figure 11 Taking the mobile terminal 1100 as an example, the construction of this electronic device will be described by way of example. Those skilled in the art will understand that, apart from components specifically designed for mobile purposes, Figure 11 The structure can also be applied to fixed types of equipment.

[0106] like Figure 11 As shown, the mobile terminal 1100 may specifically include: a processor 1101, a memory 1102, a bus 1103, a mobile communication module 1104, an antenna 1, a wireless communication module 1105, an antenna 2, a display screen 1106, a camera module 1107, an audio module 1108, a power module 1109, and a sensor module 1110.

[0107] Processor 1101 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, an encoder, a decoder, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). The method in this exemplary embodiment can be executed by an AP, GPU, or DSP. When the method involves neural network-related processing, it can be executed by an NPU. For example, the NPU can load neural network parameters and execute neural network-related algorithm instructions.

[0108] An encoder encodes (compresses) images or videos to reduce data size for easier storage or transmission. A decoder decodes (decompresses) the encoded data to restore the original image or video data. The mobile terminal 1100 can support one or more encoders and decoders, such as image formats like JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), and BMP (Bitmap), and video formats like MPEG (Moving Picture Experts Group) 1, MPEG10, H.1063, H.1064, and HEVC (High Efficiency Video Coding).

[0109] The processor 1101 can be connected to the memory 1102 or other components via the bus 1103.

[0110] The memory 1102 can be used to store computer executable program code, which includes instructions. The processor 1101 executes various functional applications and data processing of the mobile terminal 1100 by running the instructions stored in the memory 1102. The memory 1102 can also store application data, such as images, videos, and other files.

[0111] The communication function of mobile terminal 1100 can be implemented through mobile communication module 1104, antenna 1, wireless communication module 1105, antenna 2, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 1104 can provide 3G, 4G, 5G and other mobile communication solutions for mobile terminal 1100. Wireless communication module 1105 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for mobile terminal 1100.

[0112] The display screen 1106 is used to implement display functions, such as displaying user interfaces, images, and videos. The camera module 1107 is used to implement shooting functions, such as capturing images and videos, and may include a color temperature sensor array. The audio module 1108 is used to implement audio functions, such as playing audio and capturing voice. The power module 1109 is used to implement power management functions, such as charging the battery, supplying power to the device, and monitoring battery status. The sensor module 1110 may include one or more sensors to implement corresponding sensing and detection functions. For example, the sensor module 1110 may include an inertial sensor, which is used to detect the motion posture of the mobile terminal 1100 and output inertial sensing data.

[0113] It should be noted that the present disclosure also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist alone and not assembled into the electronic device.

[0114] Computer-readable storage media can be, for example—but not limited to—electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0115] A computer-readable storage medium can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof. The computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to perform the methods described in the following embodiments.

[0116] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0117] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0118] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0119] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0120] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: Obtain the image to be processed; Subject detection is performed on the image to be processed to obtain the detection box of the subject object contained in the image to be processed, the text category label corresponding to the detection box, and the category confidence. The target text category label is determined based on the category confidence level and the user selection action performed on the text category label. By combining the target text category label, the detection box, and the image to be processed, the image to be processed is subjected to image cutout processing to obtain the target subject object corresponding to the user selection operation; Wherein, the number of target text category tags is at least one; the step of combining the target text category tags, the detection box, and the image to be processed to perform image cutout processing on the image to be processed to obtain the target subject object corresponding to the user selection operation includes: The image to be processed is encoded to obtain image features; By combining the target text category label, the detection box, and the pixel information of the image to be processed, the corresponding encoded features are obtained; Based on the image features, the encoded features are decoded to obtain the corresponding decoding results; When there are multiple decoding results, the decoding result with the highest confidence score is determined as the target subject object for each target text category label; when there is only one decoding result, the decoding result is determined as the target subject object for each target text category label.

2. The image processing method according to claim 1, characterized in that, The step of combining the target text category label, the detection box, and the image to be processed to perform image cutout processing on the image to be processed, and obtaining the target subject object corresponding to the user's selection operation, includes: The initial subject object is determined from the image to be processed using the detection box; The target subject object is determined from the initial subject object based on the target text category label.

3. The image processing method according to claim 1, characterized in that, The step of performing subject detection on the image to be processed, and obtaining the detection bounding box of the subject object contained in the image to be processed, the text category label corresponding to the detection bounding box, and the category confidence score, includes: The image to be processed is subjected to feature extraction to obtain feature vectors, and the feature vectors are then convolved to obtain intermediate feature vectors. The intermediate feature vector is predicted to obtain the detection box of the main object contained in the image to be processed, the text category label, and the category confidence.

4. The image processing method according to claim 1, characterized in that, The step of determining the target text category label based on the category confidence level and the user selection operation applied to the text category label includes: Determine the text category label of the sub-image corresponding to the detection box and the category confidence of the sub-image; The target text category label is determined based on the category confidence of the sub-image and the user's selection operation.

5. The image processing method according to claim 4, characterized in that, Determining the target text category label based on the category confidence score of the sub-image and the user's selection operation includes: The comparison results of the category confidence and confidence threshold of the text category labels of the sub-image are determined; If the comparison result shows that the confidence level of the category is greater than the confidence threshold, the sub-image is used as a candidate sub-image, and the target text category label is determined based on the user selection operation from the text category labels corresponding to the candidate sub-image. If the comparison result is that the confidence level of the category is less than the confidence level threshold, candidate sub-images are determined according to the target method, and the target text category label is determined based on the default method or user response operation in the text category labels corresponding to the candidate sub-images.

6. An image processing apparatus, characterized in that, include: The image acquisition module is used to acquire the image to be processed. The subject detection module is used to perform subject detection on the image to be processed, and to obtain the detection box of the subject object contained in the image to be processed, the text category label corresponding to the detection box, and the category confidence. The label determination module is used to determine the target text category label based on the category confidence level and the user selection operation performed on the text category label; The image matting module is used to combine the target text category label, the detection box and the image to be processed to perform matting processing on the image to be processed, and obtain the target subject object corresponding to the user selection operation; Wherein, the number of target text category tags is at least one; the step of combining the target text category tags, the detection box, and the image to be processed to perform image cutout processing on the image to be processed to obtain the target subject object corresponding to the user selection operation includes: The image to be processed is encoded to obtain image features; By combining the target text category label, the detection box, and the pixel information of the image to be processed, the corresponding encoded features are obtained; Based on the image features, the encoded features are decoded to obtain the corresponding decoding results; When there are multiple decoding results, the decoding result with the highest confidence score is determined as the target subject object for each target text category label; when there is only one decoding result, the decoding result is determined as the target subject object for each target text category label.

7. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the image processing method according to any one of claims 1-5 by executing the executable instructions.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image processing method according to any one of claims 1-5.

Images