Image processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to an image processing method and device, computer equipment, a readable storage medium and a program product. The method comprises the steps of performing target detection on a to-be-labeled image, and determining a first detection area of a detection object in the to-be-labeled image; based on the to-be-labeled image, performing visual segmentation on the first detection area to obtain a target contour of the detection object; based on the target contour, performing semantic category detection on the detection object, and determining a target semantic category of the detection object; and based on the target contour and the target semantic category, obtaining segmentation data of the detection object, the segmentation data including the target contour carrying the target semantic category. By adopting the method, the accuracy of image processing can be improved.

Description

Technical Field

[0001] The present application relates to the field of imaging technology, and in particular to an image processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] In recent years, with the rapid development and widespread application of deep learning visual perception technology, the current mainstream supervised learning methods usually rely on a large amount of manually annotated image data, especially the instance segmentation algorithms used in the field of commercial service robots.

[0003] In traditional image processing methods, manually annotated image data is learned, and the image is processed based on the model obtained after learning. However, the manually annotated image data often has some incorrect annotations, which makes the model obtained after learning unable to accurately process the image, resulting in the problem of low image processing accuracy. Summary of the invention

[0004] Based on this, it is necessary to provide an image processing method, apparatus, computer device, computer-readable storage medium and computer program product that can improve accuracy in response to the above technical problems.

[0005] In a first aspect, the present application provides an image processing method, comprising:

[0006] Performing target detection on the image to be labeled, and determining a first detection area of ​​the detection object in the image to be labeled;

[0007] Based on the image to be annotated, visually segment the first detection area to obtain a target contour of the detection object;

[0008] Based on the target contour, performing semantic category detection on the detection object to determine the target semantic category of the detection object;

[0009] Based on the target contour and the target semantic category, segmentation data of the detection object is obtained, and the segmentation data includes the target contour carrying the target semantic category.

[0010] In one embodiment, before performing target detection on the image to be labeled and determining a first detection area of ​​the detection object in the image to be labeled, the method further includes:

[0011] Acquire first feature data from the image to be annotated, and perform semantic category annotation on the first feature data to obtain second feature data;

[0012] Using the second feature data to adjust parameters of the target detection model to obtain an adjusted target detection model;

[0013] The step of performing target detection on the image to be annotated and determining a first detection area of ​​the detection object in the image to be annotated includes:

[0014] The adjusted target detection model is used to perform target detection on the image to be labeled, and a first detection area of ​​the detection object in the image to be labeled is determined.

[0015] In one embodiment, obtaining first feature data from the image to be labeled includes:

[0016] For the image to be annotated, extracting original feature data of the image to be annotated;

[0017] Performing clustering processing on the original feature data to obtain at least one target cluster;

[0018] The first feature data is obtained from the at least one target cluster.

[0019] In one embodiment, the step of adjusting parameters of a target detection model using the second feature data to obtain an adjusted target detection model includes:

[0020] Determining a first network structure whose parameters are to be adjusted from the target detection model;

[0021] In the target detection model, the second feature data is used to adjust parameters of the first network structure to obtain an adjusted target detection model.

[0022] In one embodiment, after the semantic category labeling of the first feature data is performed to obtain the second feature data, the method further includes:

[0023] Using the second feature data to adjust parameters of the classification model to obtain an adjusted classification model;

[0024] The performing semantic category detection on the detection object based on the target contour to determine the target semantic category of the detection object includes:

[0025] Based on the target contour, semantic category detection is performed on the detection object through the adjusted classification model to determine the target semantic category of the detection object.

[0026] In one embodiment, the performing semantic category detection on the detection object based on the target contour to determine the target semantic category of the detection object includes:

[0027] Acquire a second detection area of ​​the target contour in the image to be labeled;

[0028] Perform semantic category detection on the second detection area to determine the target semantic category of the detection object.

[0029] In a second aspect, the present application further provides an image processing device, comprising:

[0030] A target detection module, used to perform target detection on the image to be annotated, and determine a first detection area of ​​the detection object in the image to be annotated;

[0031] A visual segmentation module, used to perform visual segmentation on the first detection area based on the image to be annotated, so as to obtain a target contour of the detection object;

[0032] A semantic category detection module, used to perform semantic category detection on the detection object based on the target contour, and determine the target semantic category of the detection object;

[0033] A segmentation data determination module is used to obtain segmentation data of the detection object based on the target contour and the target semantic category, wherein the segmentation data includes the target contour carrying the target semantic category.

[0034] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] Performing target detection on the image to be labeled, and determining a first detection area of ​​the detection object in the image to be labeled;

[0036] Based on the image to be annotated, visually segment the first detection area to obtain a target contour of the detection object;

[0037] Based on the target contour, performing semantic category detection on the detection object to determine the target semantic category of the detection object;

[0038] Based on the target contour and the target semantic category, segmentation data of the detection object is obtained, and the segmentation data includes the target contour carrying the target semantic category.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0040] Performing target detection on the image to be labeled, and determining a first detection area of ​​the detection object in the image to be labeled;

[0041] Based on the image to be annotated, visually segment the first detection area to obtain a target contour of the detection object;

[0042] Based on the target contour, performing semantic category detection on the detection object to determine the target semantic category of the detection object;

[0043] Based on the target contour and the target semantic category, segmentation data of the detection object is obtained, and the segmentation data includes the target contour carrying the target semantic category.

[0044] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0045] Performing target detection on the image to be labeled, and determining a first detection area of ​​the detection object in the image to be labeled;

[0046] Based on the image to be annotated, visually segment the first detection area to obtain a target contour of the detection object;

[0047] Based on the target contour, performing semantic category detection on the detection object to determine the target semantic category of the detection object;

[0048] Based on the target contour and the target semantic category, segmentation data of the detection object is obtained, and the segmentation data includes the target contour carrying the target semantic category.

[0049] The above-mentioned image processing method, apparatus, computer equipment, computer-readable storage medium and computer program product perform target detection on the image to be labeled and determine the first detection area of ​​the detection object in the image to be labeled; based on the image to be labeled, the first detection area is visually segmented to more accurately obtain the target contour of the detection object, then, based on the target contour of the detection object, semantic category detection is performed on the detection object to more accurately determine the target semantic category of the detection object, and then based on the target contour and the target semantic category, segmentation data of the detection object can be more accurately obtained, and the segmentation data includes the target contour carrying the target semantic category, thereby improving the accuracy of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 is a schematic flow chart of an image processing method in one embodiment;

[0052] Figure 2 A schematic flow chart of a step of visually segmenting a first detection area based on an image to be annotated to obtain a target contour of a detection object in an embodiment;

[0053] Figure 3 A schematic diagram of fine-tuning parameters of a first network structure in a target detection model in one embodiment;

[0054] Figure 4 is a schematic flow chart of an image processing method in another embodiment;

[0055] Figure 5 is a structural block diagram of an image processing device in one embodiment;

[0056] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] In one embodiment, Figure 1 As shown, an image processing method is provided. This embodiment uses the method applied to a computer device as an example for illustration. The computer device can be a terminal or a server. It can be understood that the method can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be, but is not limited to, various robots, unmanned vehicles, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The robot can be a cleaning robot, a distribution robot, a logistics robot, an inspection robot and a disinfection robot. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0059] In this embodiment, the image processing method includes the following steps:

[0060] Step S102: performing target detection on the image to be labeled, and determining a first detection area of ​​the detection object in the image to be labeled.

[0061] The detection object is an object of interest in the image to be annotated. Exemplarily, the detection object in the image to be annotated may be ground stains, human faces, arms, animals, buildings or other objects, but is not limited thereto. The first detection area is an area of ​​the detection object in the image to be annotated. The first detection area may be a rectangular frame, a circle or an area of ​​other shapes including the detection object.

[0062] The computer device obtains at least two images to be labeled.

[0063] Optionally, the computer device calls a target detection model, performs target detection on the image to be annotated through the target detection model, and determines a first detection area of ​​the detection object. The target detection model may be a model that has been trained or has parameters adjusted.

[0064] Step S104: Based on the image to be annotated, visual segmentation is performed on the first detection area to obtain a target contour of the detection object.

[0065] The target contour is the contour of the detection object in the image to be labeled, and the area of ​​the target contour is smaller than the area of ​​the first detection area.

[0066] Optionally, the computer device inputs the image to be annotated and the first detection area into a basic visual segmentation model, and uses the basic visual segmentation model to visually segment the first detection area based on the image to be annotated to obtain a target contour of the detection object. The basic visual segmentation model (SAM, Segment Anything Model) can be a model that has been trained or has its parameters adjusted.

[0067] The first detection area is used as a prompt area, so that the basic visual segmentation model outputs a fine target contour within the first detection area.

[0068] like Figure 2 As shown, the basic visual segmentation model includes a hint area encoder, an image visual feature encoder and a contour mask decoder. The computer device inputs the first detection area and the image to be labeled into the basic visual segmentation model, and the hint area encoder encodes the first detection area as the hint area and then inputs it into the contour mask decoder. The image to be labeled is encoded by the image visual feature encoder and then input into the contour mask decoder. The contour mask decoder decodes the encoded first detection area and the image to be labeled to obtain the target contour of the detection object.

[0069] The computer device trains the basic visual segmentation model, including: obtaining a training image and a training detection area in the training image, inputting the training image and the training detection area into the basic visual segmentation model, and training the basic visual segmentation model until a trained basic visual segmentation model is obtained.

[0070] Step S106: Based on the target contour, semantic category detection is performed on the detection object to determine the target semantic category of the detection object.

[0071] The target semantic category is the semantic category of the detection object. For example, if the detection object is a human face, the target semantic category of the detection object can be a human face, and further can be a smiling face or a crying face, etc.; if the detection object is an animal, the target semantic category of the detection object can be an animal, and further can be a dog, a cat, a sheep, or a cow, etc.

[0072] Optionally, the computer device calls a classification model, and performs semantic category detection on the detection object based on the target contour through the classification model to determine the target semantic category of the detection object. The classification model may be a model that has been trained or has parameters adjusted.

[0073] Optionally, based on the target contour, semantic category detection is performed on the detection object to determine the target semantic category of the detection object, including: obtaining a second detection area of ​​the target contour in the image to be labeled; and performing semantic category detection on the second detection area to determine the target semantic category of the detection object.

[0074] The second detection area is the detection area of ​​the target contour in the image to be labeled. The second detection area can be a circumscribed rectangle of the target contour in the image to be labeled, or a circumscribed circle of the target contour in the image to be labeled, but is not limited thereto.

[0075] The computer device calls the classification model, performs semantic category detection on the second detection area through the classification model, and determines the target semantic category of the detection object.

[0076] The classification model determines the scores of each candidate semantic category to which the second detection area belongs, and determines the target semantic category of the detection object based on the scores of each candidate semantic category. Optionally, the classification model can use the candidate semantic category with the highest score as the target semantic category of the detection object, or can use the candidate semantic category with the second highest score as the target semantic category of the detection object, which is not limited here.

[0077] Optionally, the computer device performs target detection on the image to be annotated, determines a first detection area of ​​the detection object in the image to be annotated, and the first detection area carries a first semantic category of the detection object; performs semantic category detection on the detection object based on the target contour, and determines a second semantic category of the detection object; compares the first semantic category and the second semantic category, and if the first semantic category and the second semantic category are consistent, uses the first semantic category as the target semantic category; if the first semantic category and the second semantic category are inconsistent, uses the second semantic category as the target semantic category.

[0078] Step S108, based on the target contour and the target semantic category, obtaining segmentation data of the detection object, the segmentation data including the target contour carrying the target semantic category.

[0079] The segmentation data is data segmented from the detection object in the image to be annotated, and the segmentation data includes a target contour of the detection object, and the target contour carries a target semantic category of the detection object.

[0080] Optionally, the computer device labels the target contour with a target semantic category to obtain segmentation data of the detected object.

[0081] The above-mentioned image processing method performs target detection on the image to be annotated and determines the first detection area of ​​the detection object in the image to be annotated; based on the image to be annotated, the first detection area is visually segmented to more accurately obtain the target contour of the detection object, then, based on the target contour of the detection object, semantic category detection is performed on the detection object to more accurately determine the target semantic category of the detection object, and then based on the target contour and the target semantic category, segmentation data of the detection object can be more accurately obtained, and the segmentation data includes the target contour carrying the target semantic category, thereby improving the accuracy of image processing.

[0082] In one embodiment, before performing target detection on the image to be labeled and determining the first detection area of ​​the detection object in the image to be labeled, the method further includes: acquiring first feature data from the image to be labeled, and performing semantic category annotation on the first feature data to obtain second feature data; adjusting parameters of a target detection model using the second feature data to obtain an adjusted target detection model; performing target detection on the image to be labeled and determining the first detection area of ​​the detection object in the image to be labeled, including: performing target detection on the image to be labeled using the adjusted target detection model to determine the first detection area of ​​the detection object in the image to be labeled.

[0083] The second feature data carries a marked semantic category.

[0084] Optionally, the computer device performs semantic category annotation at a weakly supervised level on the first feature data to obtain second feature data.

[0085] Optionally, obtaining the first feature data from the image to be labeled includes: extracting original feature data of the image to be labeled; performing clustering processing on the original feature data to obtain at least one target cluster; and obtaining the first feature data from the at least one target cluster.

[0086] Optionally, for the image to be annotated, feature extraction is performed on the image to be annotated by using the basic large model to obtain original feature data of the image to be annotated; the original feature data is a feature vector; k-means clustering is performed on the original feature data to obtain at least one target cluster; each target cluster includes at least one original feature; at least one first feature data is selected from at least one cluster. The at least one first feature data and the at least one second feature data are both small sample data sets.

[0087] Optionally, for each target cluster, the position of each original feature data is determined from the target cluster, and the original feature data closest to the center of the target cluster is determined as the first feature data.

[0088] Optionally, the computer device determines a first number of target clusters from at least one target cluster; obtains first feature data from the first number of target clusters; and the first number is less than a first threshold value. The first threshold value is a numerical threshold value specified by a user.

[0089] Optionally, the computer device counts the amount of original feature data in each target cluster, and determines at least one designated target cluster from at least one target cluster based on the amount of original feature data in each target cluster; the amount of original feature data in the designated target cluster is less than a second threshold value. The second threshold value is a numerical threshold value specified by a user.

[0090] For example, the computer device determines a designated target cluster from at least one target cluster, and the amount of original feature data in the designated target cluster is less than one percent of the amount of all images to be labeled.

[0091] For example, a computer device selects at least one important first feature data from at least one cluster, and obtains second feature data based on the important first feature data by labeling. The target detection model and classification model can be adjusted through a small amount of feature data, thereby performing target detection and semantic category detection more accurately.

[0092] Optionally, the second feature data is used to adjust the parameters of the target detection model to obtain an adjusted target detection model, including: determining a first network structure to be parameter adjusted from the target detection model; in the target detection model, the second feature data is used to adjust the parameters of the first network structure to obtain the adjusted target detection model.

[0093] The first network structure is a structure in the object detection model that needs to be parameter adjusted. The object detection model can be an open vocabulary object detection model. The open vocabulary object detection model refers to those models that can recognize and locate new categories of objects in the image that do not appear in the training set.

[0094] Optionally, the computer device determines a first network structure whose parameters are to be adjusted from the target detection model; fixes parameters of other network structures except the first network structure in the target detection model, and uses the second feature data to adjust the parameters of the first network structure to obtain an adjusted target detection model.

[0095] Exemplarily, the first network structure is a small part of the network structure in the target detection model, and the other network structures except the first network structure are the main structures of the target detection model. The computer device fine-tunes the parameters of the small part of the network structure and fixes the parameters of the main structure of the target detection model, so that the adjusted target detection model has the ability to detect the semantic category of the target, and the parameters of the target detection model can be adjusted more quickly, and a more accurate adjusted target detection model is obtained.

[0096] like Figure 3 As shown, the target detection model includes a text backbone network (Text Backbone), an image basic network structure (Image Backbone), a feature enhancement module (Feature Enhancer) and a first network structure.

[0097] The computer device obtains the text information "person, ball, racket, cat" and "a child in a red uniform", inputs the text information into the text backbone network, obtains text feature data (TextFeatures) in the text information through the text backbone network, and inputs the text feature data into the feature enhancement module; the feature enhancement module outputs fused text features (Fused Text Features).

[0098] The computer device obtains image information, inputs the image information into the image basic network structure, extracts image feature data (Image Features) from the image information, and inputs the image feature data into the feature enhancement module; the fused image features (Fused Image Features) are output through the feature enhancement module.

[0099] The computer device inputs the keys and values ​​(Keys & Values) of the fused text features into the language-guided query selection and the cross-modality decoder, and inputs the keys and values ​​(Keys & Values) of the fused image features into the language-guided query selection and the cross-modality decoder, respectively, performs cross-modal queries (Cross-Modality Queries) through the language-guided query selection, performs model outputs (Model Outputs) through the cross-modal decoder, calculates the localization loss (LocalizationLoss) of the output results, and adjusts the parameters of the first network structure based on the localization loss until an adjusted target detection model is obtained.

[0100] Optionally, after performing semantic category labeling on the first feature data to obtain the second feature data, the method further includes: using the second feature data to adjust parameters of a classification model to obtain an adjusted classification model; performing semantic category detection on the detection object based on the target contour to determine the target semantic category of the detection object, including: performing semantic category detection on the detection object based on the target contour using the adjusted classification model to determine the target semantic category of the detection object.

[0101] The classification model may be a multimodal CLIP classification model.

[0102] Optionally, the computer device uses the second feature data and the Lora fine-tuning technology to adjust the parameters of the classification model.

[0103] Optionally, a second network structure to be parameter-adjusted is determined from the classification model; in the classification model, the second characteristic data is used to adjust the parameters of the second network structure to obtain an adjusted classification model.

[0104] The second network structure is the structure that requires parameter adjustment in the classification model.

[0105] The computer device determines a second network structure to be parameter-adjusted from the classification model; fixes parameters of other network structures except the second network structure in the classification model, and adjusts the parameters of the second network structure using the second characteristic data to obtain an adjusted classification model.

[0106] Exemplarily, the computer device uses the second feature data to fine-tune some weight parameters of the multimodal CLIP classification model to obtain a classification model with high-precision classification in a specific task field. The specific task field can be set as needed.

[0107] In this embodiment, a computer device obtains first feature data from an image to be annotated, and performs semantic category annotation on the first feature data to obtain second feature data, and uses the second feature data to adjust parameters of a target detection model to obtain an adjusted target detection model. Then, by performing target detection on the image to be annotated through the adjusted target detection model, the first detection area of ​​the detection object can be more accurately determined.

[0108] Furthermore, for each image to be annotated, the original feature data of the image to be annotated is extracted, the original feature data is clustered to obtain at least one target cluster, and the first feature data is obtained from at least one target cluster. Then, semantic category annotation is performed based on the first feature data and the parameters of the model are adjusted, so that target detection and semantic category detection can be performed through the adjusted model, without the need to manually perform fine target contour annotation on all feature data, thereby improving the efficiency of image processing. Only a small amount of weak supervision information of the first feature data needs to be annotated, and the fine-tuning training of the target detection model and the classification model only needs to fine-tune a small number of parameters of the partial structure, rather than a large amount of finely annotated data and full training of the overall model structure parameters. The training time and calculation amount are greatly shortened, the overall labor cost, computing power cost, and processing cycle are greatly shortened, and the annotation quality is greatly improved.

[0109] In one embodiment, Figure 4 As shown, the computer device obtains the image to be annotated, extracts the original feature data through the feature model, and performs k-means clustering to obtain at least one target cluster, extracts the first feature data from each target cluster, performs semantic category annotation on the first feature data at a weak supervision level, and obtains the second feature data; fine-tunes the target detection model based on the second feature data to obtain the adjusted target detection model, and fine-tunes the CLIP multimodal model based on the second feature data to obtain the adjusted classification model; performs target detection on the image to be annotated through the adjusted target detection model to obtain the first detection area of ​​the detection object, and inputs the first detection area as a prompt area into the SAM model to obtain the target contour of the detection object; performs semantic category detection on the circumscribed rectangle of the target contour through the adjusted classification model to obtain the target semantic category of the detection object; then, based on the target contour and the target semantic category, high-quality segmentation data is obtained, and the segmentation data includes the target contour carrying the target semantic category. In this embodiment, a high-precision pre-annotation effect is achieved by combining multiple large models in steps, and more accurate segmentation data is obtained.

[0110] In one embodiment, an image processing method is also provided, which is applied to a computer device and includes the following steps:

[0111] Step A1: extracting original feature data of the image to be annotated; performing clustering processing on the original feature data to obtain at least one target cluster; and obtaining first feature data from the at least one target cluster.

[0112] Step A2: perform semantic category labeling on the first feature data to obtain second feature data.

[0113] Step A3, determining a first network structure to be parameter-adjusted from the target detection model; in the target detection model, adjusting the parameters of the first network structure using the second feature data to obtain an adjusted target detection model.

[0114] Step A4: Use the second feature data to adjust the parameters of the classification model to obtain an adjusted classification model.

[0115] Step A5: Perform target detection on the image to be labeled using the adjusted target detection model to determine a first detection area of ​​the detection object in the image to be labeled.

[0116] Step A6: Based on the image to be labeled, visually segment the first detection area to obtain the target contour of the detection object.

[0117] Step A7, obtaining a second detection area of ​​the target contour in the image to be annotated; performing semantic category detection on the second detection area using the adjusted classification model to determine the target semantic category of the detection object;

[0118] Step A8, based on the target contour and the target semantic category, obtaining segmentation data of the detection object, the segmentation data including the target contour carrying the target semantic category.

[0119] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0120] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

[0121] In an exemplary embodiment, Figure 5 As shown, an image processing device is provided, including: a target detection module 502, a visual segmentation module 504, a semantic category detection module 506 and a segmentation data acquisition module 508, wherein:

[0122] The target detection module 502 is used to perform target detection on the image to be labeled and determine a first detection area of ​​the detection object in the image to be labeled.

[0123] The visual segmentation module 504 is used to perform visual segmentation on the first detection area based on the image to be labeled to obtain the target contour of the detection object.

[0124] The semantic category detection module 506 is used to perform semantic category detection on the detection object based on the target contour, and determine the target semantic category of the detection object.

[0125] The segmentation data acquisition module 508 is used to obtain segmentation data of the detection object based on the target contour and the target semantic category, and the segmentation data includes the target contour carrying the target semantic category.

[0126] The above-mentioned image processing device performs target detection on the image to be annotated and determines the first detection area of ​​the detection object in the image to be annotated; based on the image to be annotated, the first detection area is visually segmented to more accurately obtain the target contour of the detection object, then, based on the target contour of the detection object, semantic category detection is performed on the detection object to more accurately determine the target semantic category of the detection object, and then based on the target contour and the target semantic category, segmentation data of the detection object can be more accurately obtained, and the segmentation data includes the target contour carrying the target semantic category, thereby improving the accuracy of image processing.

[0127] In one embodiment, the above-mentioned device also includes a feature data acquisition module and a model parameter adjustment module; the feature data acquisition module is used to obtain first feature data from the image to be annotated, and perform semantic category annotation on the first feature data to obtain second feature data; the model parameter adjustment module is used to use the second feature data to adjust the parameters of the target detection model to obtain an adjusted target detection model; the above-mentioned target detection module 502 is also used to perform target detection on the image to be annotated through the adjusted target detection model, and determine the first detection area of ​​the detection object in the image to be annotated.

[0128] In one embodiment, the feature data acquisition module is further used to extract original feature data of the image to be annotated; perform clustering processing on the original feature data to obtain at least one target cluster; and obtain first feature data from at least one target cluster.

[0129] In one embodiment, the above-mentioned model parameter adjustment module is also used to determine a first network structure to be parameter adjusted from the target detection model; in the target detection model, the second feature data is used to adjust the parameters of the first network structure to obtain an adjusted target detection model.

[0130] In one embodiment, the above-mentioned model parameter adjustment module is also used to use the second feature data to adjust the parameters of the classification model to obtain an adjusted classification model; the above-mentioned semantic category detection module 506 is also used to perform semantic category detection on the detection object based on the target contour through the adjusted classification model to determine the target semantic category of the detection object.

[0131] In one embodiment, the semantic category detection module 506 is further used to obtain a second detection area of ​​the target contour in the image to be annotated; perform semantic category detection on the second detection area to determine the target semantic category of the detection object.

[0132] Each module in the above image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0133] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0134] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0135] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: performing target detection on an image to be labeled, and determining a first detection area of ​​a detection object in the image to be labeled; performing visual segmentation on the first detection area based on the image to be labeled, and obtaining a target contour of the detection object; performing semantic category detection on the detection object based on the target contour, and determining a target semantic category of the detection object; and obtaining segmentation data of the detection object based on the target contour and the target semantic category. The segmentation data includes a target contour carrying the target semantic category.

[0136] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining first feature data from the image to be annotated, and performing semantic category annotation on the first feature data to obtain second feature data; using the second feature data to adjust parameters of the target detection model to obtain an adjusted target detection model; performing target detection on the image to be annotated using the adjusted target detection model to determine a first detection area of ​​the detection object in the image to be annotated.

[0137] In one embodiment, when the processor executes the computer program, the following steps are also implemented: for the image to be labeled, extracting the original feature data of the image to be labeled; performing clustering processing on the original feature data to obtain at least one target cluster; and obtaining first feature data from at least one target cluster.

[0138] In one embodiment, when the processor executes the computer program, the following steps are also implemented: determining a first network structure to be parameter adjusted from the target detection model; in the target detection model, adjusting the parameters of the first network structure using the second feature data to obtain an adjusted target detection model.

[0139] In one embodiment, when the processor executes the computer program, the following steps are also implemented: using the second feature data to adjust the parameters of the classification model to obtain an adjusted classification model; based on the target contour, using the adjusted classification model to perform semantic category detection on the detection object to determine the target semantic category of the detection object.

[0140] In one embodiment, when the processor executes the computer program, the following steps are further implemented: obtaining a second detection area of ​​the target contour in the image to be annotated; performing semantic category detection on the second detection area to determine the target semantic category of the detection object.

[0141] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: performing target detection on an image to be labeled, and determining a first detection area of ​​a detection object in the image to be labeled; based on the image to be labeled, visually segmenting the first detection area to obtain a target contour of the detection object; based on the target contour, performing semantic category detection on the detection object to determine a target semantic category of the detection object; based on the target contour and the target semantic category, obtaining segmentation data of the detection object, the segmentation data including the target contour carrying the target semantic category.

[0142] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining first feature data from the image to be annotated, and performing semantic category annotation on the first feature data to obtain second feature data; using the second feature data to adjust parameters of the target detection model to obtain an adjusted target detection model; performing target detection on the image to be annotated using the adjusted target detection model to determine a first detection area of ​​the detection object in the image to be annotated.

[0143] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: for the image to be labeled, extracting the original feature data of the image to be labeled; performing clustering processing on the original feature data to obtain at least one target cluster; and obtaining first feature data from at least one target cluster.

[0144] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: determining a first network structure to be parameter adjusted from a target detection model; in the target detection model, adjusting the parameters of the first network structure using the second feature data to obtain an adjusted target detection model.

[0145] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: using the second feature data to adjust the parameters of the classification model to obtain an adjusted classification model; based on the target contour, using the adjusted classification model to perform semantic category detection on the detection object to determine the target semantic category of the detection object.

[0146] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining a second detection area of ​​the target contour in the image to be annotated; performing semantic category detection on the second detection area to determine the target semantic category of the detection object.

[0147] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the following steps: performing target detection on an image to be labeled, and determining a first detection area of ​​a detection object in the image to be labeled; performing visual segmentation on the first detection area based on the image to be labeled, and obtaining a target contour of the detection object; performing semantic category detection on the detection object based on the target contour, and determining a target semantic category of the detection object; and obtaining segmentation data of the detection object based on the target contour and the target semantic category, wherein the segmentation data includes a target contour carrying the target semantic category.

[0148] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining first feature data from the image to be annotated, and performing semantic category annotation on the first feature data to obtain second feature data; using the second feature data to adjust parameters of the target detection model to obtain an adjusted target detection model; performing target detection on the image to be annotated using the adjusted target detection model to determine a first detection area of ​​the detection object in the image to be annotated.

[0149] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: for the image to be labeled, extracting the original feature data of the image to be labeled; performing clustering processing on the original feature data to obtain at least one target cluster; and obtaining first feature data from at least one target cluster.

[0150] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: determining a first network structure to be parameter adjusted from a target detection model; in the target detection model, adjusting the parameters of the first network structure using the second feature data to obtain an adjusted target detection model.

[0151] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: using the second feature data to adjust the parameters of the classification model to obtain an adjusted classification model; based on the target contour, using the adjusted classification model to perform semantic category detection on the detection object to determine the target semantic category of the detection object.

[0152] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining a second detection area of ​​the target contour in the image to be annotated; performing semantic category detection on the second detection area to determine the target semantic category of the detection object.

[0153] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0154] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0155] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0156] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: The method comprises: Performing target detection on the image to be labeled, and determining a first detection area of ​​the detection object in the image to be labeled; Based on the image to be annotated, visually segment the first detection area to obtain a target contour of the detection object; Based on the target contour, performing semantic category detection on the detection object to determine the target semantic category of the detection object; Based on the target contour and the target semantic category, segmentation data of the detection object is obtained, and the segmentation data includes the target contour carrying the target semantic category.

2. The method according to claim 1, characterized in that Before performing target detection on the image to be labeled and determining a first detection area of ​​the detection object in the image to be labeled, the method further includes: Acquire first feature data from the image to be annotated, and perform semantic category annotation on the first feature data to obtain second feature data; Using the second feature data to adjust parameters of the target detection model to obtain an adjusted target detection model; The step of performing target detection on the image to be annotated and determining a first detection area of ​​the detection object in the image to be annotated includes: The adjusted target detection model is used to perform target detection on the image to be labeled, and a first detection area of ​​the detection object in the image to be labeled is determined.

3. The method according to claim 2, characterized in that The acquiring first feature data from the image to be labeled includes: For the image to be annotated, extracting original feature data of the image to be annotated; Performing clustering processing on the original feature data to obtain at least one target cluster; The first feature data is obtained from the at least one target cluster.

4. The method according to claim 2, characterized in that: The step of adjusting parameters of the target detection model by using the second feature data to obtain an adjusted target detection model includes: Determining a first network structure whose parameters are to be adjusted from the target detection model; In the target detection model, the second feature data is used to adjust parameters of the first network structure to obtain an adjusted target detection model.

5. The method according to claim 2, characterized in that: After the semantic category labeling is performed on the first feature data to obtain the second feature data, the method further includes: Using the second feature data to adjust parameters of the classification model to obtain an adjusted classification model; The performing semantic category detection on the detection object based on the target contour to determine the target semantic category of the detection object includes: Based on the target contour, semantic category detection is performed on the detection object through the adjusted classification model to determine the target semantic category of the detection object.

6. The method according to claim 1, characterized in that The performing semantic category detection on the detection object based on the target contour to determine the target semantic category of the detection object includes: Acquire a second detection area of ​​the target contour in the image to be labeled; Perform semantic category detection on the second detection area to determine the target semantic category of the detection object.

7. An image processing device, characterized in that: The device comprises: A target detection module, used to perform target detection on the image to be annotated, and determine a first detection area of ​​the detection object in the image to be annotated; A visual segmentation module, used to perform visual segmentation on the first detection area based on the image to be annotated, so as to obtain a target contour of the detection object; A semantic category detection module, used to perform semantic category detection on the detection object based on the target contour, and determine the target semantic category of the detection object; A segmentation data determination module is used to obtain segmentation data of the detection object based on the target contour and the target semantic category, wherein the segmentation data includes the target contour carrying the target semantic category.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images