Image annotation method and apparatus, and device, and storage medium

Abstract

Disclosed in the present invention are an image annotation method and apparatus, and a device, and a storage medium. The method comprises: determining an image to be annotated and annotation prompt information, wherein the annotation prompt information is used for characterizing an annotation object that is required to be annotated; inputting said determined image and the determined annotation prompt information into a pre-trained object annotation model, wherein the object annotation model is used for: extracting image features of said input image, extracting prompt features of the input annotation prompt information, combining the extracted image features with the extracted prompt features, and outputting predicted region information on the basis of a combination result, the object annotation model comprising at least one generalization layer; and annotating said image on the basis of the predicted region information that is output by the object annotation model.

Description

An image annotation method, apparatus, device and storage medium Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image annotation method, apparatus, device, and storage medium. Background Technology

[0002] In the current field of image processing, a large number of labeled image samples are usually needed for model training to improve the model's ability to process images.

[0003] When labeling image samples, it is usually necessary to label specific objects in the image, such as human bodies, objects, faces, etc.

[0004] Currently, image sample annotation is usually done manually, which often requires a lot of manpower and is inefficient. Summary of the Invention

[0005] This invention provides an image annotation method, apparatus, device, and storage medium to address the shortcomings of related technologies.

[0006] According to a first aspect of the present invention, an image annotation method is provided, comprising:

[0007] Identify the image to be labeled and the labeling prompt information; the labeling prompt information is used to characterize the labeling object that needs to be labeled.

[0008] The identified images to be labeled and the labeling prompts are input into a pre-trained object labeling model;

[0009] The object annotation model is used to: extract image features from the input image to be annotated, extract prompt features from the input annotation prompt information, and merge the extracted image features and prompt features; output predicted region information based on the merging result; the predicted region information is used to characterize the region where the predicted annotated object is located in the input image; the object annotation model includes at least one generalization layer; the generalization layer includes at least one preset convolutional layer; the preset convolutional layer is used to perform convolution on the input feature map using a preset convolutional kernel; the preset convolutional kernel includes at least one freeze parameter with a value of 0;

[0010] The image to be labeled is labeled based on the predicted region information output by the object labeling model.

[0011] Optionally, the training method for the object annotation model includes:

[0012] A training sample set is determined; the features of any sample in the training sample set include image features and annotation prompts, the annotation prompts in the sample features are used to characterize the annotation objects that need to be annotated in the image features of the sample; the label of any sample includes: actual region information used to characterize the region where the annotation objects are located in the image features of the sample;

[0013] Determine the initial annotation model;

[0014] The initial annotation model is trained using the training sample set, and the trained initial annotation model is determined as the object annotation model;

[0015] During the training of the initial annotation model, the sample features of any sample in the training sample set are input into the current initial annotation model to obtain the predicted region information output by the current initial annotation model. The loss is determined based on the obtained predicted region information and the actual region information in the corresponding sample label. The parameters in the current initial annotation model are updated based on the determined loss.

[0016] The initial annotation model includes at least one generalization layer; the generalization layer includes at least one preset convolutional layer; the preset convolutional layer is used to perform convolution on the input feature map using a preset convolutional kernel; the preset convolutional kernel includes at least one frozen parameter with a value of 0.

[0017] Optionally, different preset convolution kernels in different preset convolutional layers may have different sizes, and / or the proportion of the frozen parameters in the kernel coefficients may differ.

[0018] Optionally, the generalization layer includes cascaded downsampling layers and upsampling layers; the output of the downsampling layer is cascaded to the input of the upsampling layer;

[0019] The downsampling layer is used to perform downsampling based on at least one preset convolutional layer; and / or, the upsampling layer is used to perform feature extraction based on at least one preset convolutional layer.

[0020] Optionally, the downsampling layer is used to perform any of the following:

[0021] Downsampling is performed based on at least two cascaded preset convolutional layers; in the at least two cascaded preset convolutional layers, the output of any preset convolutional layer is cascaded to the input of the next preset convolutional layer;

[0022] Downsampling is performed on different preset convolutional layers to obtain downsampled intermediate feature maps of different sizes. The obtained downsampled intermediate feature maps are then combined to obtain the downsampled result feature map.

[0023] Downsampling is performed based on at least two cascaded pre-defined convolutional layers; the output of any pre-defined convolutional layer is cascaded to the input of the next pre-defined convolutional layer; the downsampled intermediate feature maps of the outputs of each cascaded pre-defined convolutional layer are combined to obtain the downsampled result feature map.

[0024] Optionally, the upsampling layer is used for:

[0025] If any upsampled intermediate feature map is obtained through upsampling, features are extracted from the obtained upsampled intermediate feature map based on at least one preset convolutional layer.

[0026] Optionally, the generalization layer includes: a feature extraction layer, the downsampling layer, and the upsampling layer;

[0027] The feature extraction layer is used to extract a first feature map from the input feature map;

[0028] The downsampling layer is used to downsample the first feature map to obtain the downsampling result;

[0029] The upsampling layer is used to: fuse the downsampling result and the first feature map, and then upsample the fused result.

[0030] Optionally, during the training of the initial labeled model, a mask feature map is obtained by setting a mask for the initial feature map output by the previous layer of the current target generalization layer, and then the obtained mask feature map is input into the current target generalization layer to obtain the generalized feature map output by the current target generalization layer.

[0031] Optionally, in any round of training, the parameters of the current target generalization layer are updated according to the preset generalization loss;

[0032] The method for determining the preset generalization loss includes: in this round of training, inputting the initial feature map output by the previous layer of the current target generalization layer into the current target generalization layer to obtain the label feature map output by the current target generalization layer, and determining the preset generalization loss based on the difference between the obtained label feature map and the generalization feature map output by the current target generalization layer.

[0033] Optionally, the generalized feature map and the mask feature map have the same size.

[0034] Optionally, the features of any sample in the training sample set may also include annotation prompt information; the annotation prompt information is used to characterize the annotation objects that need to be annotated in the image features of the sample.

[0035] Optionally, determining the initial annotation model includes:

[0036] Determine the initial annotation model;

[0037] For the determined initial annotation model, add at least one generalization layer to obtain the currently determined initial annotation model.

[0038] Optionally, determining the initial annotation model includes:

[0039] An initial annotation model is obtained; the initial annotation model includes an image encoder, a cue encoder, and a decoder; the image encoder is used to extract image features from the input image; the cue encoder is used to extract cue features from the input annotation cue information;

[0040] The decoder is used to: merge the image features extracted by the image encoder with the prompt features extracted by the prompt encoder; based on the merged result and an attention mechanism, predict the probability that each pixel in the input image belongs to the region where the labeled object is located, and output the predicted region information according to the predicted probability; the predicted region information is used to characterize the region where the predicted labeled object is located in the input image.

[0041] For the obtained initial annotation model, at least one generalization layer is added to the image encoder to obtain the currently determined initial annotation model.

[0042] Optionally, the annotation prompt information includes at least one of the following:

[0043] Textual information used to characterize the objects to be annotated;

[0044] It includes the bounding box information of the area where the labeled object is located;

[0045] Includes prompt area information about the area where the object to be labeled is located;

[0046] Point information located within the area containing the object to be labeled;

[0047] Information about points located outside the area where the object to be labeled is located.

[0048] According to a second aspect of the present invention, an image annotation apparatus is provided, comprising:

[0049] A determining unit is used to determine the image to be labeled and the labeling prompt information; the labeling prompt information is used to characterize the labeling object that needs to be labeled.

[0050] The annotation unit is used to input the determined image to be annotated and the annotation prompt information into the pre-trained object annotation model;

[0051] The object annotation model is used to: extract image features from the input image to be annotated, extract prompt features from the input annotation prompt information, and merge the extracted image features and prompt features; output predicted region information based on the merging result; the predicted region information is used to characterize the region where the predicted annotated object is located in the input image; the object annotation model includes at least one generalization layer; the generalization layer includes at least one preset convolutional layer; the preset convolutional layer is used to perform convolution on the input feature map using a preset convolutional kernel; the preset convolutional kernel includes at least one freeze parameter with a value of 0;

[0052] The image to be labeled is labeled based on the predicted region information output by the object labeling model.

[0053] As can be seen from the above embodiments, the efficiency of image annotation is improved by automatically annotating images using a pre-trained object annotation model.

[0054] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0055] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0056] Figure 1 is a schematic diagram illustrating the principle of image annotation according to an embodiment of the present invention;

[0057] Figure 2 is a flowchart illustrating an image annotation method according to an embodiment of the present invention;

[0058] Figure 3 is a schematic diagram illustrating the effect of image annotation according to an embodiment of the present invention;

[0059] Figure 4 is a schematic diagram illustrating another image annotation effect according to an embodiment of the present invention;

[0060] Figure 5 is a schematic diagram of the image coding part in an initial annotation model according to an embodiment of the present invention;

[0061] Figure 6 is a schematic diagram of a generalization layer according to an embodiment of the present invention;

[0062] Figure 7 is a flowchart illustrating another image annotation method according to an embodiment of the present invention;

[0063] Figure 8 is a schematic diagram of another generalization layer structure according to an embodiment of the present invention;

[0064] Figure 9 is a schematic diagram of an image annotation device according to an embodiment of the present invention;

[0065] Figure 10 is a schematic diagram of the hardware structure of a computer device configured according to an embodiment of the present invention. Detailed Implementation

[0066] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0067] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0068] In the current field of image processing, a large number of labeled image samples are usually needed for model training to improve the model's ability to process images.

[0069] When labeling image samples, it is usually necessary to label the targets in the image, such as human bodies, objects, faces, etc.

[0070] Currently, image sample annotation is usually done manually, which often requires a lot of manpower and is inefficient.

[0071] To address the aforementioned problems, this invention discloses an image annotation method.

[0072] In this method, a pre-trained model, specifically an object annotation model, can be used to input the image to be annotated and the information of the objects to be annotated in the image into the object annotation model. The object annotation model can then identify and detect the regions where the objects to be annotated are located in the input image based on the input object information. Based on the identified regions where the objects to be annotated are located, annotation can be easily performed.

[0073] For example, the image to be labeled, along with the information about the "pencil" marks within it, can be input into an object labeling model. Based on the image and the "pencil" information, the model can identify and detect the regions where the "pencil" marks are located within the input image. Specifically, it can directly output the regions where the "pencil" marks are located in the image.

[0074] This method can automatically annotate images based on an object annotation model, improving the efficiency of image annotation. Furthermore, it can enhance the flexibility and convenience of image annotation based on object information.

[0075] This method does not limit the specific form of the information to be labeled. It can be text, a bounding box containing the labeled object, or points within the region where the labeled object is located in the image, etc.

[0076] This method does not limit the specific annotation method. It can label the area where the identified and detected objects in the image are located as the image label. Specifically, the annotation can be based on the information of the objects.

[0077] This method does not limit the specific image type or the type of annotation object to be labeled.

[0078] Optionally, the image can be an image of the display screen or an image of the display screen product, thereby enabling automatic image annotation for the display screen and improving the efficiency of image annotation for the display screen.

[0079] The type of object to be labeled can be any object in the display screen scene that needs to be labeled. For example, the object can be a cross mark on the display screen, a defect on the display screen, or an irregular area on the display screen, etc.

[0080] In this embodiment, images from the display screen scene can be further utilized to generate a corresponding training sample set, and the object annotation model can be optimized and trained to improve the annotation accuracy of the object annotation model for the display screen scene images.

[0081] Considering that images in display screen scenarios often include a wide variety of display screens, it is necessary to further improve the generalization ability of object annotation models.

[0082] Therefore, in this method, the generalization ability of the object annotation model can be improved during the training process, thereby enhancing the annotation accuracy of the object annotation model for images in different scenes. This method does not limit the specific ways to improve the generalization ability of the object annotation model; details can be found in the explanation below.

[0083] For ease of understanding, as shown in Figure 1, which is a schematic diagram illustrating the principle of image annotation according to an embodiment of the present invention.

[0084] For unlabeled images, the prompt "circle" can be input into the object annotation model so that the model can annotate the circles in the unlabeled image, thus obtaining a labeled image. In the labeled image, region labels are set using a mask method. Specifically, gray fill is used to indicate that circles in the image are labeled, serving as region labels.

[0085] The following is a detailed explanation of an image annotation method provided by an embodiment of the present invention.

[0086] As shown in Figure 2, Figure 2 is a flowchart illustrating an image annotation method according to an embodiment of the present invention.

[0087] The embodiments of the present invention do not limit the executing entity of this method. Optionally, the executing entity can be any computing device, such as a terminal, server, computer, etc.

[0088] The method may include the following steps.

[0089] S101: Determine the image to be labeled and the labeling prompt information; the labeling prompt information is used to characterize the labeling objects that need to be labeled.

[0090] S102: Input the determined image to be labeled and the labeling prompt information into the pre-trained object labeling model.

[0091] The object labeling model can be used to: extract image features from the input image to be labeled, extract prompt features from the input labeling prompt information, and merge the extracted image features and prompt features; based on the merging result, predicted region information can be output; the predicted region information can be used to characterize the region where the predicted labeled object is located in the input image.

[0092] The predicted region information is output based on the merging result. Specifically, it can be based on the attention mechanism of the merging result to predict the probability that each pixel in the input image belongs to the region where the labeled object is located, and output the predicted region information according to the predicted probability. The predicted region information can be used to characterize the region where the predicted labeled object is located in the input image.

[0093] Furthermore, the object annotation model may include at least one generalization layer; the generalization layer may include at least one preset convolutional layer; the preset convolutional layer can be used to perform convolution on the input feature map using a preset convolutional kernel; the preset convolutional kernel includes at least one frozen parameter with a value of 0. The generalization layer can improve the generalization ability of the object annotation model.

[0094] S103: Label the image to be labeled based on the predicted region information output by the object labeling model.

[0095] The above method can automatically annotate images using a pre-trained object annotation model, thereby improving the efficiency of image annotation.

[0096] Furthermore, the above method can also improve the flexibility and convenience of image annotation by utilizing annotation prompts.

[0097] This method does not limit the specific content of the image to be labeled, nor does it limit the specific form or the specific method of determination.

[0098] Optionally, the image to be labeled can be determined by directly obtaining the image to be labeled, or by obtaining an initial image and then adjusting its size to obtain the image to be labeled.

[0099] This method does not limit the number of images to be labeled. Specifically, it can involve identifying one or more images to be labeled, and then executing this method for each image.

[0100] The above method primarily explains the process of labeling a single image using an object annotation model. It can be understood that this method can be used to annotate multiple images separately.

[0101] This method does not limit the scene of the image to be labeled. Specifically, it can be an image to be labeled in a display screen scene, or it can be an image on multiple display screens.

[0102] This method does not limit the specific objects to be annotated. The objects to be annotated can be pencils, calendars, display screen defects, a component of the display screen, etc. For ease of description, the objects to be annotated in this method will be referred to as "annotation objects."

[0103] I. Regarding the labeling and prompting information.

[0104] This method does not limit the specific representation information of the labeled prompts.

[0105] Labeling prompts can be used to represent the object to be labeled. Specifically, they can represent the object to be labeled itself or the region where the object to be labeled is located in the image.

[0106] For ease of understanding, in a specific example, the annotation prompt can be in the form of the text word "pencil," thus representing the object to be annotated in text form, or it can be a rough rectangle containing the "pencil" in the image, thus representing the area of ​​the object to be annotated in the image in the form of a bounding box.

[0107] Therefore, optionally, the annotation prompt information can be used to represent at least one of the following: 1) the object to be annotated itself; 2) the area in the image where the object to be annotated is located.

[0108] It should be noted that since the object annotation model is based on the input image and the input annotation prompt information, it predicts the region where the annotated object is located in the input image. If the annotation prompt information represents the region where the annotated object is located in the input image, the accuracy of the predicted output region is usually greater than the accuracy of the region of the annotation prompt information.

[0109] For example, the annotation prompt can be a rectangle containing the annotation object, while the object annotation model output can be the precise area where the annotation object is located, specifically the area that fits the outline of the annotation object.

[0110] Therefore, it is understandable that annotation prompts can be used to characterize the approximate or rough area where the object to be annotated in the image is located.

[0111] To facilitate understanding, in a specific example, when providing hints for annotations, one could use annotation hints that represent the annotated object itself for images that are completely unannotated. However, there are also images with partial or inaccurate annotations. In these cases, the partial or inaccurate annotations could be used as annotation hints for subsequent annotation.

[0112] For example, for some images that have been collected with rough outlines of human bodies, the object annotation model in this method can be used to perform precise annotation, obtain an accurate region that fits the human body contour, and improve the efficiency of image annotation.

[0113] This method does not limit the specific form of the annotation prompts.

[0114] Optionally, the labeling information can be in the form of text, images, or audio, and can primarily represent the object itself. Examples include the text "pencil," an image of a water glass, and the audio of a calendar.

[0115] The format of annotation prompts can be either coordinate points within the image. Specifically, these can be coordinate points within the area containing the object to be labeled, or coordinate points outside that area. Coordinate points primarily represent the approximate location of the object to be labeled within the image. Of course, the number of coordinate points is not limited here; it can be understood that the annotation prompts can include several coordinate points, specifically several coordinate points within the area containing the object to be labeled in the image.

[0116] The labeling prompts can also take the form of bounding boxes in an image. Specifically, it can be a bounding box containing the object to be labeled, that is, a bounding box containing the area where the object to be labeled is located. The bounding box can be a rectangle or a polygon; the shape of the bounding box is not limited.

[0117] It is understandable that the annotation prompt information can include several bounding boxes, specifically several bounding boxes in the image that contain the annotation objects that need to be annotated. Different bounding boxes can contain the same annotation object and there can be overlapping areas.

[0118] The labeling information can also take the form of region information in the image. Specifically, it can be the entire region of the image containing the labeled object, that is, the entire region containing the area where the labeled object is located, and the area of ​​the entire region can be larger than the area where the labeled object is located. In one example, the labeling information can be in the form of a mask, thus representing the entire region of the image containing the labeled object.

[0119] Of course, when the above annotation prompts represent the area where the annotation object needs to be annotated in the image, the annotation prompts usually exist in a corresponding bound image, representing the area where the annotation object needs to be annotated in the corresponding bound image.

[0120] For example, for the image to be labeled in this method process, the labeling prompt information can be used to characterize the area where the labeled object is located in the image.

[0121] This method does not limit the number of annotation prompts. There can be one or more annotation prompts, or it can include different forms of prompts. Determining the annotation prompts can specifically involve determining one or more annotation prompts, or it can involve determining different forms of annotation prompts, etc.

[0122] For example, label prompts can include text information as well as bounding box information.

[0123] Optionally, determining annotation prompts can involve determining one or more annotation prompts, which can be in the same or different forms.

[0124] Correspondingly, for object annotation models that require input of annotation prompts, different forms of annotation prompts can be input. For annotation prompts that are not input, default values ​​or default values ​​can be used instead.

[0125] In summary, optionally, the annotation prompt information may include at least one of the following: 1) text information representing the object to be annotated; 2) bounding box information containing the area where the object to be annotated is located; 3) prompt area information containing the area where the object to be annotated is located; 4) point information located within the area where the object to be annotated is located; 5) point information located outside the area where the object to be annotated is located.

[0126] The information in the prompt area can be in the form of a mask.

[0127] This embodiment can set different forms of annotation prompts to improve the flexibility of annotation prompts and the convenience of image annotation.

[0128] Having explained the representation information and form of annotation prompts in the previous section, this method does not limit the way annotation prompts are generated.

[0129] Alternatively, the annotation prompts can be generated manually. For example, the annotation prompts can be generated manually in the form of text, which determines that the object to be annotated is "calendar". Alternatively, a rectangle containing "calendar" can be roughly generated in the image for the object to be annotated, "calendar".

[0130] Of course, the annotation prompts can also be generated automatically by the machine. For example, an object detection model or other model can be used to perform pre-detection of objects in the image, resulting in a rectangular box containing the objects that need to be annotated.

[0131] To make it easier to understand, in a specific example, labeled prompts can be generated through user interaction with the device.

[0132] For example, the image to be labeled can be displayed in the display area of ​​the device, allowing interactive operations on the image in the display area. Specifically, interactive operations can be used to punctuate, drag bounding boxes, input text, set mask areas, and so on.

[0133] Specifically, for the labeled object in the displayed image to be labeled, several points can be marked within the area where the labeled object is located through interactive operations, or several points can be marked outside the area where the labeled object is located.

[0134] Alternatively, for the labeled object in the displayed image, an interactive operation can be used to drag out a bounding box around the area where the labeled object is located, thus including the area where the labeled object is located.

[0135] Alternatively, users can interact with the displayed image to input text describing the object being labeled, which will then serve as the labeling prompt for the image.

[0136] II. Regarding the output of the object annotation model.

[0137] This method does not limit the output format of the object annotation model, as long as the output predicted region information can represent the region where the predicted labeled object is located in the input image.

[0138] Optionally, the predicted region information output by the object annotation model can be in the form of a mask, that is, a mask representing the region where the labeled object is located in the input image. Specifically, the mask can fit the outline of the labeled object in the input image. Of course, the predicted region information output by the object annotation model can also be in the form of extracted image content, that is, the image content of the region where the labeled object is located in the input image. The predicted region information output by the object annotation model can also be the outline of the labeled object in the input image and its position information in the input image, etc.

[0139] Correspondingly, this method does not limit the specific way of labeling based on the predicted region information output by the object labeling model.

[0140] Optionally, the predicted region information can be directly used as the label of the image to be labeled, or the label of the image to be labeled can be obtained after processing the balance region information.

[0141] This method does not limit the specific processing method for the information in the prediction area; it can be determined according to actual needs.

[0142] In one alternative embodiment, the predicted region information can be further adjusted manually to improve the accuracy of the annotation. Since the predicted region information is automatically generated, the subsequent manual workload is greatly reduced, thereby improving the accuracy, precision, and efficiency of image annotation.

[0143] Of course, the predicted region information can also be normalized. For example, if the image label needs to be in the form of a rectangular box, a more accurate rectangular box can be generated based on the predicted region information to serve as the label for the image to be labeled.

[0144] To make it easier to understand, specific examples are given below.

[0145] As shown in Figure 3, Figure 3 is a schematic diagram illustrating the effect of image annotation according to an embodiment of the present invention.

[0146] Specifically, for the crosshair mark area of ​​the display screen, the corresponding contour can be predicted and output as area information (crosshair contour) by using the coordinate point annotation information (white dots in the crosshair mark area) in the crosshair mark area.

[0147] As shown in Figure 4, Figure 4 is a schematic diagram illustrating another image annotation effect according to an embodiment of the present invention.

[0148] Figure 4 shows four different image annotation scenarios.

[0149] The image in the upper left corner can be based on coordinate point-based annotation prompts to determine the contour-based predicted region information. Specifically, it can use the coordinate points (black dots) within the region where the object is located and the coordinate points (white dots) outside the region where the object is located as annotation prompts, and use the object annotation model to output the contour-based predicted region information.

[0150] The image in the upper right corner can be used to determine the predicted region information in the form of a bounding box based on the annotation prompts. Specifically, it can use the rectangular box containing the annotated object as the annotation prompt, and use the object annotation model to output the predicted region information in the form of a contour.

[0151] The image in the lower left corner can be used to generate a mask region based on the predicted region information in the form of a contour. The mask region can be stored as a JSON file.

[0152] The image in the lower right corner can be a predicted region based on the contour form, generating a polygonal region with a set of polygon vertices. The vertices can be dragged manually, which makes it easy to adjust the vertices of the polygons to improve the accuracy of the predicted region information and the accuracy of the image annotation, greatly reducing the amount of manual work.

[0153] III. The structure of the object annotation model.

[0154] This method does not limit the specific structure of the object annotation model.

[0155] Optionally, the object annotation model may include an image encoder for the image input, a cue encoder for the annotation prompts, a decoder, and so on. Of course, the object annotation model may also include fully connected layers, classification layers, and other structures.

[0156] Optionally, the object annotation model may include an image encoder, a cue encoder, and a decoder; the image encoder can be used to extract image features from the input image; the cue encoder can be used to extract cue features from the input annotation cue information.

[0157] The decoder can be used to: merge the image features extracted by the image encoder with the cue features extracted by the cue encoder; output predicted region information based on the merging result; the predicted region information can be used to characterize the region where the predicted labeled object is located in the input image.

[0158] Specifically, the decoder can be used to: merge the image features extracted by the image encoder with the cue features extracted by the cue encoder; based on the merged result, predict the probability that each pixel in the input image belongs to the region where the labeled object is located, and output the predicted region information according to the predicted probability; the predicted region information can be used to characterize the region where the predicted labeled object is located in the input image.

[0159] In one alternative embodiment, in order to improve the generalization ability of the object annotation model, the object annotation model may include a generalization layer, which can be used to improve the generalization ability of the object annotation model.

[0160] Optionally, the object annotation model may include at least one generalization layer; the generalization layer may include at least one preset convolutional layer; the preset convolutional layer may be used to perform convolution on the input feature map using a preset convolutional kernel; the preset convolutional kernel includes at least one frozen parameter with a value of 0.

[0161] In the object labeling model, the generalization layer can be a structure that exists when the model is built, or it can be a structure added later, or it can be a structure added before training, or it can be a structure that is added after the object labeling model is trained.

[0162] For example, to improve the generalization ability of the model, a generalization layer can be added to the object labeling model before further model training.

[0163] In an optional embodiment, for ease of description, the annotation model before training can be referred to as the initial annotation model. The initial annotation model can be trained using the training sample set, and the trained initial annotation model can be determined as the object annotation model.

[0164] This method does not limit the training method of the object annotation model.

[0165] Optionally, the training method for the object annotation model may include: determining a training sample set; the features of any sample in the training sample set may include image features, and the label of any sample may include: regional information of the region where the object is located in the image features of the sample; determining an initial annotation model; and training the initial annotation model using the training sample set to obtain the trained object annotation model.

[0166] Of course, optionally, the features of any sample in the training sample set may also include annotation prompts; the annotation prompts can be used to characterize the objects that need to be annotated in the image features of the sample.

[0167] Optionally, the training method for the object annotation model may specifically include: determining a training sample set; the features of any sample in the training sample set may include image features and annotation prompts, and the annotation prompts in the sample features may be used to characterize the annotation objects that need to be annotated in the image features of the sample; the label of any sample may include: actual region information used to characterize the region where the annotation objects are located in the image features of the sample; determining an initial annotation model; training the initial annotation model using the training sample set, and determining the trained initial annotation model as the object annotation model.

[0168] Accordingly, during the training of the initial annotation model, the sample features of any sample in the training sample set can be input into the current initial annotation model to obtain the predicted region information output by the current initial annotation model. The loss is determined based on the obtained predicted region information and the actual region information in the corresponding sample label, and the parameters in the current initial annotation model are updated based on the determined loss.

[0169] The initial annotation model may include at least one generalization layer; the generalization layer may include at least one preset convolutional layer; the preset convolutional layer may be used to perform convolution on the input feature map using a preset convolutional kernel; the preset convolutional kernel may include at least one frozen parameter with a value of 0.

[0170] For a more detailed explanation of model training, please refer to the explanation below.

[0171] The model structure can remain unchanged for both the initial labeled model before training and the object labeled model after training.

[0172] Optionally, the initial annotation model can be used to extract image features from the input image to be annotated, extract prompt features from the input annotation prompt information, merge the extracted image features and prompt features, and output predicted region information based on the merging result.

[0173] Optionally, the initial annotation model may include an image encoder, a cue encoder, and a decoder.

[0174] The image encoder can be used to extract image features from the input image; the cue encoder can be used to extract cue features from the input annotation cue information; the decoder can be used to: merge the image features extracted by the image encoder and the cue features extracted by the cue encoder; output prediction region information based on the merging result; the prediction region information is used to characterize the region where the predicted annotation object is located in the input image.

[0175] Specifically, the decoder can be used to: merge the image features extracted by the image encoder with the cue features extracted by the cue encoder; based on the merged result, predict the probability that each pixel in the input image belongs to the region where the labeled object is located, and output the predicted region information according to the predicted probability; the predicted region information can be used to characterize the region where the predicted labeled object is located in the input image.

[0176] Accordingly, the generalization layer can be added to the initial labeled model before model training. This method does not limit the specific location of the generalization layer; it can be added to any structure of the initial labeled model, specifically to the encoding and / or decoding parts, to improve the generalization ability of the initial labeled model. Furthermore, the generalization ability of the object labeled model can be improved through further model training.

[0177] The initial annotation model may include at least one generalization layer, thereby improving the generalization ability of the object annotation model through training.

[0178] This method does not limit the position and number of generalization layers in the initial annotation model.

[0179] Optionally, the initial annotation model may include one or more generalization layers, and the structures of different generalization layers may be different or the same.

[0180] Correspondingly, the encoding part of the initial annotation model can contain one or more generalization layers. Similarly, the decoding part of the initial annotation model can also contain one or more generalization layers.

[0181] Since the generalization layer is mainly used to improve the generalization ability of the model, the feature map sizes of the input and output of the generalization layer can be the same, so as not to affect the feature map sizes of the layers before and after the generalization layer.

[0182] For ease of understanding, as shown in Figure 5, Figure 5 is a schematic diagram of the structure of the image encoding part in an initial annotation model according to an embodiment of the present invention.

[0183] The image coding section can include three image coding layers and three generalization layers. The input of the generalization layer can be cascaded to the output of the previous image coding layer, and the output of the generalization layer can be cascaded to the input of the next layer.

[0184] Understandably, other parts of the initial annotation model may also contain generalization layers.

[0185] In an alternative embodiment, the generalization layer may be added additionally to the initial labeled model to improve the model's generalization ability.

[0186] Determining the initial annotation model can specifically involve: obtaining the initial annotation model; and adding at least one generalization layer to the image encoder for the obtained initial annotation model to obtain the currently determined initial annotation model.

[0187] Alternatively, for the obtained initial annotation model, at least one generalization layer can be added to at least one of the following to obtain the currently determined initial annotation model: image encoder, cue encoder, and decoder.

[0188] This embodiment can improve the generalization ability of the annotation model by improving the structure of the annotation model and adding a generalization layer.

[0189] The following explanation addresses the generalization layers in the initial annotation model and the object annotation model. For ease of description, the initial annotation model and the object annotation model will be collectively referred to as the annotation model.

[0190] It is understandable that the model structure, including the generalization layer structure, can remain unchanged for both the initial labeled model before training and the object labeled model after training.

[0191] This method does not limit the internal structure of the generalization layer.

[0192] Alternatively, the generalization layer can adopt a residual network structure or a UET network structure.

[0193] Optionally, the generalization layer can extract features from the output feature map of the previous layer to obtain a new feature map, and then merge the new feature map with the output feature map of the previous layer before outputting it.

[0194] Optionally, the generalization layer may include at least one pre-defined convolutional layer. This pre-defined convolutional layer can extract features using dilated convolution, thereby expanding the receptive field of the convolutional kernel. This helps the annotation model better understand the semantic information in the image and improves its generalization ability. Furthermore, dilated convolution has a relatively small impact on model computation, reducing computational cost compared to other convolution methods.

[0195] This method does not limit the form of dilated convolution used in the preset convolutional layers.

[0196] Optionally, the preset convolutional layer can be used to perform convolution on the input feature map using a preset convolutional kernel; the preset convolutional kernel may include at least one frozen parameter with a value of 0.

[0197] The frozen parameters in the preset convolution kernel are not updated during training, so they can be set to 0, which means that some pixels do not participate in the operation of the convolution kernel. This can increase the size of the convolution kernel and increase its receptive field, which can help the annotation model better understand the semantic information in the image and improve the generalization ability of the annotation model.

[0198] To facilitate the provision of receptive fields of different sizes and further improve the generalization ability of the labeled model, different sizes can be set for the pre-convolutional kernels of different pre-convolutional layers, such as 3*3, 5*5, and 9*9. Different freezing parameter percentages can also be set for the pre-convolutional kernels of different pre-convolutional layers; for example, the hole rate or dilation rate can differ between the pre-convolutional kernels of different pre-convolutional layers, such as 4%, 8%, 12%, 16%, 20%, etc.

[0199] Furthermore, the location of the freeze parameters is not limited between the preset convolution kernels of different preset convolutional layers; they can be the same or different.

[0200] By using different preset convolutional kernels, receptive fields can be provided for different situations, which can help the annotation model better understand the semantic information in the image and improve the generalization ability of the annotation model.

[0201] Therefore, optionally, different preset convolution kernels in different preset convolutional layers may have different sizes, and / or the proportion of the frozen parameters in the kernel coefficients may be different.

[0202] This embodiment can utilize different preset convolution kernels in the generalization layer to provide different receptive fields, which can help the annotation model better understand the semantic information in the image and improve the generalization ability of the annotation model.

[0203] This method does not limit the number of pre-defined convolutional layers or their deployment in the generalization layer.

[0204] Optionally, the generalization layer may include one or more preset convolutional layers, which can be deployed in a cascaded manner or in parallel manner.

[0205] In an optional embodiment, in addition to the preset convolutional layer described above, the generalization layer can also use a combination of upsampling and downsampling to utilize feature maps of different sizes to help the annotation model better understand the semantic information in the image and improve the generalization ability of the annotation model.

[0206] Therefore, the generalization layer may optionally include an upsampling layer and a downsampling layer.

[0207] This embodiment does not limit the number or deployment method of upsampling and downsampling layers. Optionally, the generalization layer may contain one or more upsampling layers, or one or more downsampling layers.

[0208] Correspondingly, the output of a downsampling layer can be concatenated to the input of an upsampling layer. Alternatively, multiple consecutive downsampling layers can be concatenated, with the output of the last downsampling layer concatenated to the input of the first upsampling layer, and so on. Of course, the output of a downsampling layer can also be concatenated to the input of an upsampling layer, and then the output of that upsampling layer can be concatenated to the input of another downsampling layer, and so on.

[0209] This embodiment does not limit the specific structure and algorithm of the upsampling layer and the downsampling layer.

[0210] Optionally, the upsampling layer can perform upsampling directly, or feature extraction can be performed after upsampling. Specifically, feature extraction can be performed using the aforementioned preset convolutional layer. Similarly, the downsampling layer can perform downsampling directly, or feature extraction can be performed after downsampling using the aforementioned preset convolutional layer.

[0211] Therefore, optionally, the generalization layer may include cascaded downsampling layers and upsampling layers; the output of the downsampling layer is cascaded to the input of the upsampling layer; wherein the downsampling layer may be used to perform downsampling based on at least one preset convolutional layer; and / or, the upsampling layer may be used to perform feature extraction based on at least one preset convolutional layer.

[0212] Of course, downsampling layers can also be used for feature extraction based on at least one pre-defined convolutional layer.

[0213] This embodiment can combine upsampling and downsampling, as well as pre-defined convolutional layers with dilated convolutions, to help the annotation model better understand the semantic information in the image and improve the generalization ability of the annotation model.

[0214] Correspondingly, optionally, the downsampling layer can be used to perform any of the following:

[0215] 1) Downsampling is performed based on at least two cascaded preset convolutional layers; in the at least two cascaded preset convolutional layers, the output of any preset convolutional layer is cascaded to the input of the next preset convolutional layer.

[0216] 2) Perform downsampling according to different preset convolutional layers to obtain downsampled intermediate feature maps of different sizes, and then combine the obtained downsampled intermediate feature maps to obtain the downsampled result feature map.

[0217] 3) Downsampling is performed based on at least two cascaded preset convolutional layers; the output of any preset convolutional layer in the cascaded at least two preset convolutional layers is cascaded to the input of the next preset convolutional layer; the downsampled intermediate feature maps of the outputs of each cascaded preset convolutional layer are combined to obtain the downsampled result feature map.

[0218] The downsampling layer can combine multiple pre-defined convolutional layers to perform downsampling, which can improve the generalization ability of the annotation model. Furthermore, the pre-defined convolutional kernels of different pre-defined convolutional layers can be different, thereby providing different receptive fields. By cascading or directly integrating downsampled intermediate feature maps of different sizes, multi-scale semantic features can be used to help the annotation model better understand the semantic information in the image and improve the generalization ability of the annotation model.

[0219] It should be noted that multiple pre-defined convolutional layers in the downsampling layer can be cascaded in order of pre-defined kernel dilation rates, from largest to smallest or smallest to largest, to facilitate the acquisition of multi-scale semantic features. Of course, they can also be cascaded in a non-fixed order.

[0220] Furthermore, there is no limitation on the specific method of synthesizing the intermediate downsampled feature maps to obtain the downsampled result feature maps. Optionally, the sizes of each downsampled intermediate feature map can be adjusted before feature extraction, or the individual downsampled intermediate feature maps can be input into a convolutional layer for synthesis.

[0221] Optionally, the upsampling layer can be used to: extract features from any upsampled intermediate feature map obtained through upsampling, based on at least one preset convolutional layer.

[0222] In this embodiment, the upsampling layer can be combined with at least one preset convolutional layer to facilitate the improvement of the generalization ability of the labeled model.

[0223] Having explained the pre-convolutional layer, upsampling layer, and downsampling layer in the generalization layer above, this specification also provides an optional embodiment for easier understanding, to explain the overall structural composition of the generalization layer.

[0224] Optionally, the generalization layer may include a downsampling layer and an upsampling layer; the downsampling layer can be used to downsample the input feature map to obtain a downsampling result; the upsampling layer can be used to fuse the downsampling result and the input feature map, and then upsample the fused result.

[0225] For details on the structure and operation of the upsampling and downsampling layers, please refer to the explanation above.

[0226] In this embodiment, the downsampling results of the downsampling layer can be fused with the input feature map to facilitate the annotation model's further understanding of the semantic information in the image and improve the generalization ability of the annotation model.

[0227] Furthermore, a feature extraction layer can be added to the generalization layer to improve the feature extraction performance and generalization ability of the labeled model. Therefore, the generalization layer can optionally include: a feature extraction layer, a downsampling layer, and an upsampling layer; the feature extraction layer is used to extract a first feature map from the input feature map; the downsampling layer is used to downsample the first feature map to obtain a downsampling result; and the upsampling layer is used to fuse the downsampling result and the first feature map, and then upsample the fused result.

[0228] This embodiment can fuse the downsampling results of the downsampling layer with the first feature map extracted by the feature extraction layer, which facilitates the annotation model to further understand the semantic information in the image and improves the generalization ability of the annotation model.

[0229] For ease of understanding, as shown in Figure 6, which is a schematic diagram of a generalization layer according to an embodiment of the present invention.

[0230] The generalization layer may include one feature extraction layer, one downsampling layer, and one upsampling layer.

[0231] The feature extraction layer extracts a first feature map from the input feature map; the downsampling layer downsamples the first feature map to obtain a downsampled result; and the upsampling layer fuses the downsampled result and the first feature map, then upsamples the fused result to obtain an output feature map. The input and output feature maps of the generalization layer can have the same size.

[0232] IV. Training of the object annotation model.

[0233] This method does not limit the training method of the object annotation model. Training can be performed using a specific training sample set.

[0234] Optionally, the training method for the object annotation model may specifically include: determining a training sample set; the features of any sample in the training sample set may include image features and annotation prompts, and the annotation prompts in the sample features may be used to characterize the annotation objects that need to be annotated in the image features of the sample; the label of any sample may include: actual region information used to characterize the region where the annotation objects are located in the image features of the sample; determining an initial annotation model; training the initial annotation model using the training sample set, and determining the trained initial annotation model as the object annotation model.

[0235] This embodiment does not limit the specific method of training the initial annotation model. Optionally, during the training of the initial annotation model, the sample features of any sample in the training sample set can be input into the current initial annotation model to obtain the predicted region information output by the current initial annotation model. The loss is then determined based on the obtained predicted region information and the actual region information in the corresponding sample label, and the parameters in the current initial annotation model are updated based on the determined loss.

[0236] This embodiment does not limit the specific method of determining the loss, nor the specific method of updating the model parameters, nor the stopping condition of training.

[0237] Specifically, the loss can be calculated by comparing the difference between the predicted and actual regional information, or by calculating the non-overlapping parts between the predicted and actual regional information, and so on.

[0238] The model parameters are updated based on the loss, either by loss backpropagation or by random updates.

[0239] The stopping condition for training can be that the training has been iterated to a preset number of times, or that the loss is less than a preset loss, etc.

[0240] The initial annotation model may include at least one generalization layer; the generalization layer may include at least one preset convolutional layer; the preset convolutional layer may be used to perform convolution on the input feature map using a preset convolutional kernel; the preset convolutional kernel may include at least one frozen parameter with a value of 0.

[0241] In one alternative embodiment, when specifically training the labeled model, the generalization layer can be specifically trained and updated.

[0242] One approach is to improve the generalization ability of the generalization layer and the labeled model by adding noise to the training samples before training.

[0243] Specifically, this can involve adding noise to the training samples in the training sample set to obtain multiple noisy training samples, thereby expanding the number of training samples and improving the training effect and generalization ability of the labeled model.

[0244] Therefore, optionally, during the training of the initial labeled model, a mask feature map can be obtained by setting a mask for the initial feature map output by the previous layer of the current target generalization layer, and then the obtained mask feature map can be input into the current target generalization layer to obtain the generalized feature map output by the current target generalization layer.

[0245] The target generalization layer can be any generalization layer in the labeled model; for ease of description, we will refer to any generalization layer as the target generalization layer. Since the parameters of the target generalization layer may be updated as training progresses, we can input the mask feature map for the current target generalization layer and obtain the generalized feature map output by the current target generalization layer.

[0246] This embodiment does not limit the method of setting the mask, as long as it can add noise by adding a mask. This embodiment also does not limit the specific noise addition method. Specifically, it can be Gaussian random noise.

[0247] Training with a noisy mask feature map can also increase the number of training samples and improve the generalization ability of the generalization layer.

[0248] This embodiment can improve the generalization ability of the generalization layer by adding noise to the feature maps during the training process.

[0249] Understandably, the various generalization layers in the labeled model can also be trained using the same noisy training method. The mask settings between different generalization layers can be the same or different.

[0250] Alternatively, during training, only the generalization layer can be trained and its parameters updated, while other layers in the annotation model can remain unupdated, thereby improving training efficiency. Of course, the initial annotation model itself can be pre-trained and possess some image annotation capabilities.

[0251] Correspondingly, when training the generalization layer, we can focus on adding noise to the feature maps.

[0252] Optionally, in any round of training, the parameters of the current target generalization layer can be updated according to the preset generalization loss; specifically, other parameters in the labeled model may not need to be updated.

[0253] Specifically, the method for determining the preset generalization loss may include: in this round of training, inputting the initial feature map output by the previous layer of the current target generalization layer into the target generalization layer to obtain the label feature map output by the current target generalization layer, and determining the preset generalization loss based on the difference between the obtained label feature map and the generalization feature map output by the current target generalization layer.

[0254] This can be achieved by comparing the generalized feature map obtained after processing the noisy feature map with the labeled feature map obtained after processing the unnoisy feature map, and determining the corresponding loss.

[0255] As the generalization layer is trained and its parameters are updated, the generalized feature maps obtained based on different noisy feature maps can gradually approach the label feature maps obtained based on noisy feature maps. In other words, the generalized feature maps with different noise addition methods can gradually approach the label feature maps, thereby enabling the generalization layer to learn the semantic information in the image, reduce the influence of noise, and improve the generalization ability of the generalization layer.

[0256] Alternatively, the generalized feature map and the mask feature map can have the same size, so that the input feature map and the output feature map of the generalization layer have the same size, reducing the impact on the feature map size of the layers before and after the generalization layer.

[0257] Correspondingly, the following at least two feature maps can have the same size: generalized feature map, mask feature map, initial feature map, and label feature map.

[0258] It should be noted that the above-mentioned noisy generalization training for the generalization layer can be performed only during the training process. After training, noisy processing can be omitted during the actual use of the object labeling model.

[0259] Therefore, optionally, for the trained object annotation model, the determined image to be annotated and the annotation prompt information can be input into the pre-trained object annotation model.

[0260] The object labeling model can be used to: extract image features from the input image to be labeled, extract prompt features from the input labeling prompt information, and merge the extracted image features and prompt features; output predicted region information based on the merging result; the predicted region information is used to characterize the region where the predicted labeled object is located in the input image.

[0261] Furthermore, the object annotation model also includes a generalization layer, the specific location of which is not limited. It can be used to extract image features, to provide cue features, to merge image features and cue features, or to decode and output predicted region information, etc.

[0262] For any generalization layer, the feature map output by the previous layer can be input into the current generalization layer for processing, and the feature map output by the current generalization layer can be input into the next cascaded layer. During this process, noise reduction can be performed on the feature map without adding noise.

[0263] In addition to the above-described method and process, this invention also provides a method embodiment for a display screen scenario.

[0264] As shown in Figure 7, Figure 7 is a flowchart illustrating another image annotation method according to an embodiment of the present invention.

[0265] S201: Determine the display screen image to be annotated, and the annotation prompt information; the annotation prompt information is used to characterize the annotation object that needs to be annotated.

[0266] S202: Input the determined display screen image to be labeled and the labeling prompt information into the pre-trained object labeling model.

[0267] The object labeling model can be used to: extract image features from the input image of the display screen to be labeled, extract prompt features from the input labeling prompt information, and merge the extracted image features and prompt features; based on the merging result, predicted region information can be output; the predicted region information can be used to characterize the region where the predicted labeled object is located in the input image.

[0268] The predicted region information is output based on the merging result. Specifically, it can be based on the attention mechanism of the merging result to predict the probability that each pixel in the input image belongs to the region where the labeled object is located, and output the predicted region information according to the predicted probability. The predicted region information can be used to characterize the region where the predicted labeled object is located in the input image.

[0269] Furthermore, the object annotation model may include at least one generalization layer; the generalization layer may include at least one preset convolutional layer; the preset convolutional layer can be used to perform convolution on the input feature map using a preset convolutional kernel; the preset convolutional kernel includes at least one frozen parameter with a value of 0. The generalization layer can improve the generalization ability of the object annotation model.

[0270] S203: Based on the predicted region information output by the object annotation model, annotate the display screen image to be annotated.

[0271] Optionally, the training method for the object annotation model includes:

[0272] Determine the training sample set; the features of any sample in the training sample set include image features, and the label of any sample includes: regional information of the area where the object is located in the image features used to characterize the sample;

[0273] Determine the initial annotation model;

[0274] The initial annotation model is trained using the training sample set to obtain the trained object annotation model;

[0275] The initial annotation model includes at least one generalization layer; the generalization layer includes at least one preset convolutional layer; the preset convolutional layer is used to perform convolution on the input feature map using a preset convolutional kernel; the preset convolutional kernel includes at least one frozen parameter with a value of 0.

[0276] Optionally, the training method for the object annotation model includes:

[0277] Determine the training sample set; the features of any sample in the training sample set include image features and annotation prompts. The annotation prompts in the sample features are used to characterize the objects that need to be annotated in the image features of the sample; the label of any sample includes: the actual region information that characterizes the region where the annotated objects are located in the image features of the sample.

[0278] Determine the initial annotation model;

[0279] The initial annotation model is trained using the training sample set, and the trained initial annotation model is determined as the object annotation model.

[0280] During the training of the initial annotation model, the sample features of any sample in the training sample set are input into the current initial annotation model to obtain the predicted region information output by the current initial annotation model. The loss is determined based on the obtained predicted region information and the actual region information in the corresponding sample label. The parameters in the current initial annotation model are updated based on the determined loss.

[0281] The initial annotation model includes at least one generalization layer; the generalization layer includes at least one preset convolutional layer; the preset convolutional layer is used to perform convolution on the input feature map using a preset convolutional kernel; the preset convolutional kernel includes at least one frozen parameter with a value of 0.

[0282] Optionally, different preset convolution kernels in different preset convolutional layers may have different sizes, and / or the proportion of the frozen parameters in the kernel coefficients may differ.

[0283] Optionally, the generalization layer includes cascaded downsampling and upsampling layers; the output of the downsampling layer is cascaded to the input of the upsampling layer.

[0284] The downsampling layer is used to downsample based on at least one preset convolutional layer; and / or, the upsampling layer is used to extract features based on at least one preset convolutional layer.

[0285] Optionally, the downsampling layer is used to perform any of the following:

[0286] Downsampling is performed based on at least two pre-defined convolutional layers in a cascade; in the at least two pre-defined convolutional layers in a cascade, the output of any pre-defined convolutional layer is cascaded to the input of the next pre-defined convolutional layer;

[0287] Downsampling is performed on different preset convolutional layers to obtain downsampled intermediate feature maps of different sizes. The obtained downsampled intermediate feature maps are then combined to obtain the downsampled result feature map.

[0288] Downsampling is performed based on at least two cascaded pre-defined convolutional layers; the output of any pre-defined convolutional layer is cascaded to the input of the next pre-defined convolutional layer; the downsampled intermediate feature maps of the outputs of each cascaded pre-defined convolutional layer are combined to obtain the downsampled result feature map.

[0289] Optionally, the upsampling layer is used for:

[0290] If any upsampled intermediate feature map is obtained through upsampling, features are extracted from the obtained upsampled intermediate feature map based on at least one preset convolutional layer.

[0291] Optionally, the generalization layer includes: a feature extraction layer, a downsampling layer, and an upsampling layer;

[0292] The feature extraction layer is used to extract the first feature map from the input feature map;

[0293] The downsampling layer is used to downsample the first feature map to obtain the downsampling result;

[0294] The upsampling layer is used to fuse the downsampling result and the first feature map, and then upsample the fused result.

[0295] Optionally, during the training of the initial labeled model, a mask feature map is obtained by setting a mask for the initial feature map output by the previous layer of the current target generalization layer, and then the obtained mask feature map is input into the current target generalization layer to obtain the generalized feature map output by the current target generalization layer.

[0296] Optionally, in any round of training, the parameters of the current target generalization layer are updated according to the preset generalization loss;

[0297] The method for determining the preset generalization loss includes: in this round of training, inputting the initial feature map output by the previous layer of the current target generalization layer into the current target generalization layer to obtain the label feature map output by the current target generalization layer, and determining the preset generalization loss based on the difference between the obtained label feature map and the generalization feature map output by the current target generalization layer.

[0298] Optionally, the generalized feature map and the mask feature map have the same size.

[0299] Optionally, the features of any sample in the training sample set may also include annotation prompts; the annotation prompts are used to characterize the objects that need to be annotated in the image features of the sample.

[0300] Optionally, determining the initial annotation model includes: determining the initial annotation model; and adding at least one generalization layer to the determined initial annotation model to obtain the currently determined initial annotation model.

[0301] Optionally, determining the initial annotation model includes: obtaining an initial annotation model; the initial annotation model includes an image encoder, a cue encoder, and a decoder; the image encoder is used to extract image features from the input image; the cue encoder is used to extract cue features from the input annotation cue information; the decoder is used to: merge the image features extracted by the image encoder and the cue features extracted by the cue encoder; output prediction region information based on the merging result; the prediction region information is used to characterize the region where the predicted annotation object is located in the input image; for the obtained initial annotation model, at least one generalization layer is added to the image encoder to obtain the currently determined initial annotation model.

[0302] Optionally, the labeling information includes at least one of the following:

[0303] Textual information used to characterize the objects to be annotated;

[0304] It includes the bounding box information of the area where the labeled object is located;

[0305] Includes prompt area information about the area where the object to be labeled is located;

[0306] Point information located within the area containing the object to be labeled;

[0307] Information about points located outside the area where the object to be labeled is located.

[0308] For a detailed explanation of this embodiment, please refer to the preceding text.

[0309] For ease of understanding, this invention also provides an application embodiment.

[0310] This embodiment demonstrates image annotation in a display screen scenario.

[0311] This embodiment proposes an intelligent sample annotation method and system, which is mainly used to solve the problems of slow, inefficient, difficult and slow product launch caused by the need for manual sample annotation in the process of using deep learning in the display industry.

[0312] This embodiment makes strong adaptive adjustments to the object annotation model for the display industry, enabling one-click or multi-point annotation in most scenarios in the display industry. This function can improve annotation efficiency by ten or even dozens of times.

[0313] This embodiment can optimize the structure of the object annotation model according to the characteristics of the display screen data, and train the optimized object annotation model with display screen industry data, so that the model has strong adaptability to the display screen industry, realizes a rapid improvement in the annotation efficiency of the display screen industry, thereby accelerating the industrial AI deployment process, reducing the product iteration cycle, and improving the quality and efficiency of the factory.

[0314] This embodiment uses a method of optimizing and training an object annotation model based on display screen industry data. By interactively segmenting and marking the labeled targets on the display screen data with minimal prompts, mask information is automatically obtained, thereby achieving the goal of intelligent annotation of display screen industry data.

[0315] Because the target in a display screen scene is smaller, the scene is more diverse, and the mobility is poor compared to a natural scene, the model needs to be optimized to better suit the needs of the display screen.

[0316] Therefore, in this embodiment, the focus is on migrating the object annotation model to the display screen industry. Based on the characteristics of display screen industry data, the structure of the object annotation model is optimized in a targeted manner, and the object annotation model is trained using display screen industry data to make it highly adaptable to display screen scenarios.

[0317] This embodiment involves the migration of datasets and focuses on data processing, therefore the image encoder needs to be optimized.

[0318] This embodiment proposes adding a generalization layer during the image encoding stage.

[0319] As shown in Figure 8, Figure 8 is a schematic diagram of another generalization layer structure according to an embodiment of the present invention.

[0320] The generalization layer can be an improved version of the Unet structure. First, random Gaussian noise is added to the input image. The purpose of adding noise is to randomly mask the original image. This method forces the model to learn data features, increasing the image's generalization ability and thus achieving high accuracy in new scenes. This step is important because display screen scenes differ from natural scenes in that they are highly independent, with weak correlations between scenes. To create a universal display screen annotation software, the model's generalization ability needs to be emphasized.

[0321] After the original feature maps are noise-added, they are fed into the lightweight network Mobilenetv3 for feature extraction. Mobilenetv3 is used here because this model is very accurate in extracting features from images in the display industry, and the number of network parameters is small, which can achieve rapid training of industrial data while ensuring accuracy.

[0322] After feature extraction, the convolutional combination is optimized using the dilation rate by performing multiple dilated convolutions to achieve a pyramid pooling effect. This step improves the extraction performance of multi-scale semantic features. Specifically, four cascaded dilated convolutional layers and one pooling layer can be used, and the multi-scale semantic features can be aggregated through the convolutional layers.

[0323] Finally, the features extracted by pyramid pooling and the features extracted by Mobilenetv3 are fused together, followed by a convolution and upsampling to obtain the output feature map.

[0324] This embodiment introduces a generalization layer in the image encoding stage, which can effectively enhance features by adding a feature extraction mechanism while keeping the input and output dimensions unchanged. In addition, it also expands the feature field of view, which is beneficial for the model to perceive small targets and the whole image.

[0325] After the model is optimized, it is trained on a large dataset of the display industry. In this embodiment, approximately 5 million industry data points can be used for training to obtain the display object annotation model.

[0326] After the display screen object annotation model is generated, this embodiment can utilize the interactive and promptable segmentation concept of the display screen object annotation model to input some prompt information into the model, such as the object's position information. This embodiment can provide two modes: point information and box information of the object's position, allowing users to choose one. By encoding and embedding this information, it is fused with the image feature space, and then the decoder effectively maps the image onto the mask to obtain the complete annotation information of the object.

[0327] Among them, point information refers to the information of a point located in the area where the object is located. Point information is divided into two types: positive selection and negative selection. Positive selection indicates that the current point is within the area where the object is located, while negative selection indicates that the current point is outside the area where the object is located. Box information is rectangular tooltip information, and the labeled object is usually contained within the box.

[0328] In this embodiment, a post-processing flow after mask generation is also designed. After the mask is generated, it is generated into a shape that can be adjusted, such as a polygon or a rectangle. A point set consisting of a finite number of points is generated by sampling the polygon, and four corner points are generated for the rectangle. After the generation is completed, the shape can be manually adjusted. The polygon is adjusted by moving the outline points, and the rectangle is adjusted by adjusting the position of the four corner points.

[0329] The software supports both manual and intelligent annotation modes. Manual annotation supports rectangular, polygonal, circular, and linear annotation modes.

[0330] Intelligent annotation loads the local object annotation model and extracts the target object by clicking the point / box button on the interface. When using point selection position information, it can be divided into positive position information and negative position information, representing correct position information and incorrect position information, respectively.

[0331] By repeatedly selecting interactive location points, the correct mask information is finally formed.

[0332] For a specific example, please refer to Figure 4.

[0333] Once the mask is generated, the label information is input to form the mask information for the current intelligent recognition. This mask information will be saved locally in JSON file format.

[0334] If you are not satisfied with the recognition result, you can generate polygons and then manually drag the polygon point set to adjust the mask shape. As shown in Figure 4, the surrounding circle of the lower right image is the polygon point set, and each point can be dragged for adjustment.

[0335] Once the entire process is complete, the data annotation will also end.

[0336] This embodiment proposes a function to optimize the annotation model according to the characteristics of the display screen scene; proposes a function to migrate the annotation model to the display screen scene using display screen data; and proposes a function to perform post-processing adjustment after intelligent annotation.

[0337] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they have not been described one by one. Therefore, the arbitrary combination of various technical features in the above embodiments also falls within the scope of this invention.

[0338] Corresponding to the above method embodiments, this invention also provides an apparatus embodiment.

[0339] As shown in Figure 9, Figure 9 is a structural schematic diagram of an image annotation device according to an embodiment of the present invention.

[0340] The device may include the following units.

[0341] The determining unit 301 is used to determine the image to be labeled and the labeling prompt information; the labeling prompt information is used to characterize the labeling object that needs to be labeled.

[0342] The annotation unit 302 is used to input the determined image to be annotated and the annotation prompt information into the pre-trained object annotation model; and to annotate the image to be annotated according to the predicted region information output by the object annotation model.

[0343] The object labeling model can be used to: extract image features from the input image to be labeled, extract prompt features from the input labeling prompt information, and merge the extracted image features and prompt features; based on the merging result, predicted region information can be output; the predicted region information can be used to characterize the region where the predicted labeled object is located in the input image.

[0344] The predicted region information is output based on the merging result. Specifically, it can be based on the attention mechanism of the merging result to predict the probability that each pixel in the input image belongs to the region where the labeled object is located, and output the predicted region information according to the predicted probability. The predicted region information can be used to characterize the region where the predicted labeled object is located in the input image.

[0345] Furthermore, the object annotation model may include at least one generalization layer; the generalization layer may include at least one preset convolutional layer; the preset convolutional layer can be used to perform convolution on the input feature map using a preset convolutional kernel; the preset convolutional kernel includes at least one frozen parameter with a value of 0. The generalization layer can improve the generalization ability of the object annotation model.

[0346] Optionally, the training method for the object annotation model includes:

[0347] Determine the training sample set; the features of any sample in the training sample set include image features, and the label of any sample includes: regional information of the area where the object is located in the image features used to characterize the sample;

[0348] Determine the initial annotation model;

[0349] The initial annotation model is trained using the training sample set to obtain the trained object annotation model;

[0350] The initial annotation model includes at least one generalization layer; the generalization layer includes at least one preset convolutional layer; the preset convolutional layer is used to perform convolution on the input feature map using a preset convolutional kernel; the preset convolutional kernel includes at least one frozen parameter with a value of 0.

[0351] Optionally, the training method for the object annotation model includes:

[0352] Determine the training sample set; the features of any sample in the training sample set include image features and annotation prompts. The annotation prompts in the sample features are used to characterize the objects that need to be annotated in the image features of the sample; the label of any sample includes: the actual region information that characterizes the region where the annotated objects are located in the image features of the sample.

[0353] Determine the initial annotation model;

[0354] The initial annotation model is trained using the training sample set, and the trained initial annotation model is determined as the object annotation model.

[0355] During the training of the initial annotation model, the sample features of any sample in the training sample set are input into the current initial annotation model to obtain the predicted region information output by the current initial annotation model. The loss is determined based on the obtained predicted region information and the actual region information in the corresponding sample label. The parameters in the current initial annotation model are updated based on the determined loss.

[0356] The initial annotation model includes at least one generalization layer; the generalization layer includes at least one preset convolutional layer; the preset convolutional layer is used to perform convolution on the input feature map using a preset convolutional kernel; the preset convolutional kernel includes at least one frozen parameter with a value of 0.

[0357] Optionally, different preset convolution kernels in different preset convolutional layers may have different sizes, and / or the proportion of the frozen parameters in the kernel coefficients may differ.

[0358] Optionally, the generalization layer includes cascaded downsampling and upsampling layers; the output of the downsampling layer is cascaded to the input of the upsampling layer.

[0359] The downsampling layer is used to downsample based on at least one preset convolutional layer; and / or, the upsampling layer is used to extract features based on at least one preset convolutional layer.

[0360] Optionally, the downsampling layer is used to perform any of the following:

[0361] Downsampling is performed based on at least two pre-defined convolutional layers in a cascade; in the at least two pre-defined convolutional layers in a cascade, the output of any pre-defined convolutional layer is cascaded to the input of the next pre-defined convolutional layer;

[0362] Downsampling is performed on different preset convolutional layers to obtain downsampled intermediate feature maps of different sizes. The obtained downsampled intermediate feature maps are then combined to obtain the downsampled result feature map.

[0363] Downsampling is performed based on at least two cascaded pre-defined convolutional layers; the output of any pre-defined convolutional layer is cascaded to the input of the next pre-defined convolutional layer; the downsampled intermediate feature maps of the outputs of each cascaded pre-defined convolutional layer are combined to obtain the downsampled result feature map.

[0364] Optionally, the upsampling layer is used for:

[0365] If any upsampled intermediate feature map is obtained through upsampling, features are extracted from the obtained upsampled intermediate feature map based on at least one preset convolutional layer.

[0366] Optionally, the generalization layer includes: a feature extraction layer, a downsampling layer, and an upsampling layer;

[0367] The feature extraction layer is used to extract the first feature map from the input feature map;

[0368] The downsampling layer is used to downsample the first feature map to obtain the downsampling result;

[0369] The upsampling layer is used to fuse the downsampling result and the first feature map, and then upsample the fused result.

[0370] Optionally, during the training of the initial labeled model, a mask feature map is obtained by setting a mask for the initial feature map output by the previous layer of the current target generalization layer, and then the obtained mask feature map is input into the current target generalization layer to obtain the generalized feature map output by the current target generalization layer.

[0371] Optionally, in any round of training, the parameters of the current target generalization layer are updated according to the preset generalization loss;

[0372] The method for determining the preset generalization loss includes: in this round of training, inputting the initial feature map output by the previous layer of the current target generalization layer into the current target generalization layer to obtain the label feature map output by the current target generalization layer, and determining the preset generalization loss based on the difference between the obtained label feature map and the generalization feature map output by the current target generalization layer.

[0373] Optionally, the generalized feature map and the mask feature map have the same size.

[0374] Optionally, the features of any sample in the training sample set may also include annotation prompts; the annotation prompts are used to characterize the objects that need to be annotated in the image features of the sample.

[0375] Optionally, determining the initial annotation model includes: determining the initial annotation model; and adding at least one generalization layer to the determined initial annotation model to obtain the currently determined initial annotation model.

[0376] Optionally, determining the initial annotation model includes: obtaining an initial annotation model; the initial annotation model includes an image encoder, a cue encoder, and a decoder; the image encoder is used to extract image features from the input image; the cue encoder is used to extract cue features from the input annotation cue information; the decoder is used to: merge the image features extracted by the image encoder and the cue features extracted by the cue encoder; output prediction region information based on the merging result; the prediction region information is used to characterize the region where the predicted annotation object is located in the input image; for the obtained initial annotation model, at least one generalization layer is added to the image encoder to obtain the currently determined initial annotation model.

[0377] Optionally, the labeling information includes at least one of the following:

[0378] Textual information used to characterize the objects to be annotated;

[0379] It includes the bounding box information of the area where the labeled object is located;

[0380] Includes prompt area information about the area where the object to be labeled is located;

[0381] Point information located within the area containing the object to be labeled;

[0382] Information about points located outside the area where the object to be labeled is located.

[0383] For a detailed explanation, please refer to the above method implementation examples.

[0384] This invention also provides a computer device, which includes at least a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described method embodiments.

[0385] This invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the above method embodiments.

[0386] Figure 10 is a schematic diagram of the hardware structure of a computer device configured according to an embodiment of the present invention. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are internally connected to each other via the bus 1050.

[0387] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0388] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of the present invention are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0389] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0390] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0391] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0392] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of the present invention, and not necessarily all the components shown in the figures.

[0393] This invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described method embodiments.

[0394] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the above-described method embodiments.

[0395] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0396] This invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement any of the above-described method embodiments.

[0397] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the embodiments of the present invention, in essence or the part that contributes, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0398] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0399] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. In implementing the embodiments of the present invention, the functions of each module can be implemented in one or more software and / or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0400] The above description is merely a specific implementation of the embodiments of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the embodiments of the present invention, and these improvements and modifications should also be considered as protection of the embodiments of the present invention.

[0401] In this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "multiple" refers to two or more unless otherwise expressly defined.

[0402] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. The invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0403] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. An image annotation method, characterized in that, include: Identify the image to be labeled and the labeling prompts; The annotation prompt information is used to identify the annotation object that needs to be annotated; The identified images to be labeled and the labeling prompts are input into a pre-trained object labeling model; The object annotation model is used to: extract image features from the input image to be annotated, extract prompt features from the input annotation prompt information, and merge the extracted image features and prompt features; The predicted region information is output based on the merging result; the predicted region information is used to characterize the region where the predicted labeled object is located in the input image; the object labeling model includes at least one generalization layer; the generalization layer includes at least one preset convolutional layer; the preset convolutional layer is used to perform convolution on the input feature map using a preset convolutional kernel; the preset convolutional kernel includes at least one freeze parameter with a value of 0; The image to be labeled is labeled based on the predicted region information output by the object labeling model.

2. The method according to claim 1, characterized in that, The training method for the object annotation model includes: A training sample set is determined; the features of any sample in the training sample set include image features and annotation prompts, the annotation prompts in the sample features are used to characterize the annotation objects that need to be annotated in the image features of the sample; the label of any sample includes: actual region information used to characterize the region where the annotation objects are located in the image features of the sample; Determine the initial annotation model; The initial annotation model is trained using the training sample set, and the trained initial annotation model is determined as the object annotation model; During the training of the initial annotation model, the sample features of any sample in the training sample set are input into the current initial annotation model to obtain the predicted region information output by the current initial annotation model. The loss is determined based on the obtained predicted region information and the actual region information in the corresponding sample label. The parameters in the current initial annotation model are updated based on the determined loss. The initial annotation model includes at least one generalization layer; the generalization layer includes at least one preset convolutional layer; the preset convolutional layer is used to perform convolution on the input feature map using a preset convolutional kernel; the preset convolutional kernel includes at least one frozen parameter with a value of 0.

3. The method according to claim 1 or 2, characterized in that, Different preset convolutional kernels in different preset convolutional layers have different sizes and / or different proportions of the frozen parameters in the kernel coefficients.

4. The method according to claim 1 or 2, characterized in that, The generalization layer includes cascaded downsampling layers and upsampling layers; the output of the downsampling layer is cascaded to the input of the upsampling layer. The downsampling layer is used to perform downsampling based on at least one preset convolutional layer; And / or, the upsampling layer is used for feature extraction based on at least one preset convolutional layer.

5. The method according to claim 4, characterized in that, The downsampling layer is used to perform any of the following: Downsampling is performed based on at least two cascaded preset convolutional layers; in the at least two cascaded preset convolutional layers, the output of any preset convolutional layer is cascaded to the input of the next preset convolutional layer; Downsampling is performed on different preset convolutional layers to obtain downsampled intermediate feature maps of different sizes. The obtained downsampled intermediate feature maps are then combined to obtain the downsampled result feature map. Downsampling is performed based on at least two cascaded pre-defined convolutional layers; the output of any pre-defined convolutional layer is cascaded to the input of the next pre-defined convolutional layer; the downsampled intermediate feature maps of the outputs of each cascaded pre-defined convolutional layer are combined to obtain the downsampled result feature map.

6. The method according to claim 4, characterized in that, The upsampling layer is used for: If any upsampled intermediate feature map is obtained through upsampling, features are extracted from the obtained upsampled intermediate feature map based on at least one preset convolutional layer.

7. The method according to claim 4, characterized in that, The generalization layer includes: a feature extraction layer, the downsampling layer, and the upsampling layer; The feature extraction layer is used to extract a first feature map from the input feature map; The downsampling layer is used to downsample the first feature map to obtain the downsampling result; The upsampling layer is used to: fuse the downsampling result and the first feature map, and then upsample the fused result.

8. The method according to claim 2, characterized in that, During the training of the initial labeled model, a mask feature map is obtained by setting a mask for the initial feature map output by the previous layer of the current target generalization layer. The obtained mask feature map is then input into the current target generalization layer to obtain the generalized feature map output by the current target generalization layer.

9. The method according to claim 8, characterized in that, In any round of training, the parameters of the current target generalization layer are updated according to the preset generalization loss; The method for determining the preset generalization loss includes: in this round of training, inputting the initial feature map output by the previous layer of the current target generalization layer into the current target generalization layer to obtain the label feature map output by the current target generalization layer, and determining the preset generalization loss based on the difference between the obtained label feature map and the generalization feature map output by the current target generalization layer.

10. The method according to claim 8, characterized in that, The generalized feature map and the mask feature map have the same size.

11. The method according to claim 2, characterized in that, The determination of the initial annotation model includes: An initial annotation model is obtained; the initial annotation model includes an image encoder, a cue encoder, and a decoder; the image encoder is used to extract image features from the input image; the cue encoder is used to extract cue features from the input annotation cue information; The decoder is used to: merge the image features extracted by the image encoder with the cue features extracted by the cue encoder; output predicted region information based on the merging result; the predicted region information is used to characterize the region where the predicted labeled object is located in the input image; For the obtained initial annotation model, at least one generalization layer is added to the image encoder to obtain the currently determined initial annotation model.

12. The method according to claim 1 or 2, characterized in that, The annotation prompt information includes at least one of the following: Textual information used to characterize the objects to be annotated; It includes the bounding box information of the area where the labeled object is located; Includes prompt area information about the area where the object to be labeled is located; Point information located within the area containing the object to be labeled; Information about points located outside the area where the object to be labeled is located.

13. An image annotation device, characterized in that, include: The determining unit is used to determine the image to be labeled and the labeling prompt information; The annotation prompt information is used to identify the annotation object that needs to be annotated; The annotation unit is used to input the determined image to be annotated and the annotation prompt information into the pre-trained object annotation model; The object annotation model is used to: extract image features from the input image to be annotated, extract prompt features from the input annotation prompt information, and merge the extracted image features and prompt features; The predicted region information is output based on the merging result; the predicted region information is used to characterize the region where the predicted labeled object is located in the input image; the object labeling model includes at least one generalization layer; the generalization layer includes at least one preset convolutional layer; the preset convolutional layer is used to perform convolution on the input feature map using a preset convolutional kernel; the preset convolutional kernel includes at least one freeze parameter with a value of 0; The image to be labeled is labeled based on the predicted region information output by the object labeling model.

14. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1 to 12.

15. A computer-readable storage medium storing a computer program, characterized in that, The computer program, when executed by a processor, implements the method of any one of claims 1 to 12.

16. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method of any one of claims 1 to 12.

Images