Image annotation processing method and device, electronic equipment and medium

Abstract

The invention relates to the technical field of image annotation processing, and provides an image annotation processing method and device, electronic equipment and a medium. According to the method, feature extraction processing can be performed on a to-be-recognized image to obtain a feature vector of the to-be-recognized image, and feature extraction processing is performed on a prompt image to obtain a feature vector of the prompt image; performing feature extraction processing on the position information of the labeled object in the prompt object to obtain a feature vector of the position information; performing feature extraction processing on the prompt text to obtain a feature vector of the prompt text; performing fusion processing on the feature vector of the prompt image, the feature vector of the position information and the feature vector of the prompt text to obtain a prompt vector of the annotation object; and according to the feature vector of the to-be-identified image and the prompt vector of the annotation object, annotating the to-be-identified image to obtain annotation information of the annotation object in the to-be-identified object. Through the feature extraction and fusion processing method, the image annotation effect and efficiency are remarkably improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image annotation processing, and in particular to an image annotation processing method, device, electronic device and medium. Background Art

[0002] With the rapid development of Internet technology, image annotation technology has gradually become an important research direction. Annotating image content through annotation models can provide important reference for subsequent strategy formulation. However, although the existing intelligent annotation algorithms have achieved certain results, there are still some shortcomings. Among them, some annotation models are based on meta-learning training. Meta-learning is a method of training models, and its goal is to enable the model to quickly adapt to new tasks or data sets. However, the annotation model trained by meta-learning has the problem of insufficient detection accuracy in small sample detection algorithms such as metric learning. Metric learning is a method of learning similarity metrics, which is used to perform detection and classification tasks in small sample conditions. Therefore, in response to this problem, the present invention aims to propose an improved image annotation method. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure provide an image annotation processing method, device, electronic device and computer-readable storage medium to solve the technical problem in the prior art that the annotation model trained by meta-learning has insufficient detection accuracy in small sample detection algorithms such as metric learning.

[0004] In a first aspect of an embodiment of the present disclosure, a method for image annotation is provided, including: obtaining an image to be identified, a prompt image, position information of an annotated object in a prompt object, and a prompt text; performing feature extraction processing on the image to be identified to obtain a feature vector of the image to be identified, and performing feature extraction processing on the prompt image to obtain a feature vector of the prompt image; performing feature extraction processing on the position information of the annotated object in the prompt object to obtain a feature vector of the position information; performing feature extraction processing on the prompt text to obtain a feature vector of the prompt text; performing fusion processing on the feature vector of the prompt image, the feature vector of the position information, and the feature vector of the prompt text to obtain a prompt vector of the annotated object; annotating the image to be identified according to the feature vector of the image to be identified and the prompt vector of the annotated object to obtain annotation information of at least one annotated object in the object to be identified.

[0005] According to a second aspect of the disclosed embodiments, an image annotation processing device is provided, including: an acquisition module, used to acquire an image to be identified, a prompt image, position information of an annotated object in a prompt object, and a prompt text; a first feature extraction module, used to perform feature extraction processing on the image to be identified to obtain a feature vector of the image to be identified, and to perform feature extraction processing on the prompt image to obtain a feature vector of the prompt image; a second feature extraction module, used to perform feature extraction processing on the position information of the annotated object in the prompt object to obtain a feature vector of the position information; a third feature extraction module, used to perform feature extraction processing on the prompt text to obtain a feature vector of the prompt text; a fusion module, used to perform fusion processing on the feature vector of the prompt image, the feature vector of the position information, and the feature vector of the prompt text to obtain a prompt vector of the annotated object; and a labeling module, used to label the image to be identified according to the feature vector of the image to be identified and the prompt vector of the annotated object to obtain labeling information of at least one annotated object in the object to be identified.

[0006] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0007] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0008] Compared with the prior art, the disclosed embodiment has the following beneficial effects: the disclosed embodiment can extract and fuse multiple information sources (images to be identified, prompt images, location information, and prompt text) to more comprehensively capture and understand the features of the labeled object, thereby improving the labeling accuracy, especially the recognition ability in complex scenes. Through the fusion of feature vectors, limited sample data can be effectively used for labeling, the detection accuracy in the case of small samples can be improved, and the challenges faced by traditional methods in small sample learning can be solved. It can also be flexibly applied to different types of image labeling tasks, has strong adaptability, and can handle various scenes and labeling requirements. In addition, through the fusion processing of feature vectors, the dependence on a single information source is reduced, making the labeling process more efficient, and the labeling task can be completed in a shorter time to meet the needs of real-time applications. The technical solution reduces the dependence on manual intervention and has a high degree of automation, thereby reducing labor costs and the risk of human errors and improving overall work efficiency. The solution is not only suitable for image labeling, but can also be extended to multiple fields such as video analysis, intelligent monitoring, and autonomous driving, and has broad application potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present invention can be applied;

[0011] Figure 2 It is a flowchart of an image annotation processing method provided by an embodiment of the present disclosure;

[0012] Figure 3 is a flowchart of another image annotation processing method provided by an embodiment of the present disclosure;

[0013] Figure 4 is a block diagram of an image annotation processing device provided by an embodiment of the present disclosure;

[0014] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0016] It should be noted that the user information (including but not limited to terminal device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0017] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present invention can be applied.

[0018] like Figure 1As shown, the system architecture 100 may include one or more of a first terminal device 101, a second terminal device 102, and a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0019] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0020] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send data, etc. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, desktop computers, and the like.

[0021] The server 105 may be a server that provides various services. For example, the server 105 may obtain the image to be identified, the prompt image, the position information of the marked object in the prompt object, and the prompt text from the first terminal device 103 (which may also be the second terminal device 102 or the third terminal device 103); perform feature extraction processing on the image to be identified to obtain the feature vector of the image to be identified, and perform feature extraction processing on the prompt image to obtain the feature vector of the prompt image; perform feature extraction processing on the position information of the marked object in the prompt object to obtain the feature vector of the position information; perform feature extraction processing on the prompt text to obtain the feature vector of the prompt text; perform fusion processing on the feature vector of the prompt image, the feature vector of the position information, and the feature vector of the prompt text to obtain the prompt vector of the marked object; perform annotation processing on the image to be identified according to the feature vector of the image to be identified and the prompt vector of the marked object, and obtain the annotation information of at least one marked object in the object to be identified.

[0022] In some embodiments, the image annotation processing method provided by the embodiment of the present invention is generally executed by the server 105, and accordingly, the image annotation processing device is generally set in the server 105. In other embodiments, some terminal devices may have functions similar to those of the server to execute the method. Therefore, the image annotation processing method provided by the embodiment of the present invention is not limited to being executed on the server side.

[0023] The image annotation processing method and device according to the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0024] Figure 2 is a flowchart of an image annotation processing method provided by an embodiment of the present disclosure. The method provided by an embodiment of the present disclosure can be executed by any electronic device with computer processing capabilities, for example, the electronic device can be Figure 1 The server is shown.

[0025] like Figure 2 As shown, the image annotation processing method includes steps S210 to S260.

[0026] In step S210, the image to be recognized, the prompt image, the position information of the marked object in the prompt object, and the prompt text are obtained.

[0027] Step S220 , performing feature extraction processing on the image to be identified to obtain a feature vector of the image to be identified, and performing feature extraction processing on the prompt image to obtain a feature vector of the prompt image.

[0028] In step S230, feature extraction processing is performed on the position information of the marked object in the prompt object to obtain a feature vector of the position information.

[0029] In step S240, feature extraction is performed on the prompt text to obtain a feature vector of the prompt text.

[0030] In step S250, the feature vector of the prompt image, the feature vector of the position information, and the feature vector of the prompt text are fused to obtain a prompt vector of the labeled object.

[0031] In step S260, the image to be identified is annotated according to the feature vector of the image to be identified and the hint vector of the annotated object, so as to obtain the annotation information of at least one annotated object among the objects to be identified.

[0032] This method can extract and fuse features from multiple information sources (images to be identified, prompt images, location information, and prompt text) to more comprehensively capture and understand the features of the labeled object, thereby improving the accuracy of labeling, especially the recognition ability in complex scenarios. Through the fusion of feature vectors, limited sample data can be effectively used for labeling, improving the detection accuracy in small sample cases, solving the challenges faced by traditional methods in small sample learning, and being flexibly applicable to different types of image labeling tasks. It has strong adaptability and can handle various scenarios and labeling requirements. In addition, through the fusion of feature vectors, the dependence on a single information source is reduced, making the labeling process more efficient, and the labeling task can be completed in a shorter time to meet the needs of real-time applications. This technical solution reduces the dependence on manual intervention and has a high degree of automation, thereby reducing labor costs and the risk of human errors and improving overall work efficiency. This solution is not only suitable for image labeling, but can also be extended to multiple fields such as video analysis, intelligent monitoring, and autonomous driving, and has broad application potential.

[0033] In some embodiments of the present disclosure, the above-mentioned image to be identified may refer to an image processed when performing a labeling task, and these images may contain multiple or one object to be identified and labeled. By processing the image to be identified, key information of the labeled object can be obtained, such as: Position information: The specific position of the labeled object in the image, usually expressed in the form of coordinates, which can indicate the starting point and the end point of the object. Size information: The size of the labeled object, including width, height, etc., helps to further analyze the characteristics and relative position of the object. The prompt image can be an image that provides guidance for the labeling process, and these images contain the labeled objects in the image to be identified. The role of the prompt image is to provide contextual information to help the model better understand the target in the image to be identified. The labeled object of the prompt image usually has the following characteristics: Position information: In the prompt image, the position information of the labeled object can be obtained by labeling it in advance. This is usually expressed as a bounding box or keypoints, which indicate the specific position and shape of the object in the image. The prompt text can be descriptive information about the labeled object, which is intended to further guide the labeling process. The prompt text can include the following: Description information: A detailed description of the features of the annotated object, such as color, shape, type, etc., to help the annotation model better understand the attributes of the object. Task instructions: Provide specific annotation task requirements, such as the object category to be annotated, the annotation method, etc., to ensure the accuracy and consistency of the annotation process. By combining the image to be identified, the prompt image and its location information and the prompt text, the accuracy and efficiency of annotation can be effectively improved, making the annotation process more intelligent and automated.

[0034] In some embodiments of the present disclosure, performing feature extraction processing on the image to be identified to obtain a feature vector of the image to be identified includes encoding the image to be identified through an image encoder to obtain a feature vector of the image to be identified; performing feature extraction processing on the prompt image to obtain a feature vector of the prompt image includes encoding the prompt image through an image encoder to obtain a feature vector of the prompt image. For example, a specific image encoder (e.g., VisionTransformer, ViT for short) is used to encode the image to be identified. The main function of the image encoder is to convert the input image into a high-dimensional feature vector, which can effectively capture important information and features in the image. After encoding, the image to be identified will be converted into a feature vector. This feature vector contains a variety of information of the image, such as color distribution, texture features, shape features, etc., which can provide a basis for subsequent annotation tasks. Similarly, the prompt image is also encoded by the image encoder. This ensures the consistency of the image to be identified and the prompt image in the feature space. After the prompt image is encoded, the generated feature vector also contains the key information of the annotated object in the prompt image. This information will help the model better understand the target in the image to be identified during the annotation process. In this embodiment, the image encoder is trained using the MAE (Masked Autoencoder) method. MAE is a self-supervised learning method that randomly masks parts of the input image and trains the model to reconstruct the masked parts, thereby learning the effective features of the image.

[0035] In practical applications, the parameters of the image encoder are "frozen" after training, that is, they are no longer updated. The purpose of this approach is to maintain the stability and consistency of the encoder during feature extraction, ensuring that it can effectively extract features from the image to be identified and the prompt image without being affected by parameter adjustments during subsequent task training. Through the above feature extraction process, the feature vectors of the image to be identified and the prompt image can be effectively extracted and used for subsequent annotation tasks. This method not only improves the accuracy and efficiency of feature extraction, but also provides richer contextual information for the annotation model, thereby improving the overall annotation effect.

[0036] In some embodiments of the present disclosure, feature extraction is performed on the position information of the annotated object in the prompt object, and obtaining the feature vector of the position information includes encoding the position information of the annotated object in the prompt object through a position encoder to obtain the feature vector of the position information. For example, the position information refers to the specific position of the annotated object in the prompt image, which is usually expressed in the form of coordinates, including: Bounding Box: defines the rectangular area of ​​the annotated object in the image, represented by the coordinates of the upper left corner and the lower right corner. Keypoints: specific key points of the annotated object, such as the coordinates of the corners or center points of the object. These position information are important bases for annotation, and can help the model accurately identify and locate the object in the image to be identified. In order to effectively process the position information, a position encoder (e.g., PositionPrompt Encoder) is used to encode the position information of the annotated object in the prompt object. The main task of the position encoder is to convert the original position information (such as coordinates) into a high-dimensional feature vector. This process can extract potential features related to the position information by mapping the position information to a specific feature space. The position encoder receives position information as input and generates a feature vector of position information through a series of linear transformations, activation functions and possible nonlinear mappings. This feature vector can effectively capture the contextual features of the position information. After being processed by the position encoder, the obtained position information feature vector contains the following information: Spatial relationship: describes the position of the annotated object in the prompt image and its relative relationship with other objects. Scale information: provides size and shape information of the annotated object to help the model understand the characteristics of the object. By extracting features from the position information of the annotated object in the prompt object, the generated position information feature vector will be combined with the feature vectors of the image to be identified and the prompt image to provide important contextual information for subsequent annotation tasks. This method not only enhances the model's ability to understand the annotated object, but also improves the accuracy and efficiency of annotation, making the annotation process more intelligent and automated.

[0037] In some embodiments of the present disclosure, feature extraction is performed on the prompt text to obtain a feature vector of the prompt text, including encoding the prompt text through a text encoder to obtain a feature vector of the prompt text. For example, the prompt text refers to descriptive text information related to the image to be recognized, which is usually used to guide the model to recognize specific objects or features in the image. The prompt text may include: descriptive words: such as "cat", "car", etc., referring to specific objects. Contextual information: such as "cat on the grass", provides the environment and location of the object. In order to effectively process the prompt text, a text encoder (for example, CLIP encoder) is used to encode it. The specific steps are as follows: The main task of the text encoder is to convert the original prompt text into a high-dimensional feature vector. This process extracts the potential semantic features of the text by mapping the text into a specific feature space. The text encoder receives the prompt text as input, and generally includes the following steps: Text preprocessing: the text is processed by word segmentation, stop word removal, etc., to facilitate subsequent encoding. Embedding generation: the processed text is converted into a preliminary vector representation using an embedding layer. Feature extraction: these embeddings are further processed by a deep learning model (such as a Transformer architecture) to generate a final text feature vector. After being processed by the text encoder, the resulting prompt text feature vector contains the following information: Semantic information: describes the core meaning of the prompt text and helps the model understand the relationship between text and image. Contextual features: provides contextual information related to the prompt text to enhance the model's understanding of the image content. By extracting features from the prompt text, the generated text feature vector will be combined with the feature vectors of the image to be identified and the prompt image to provide important semantic information for subsequent annotation tasks. This method not only improves the model's ability to understand the relationship between image and text, but also enhances the accuracy and efficiency of annotation, making the annotation process more intelligent and automated. At the same time, by utilizing the cross-modal characteristics of the CLIP encoder, the association between image and text becomes closer, further improving the performance of the overall model.

[0038] In some embodiments of the present disclosure, the feature vector of the prompt image, the feature vector of the position information, and the feature vector of the prompt text are fused to obtain the prompt vector of the annotated object. For example, the prompt image feature vector is extracted by an image encoder (such as the ViT architecture) and represents the visual features of the prompt image. These features include information such as color, shape, texture, etc., which can reflect the main content of the image. The position information feature vector is generated by a position encoder and represents the specific position of the annotated object in the prompt image. These feature vectors contain spatial relationships, scale information, etc., which help the model understand the location of the object. The prompt text feature vector is generated by a text encoder (such as a CLIP encoder) and captures the semantic information of the prompt text. These feature vectors can reflect the association between text and image and enhance the model's understanding ability. Information from different modalities (image, text, position) is integrated to form a comprehensive feature representation. This representation can better capture the characteristics of the annotated object and improve the recognition and annotation capabilities of the model. After fusion processing, the obtained hint vector of the annotated object combines the visual features of the hint image, the spatial features of the location information and the semantic features of the hint text to form a multimodal comprehensive feature representation, which enhances the model's ability to understand the annotated object and enables the model to more accurately identify and locate the object in the image to be identified.

[0039] By effectively fusing the prompt image feature vector, location information feature vector, and prompt text feature vector, the final generated prompt vector will provide strong support for subsequent annotation tasks. This fusion not only improves the accuracy and robustness of the model, but also enhances its adaptability to complex scenes, making the annotation process more efficient and intelligent.

[0040] In some embodiments of the present disclosure, the image to be identified is annotated according to the feature vector of the image to be identified and the prompt vector of the annotated object, and the annotation information of at least one annotated object in the object to be identified is obtained. For example, the feature vector of the image to be identified is extracted by an image encoder (such as the ViT architecture) and represents the visual features of the image to be identified. These features include color, shape, texture and other important information in the image, which can reflect the overall content of the image. The prompt vector of the annotated object is generated by fusing the prompt image feature vector, the position information feature vector and the prompt text feature vector, and contains comprehensive information related to the object to be identified. The vector provides semantic and spatial information about the annotated object to help the model to accurately annotate. The annotation process is performed according to the feature vector of the image to be identified and the prompt vector of the annotated object, and finally the annotation information of at least one annotated object is generated. This process not only improves the accuracy and efficiency of annotation, but also enhances the adaptability of the model in complex scenes. By combining multimodal features, the model can better understand the image content, thereby achieving more intelligent automatic annotation.

[0041] Figure 3 FIG. 1 is a flow chart of another image annotation processing method provided by an embodiment of the present disclosure. Figure 3 The above step S250 may specifically include steps S310 to S330.

[0042] In step S310, the feature vector of the prompt image, the feature vector of the position information, and the feature vector of the prompt text are processed by the proxy attention module to obtain an initial feature vector of the labeled object.

[0043] In step S320, the initial feature vector of the labeled object is processed by a feed-forward network to obtain a prompt vector of the labeled object.

[0044] This method can generate corresponding question text sequences based on each training text sequence through a large language model. Each training text sequence is spliced ​​with its corresponding question text sequence to obtain multiple question-answer training text sequences, and the vector retrieval target of the target version is updated based on each question-answer training text sequence in a supervised manner. In this way, the method of using a language model to generate questions, splice question-answer text sequences, and supervised learning can significantly improve the performance of the vector retrieval model, making it more effective and accurate in various applications.

[0045] In some embodiments of the present disclosure, in the prompt encoder (PromptEncoder), the design and function of each module is the key to achieve efficient labeling. The following is a detailed description of each component in the prompt encoder:

[0046] Position encoders can be used to process the position information of the annotated objects in the prompt image. Since image features are usually planar data, position encoders introduce position information to enable the model to understand the spatial relationship of the annotated objects in the image. Position encoding can be generated by position encoders. By adding the position encoding to the image feature vector, each feature vector contains position information. Position encoders can extract the spatial features of the image through convolutional layers or other neural network structures to further enhance the model's understanding of position information.

[0047] The text encoder can be used to process the prompt text and extract its semantic features. The text encoder usually uses a pre-trained language model (such as CLIP, etc.) to generate the feature vector of the text. The prompt text is input into the text encoder to generate the corresponding text embedding vector. These vectors capture the semantic information of the text and are combined with the image feature vector. The text encoder can understand the contextual information of the text and ensure that the generated feature vector can accurately reflect the meaning of the prompt text.

[0048] The proxy attention module can be used to process the feature vector of the prompt image, the feature vector of the position information, and the feature vector of the prompt text to generate an initial feature vector of the annotated object. This module enhances the interaction between different modal features through the attention mechanism. For example, by calculating the attention weights between different feature vectors, the model can focus on the information most relevant to the annotated object. This mechanism can be implemented through self-attention or cross-attention. The feature vectors processed by attention are fused to generate a comprehensive initial feature vector. This vector contains multimodal information from images, positions, and texts.

[0049] The feedforward network can be used to further process the initial feature vector of the annotated object to generate the final annotated object hint vector. The feedforward network is usually composed of multiple fully connected layers and can learn complex feature transformations. The feedforward network inputs the initial feature vector into multiple fully connected layers and performs nonlinear transformations through activation functions (such as ReLU or GELU). This process can enhance the expressiveness of the feature representation. The output layer of the feedforward network can be configured according to the requirements of different tasks, such as outputting information such as the category, location, and confidence of the annotated object.

[0050] Through the collaborative work of the above modules, the prompt encoder can effectively process the position information of the prompt image, the semantic information of the prompt text, and the image features. The specific process is as follows: The position encoder processes the position information in the prompt image and generates a position feature vector. The text encoder processes the prompt text and generates a text feature vector. The proxy attention module receives the feature vectors from the position encoder, text encoder, and prompt image, calculates the attention weights, and generates an initial feature vector. The feedforward network further processes the initial feature vector to obtain the final annotation object prompt vector. Through the design and processing of this series of modules, the prompt encoder can generate a comprehensive prompt vector that contains multimodal information, which provides strong support for subsequent annotation tasks and improves the accuracy and robustness of the model.

[0051] In the traditional attention mechanism, calculating the similarity between each query and all keys is a computationally intensive process, especially when dealing with large inputs, with a time complexity of O(N 2 d), where N is the number of input features and d is the feature dimension. The proxy attention module reduces the amount of computation by introducing an agent vector, thereby achieving more efficient global information modeling. The proxy attention module expands the attention triple (Q, K, V) into four groups (Q, A, K, V), where A is an additional set of agent vectors.

[0052] The proxy attention mechanism in the proxy attention module is as follows:

[0053] Step 1: Proxy feature aggregation, taking the proxy vector A as the query, and performing attention calculation with the key K and value V. The calculation method is:

[0054] Agent-Attention(Q,A,K,V)=Attention(Q,A,Attention(A,K,V))

[0055] Among them, the similarity between the proxy vector and all keys is calculated by the above formula. After the activation function, the weight obtained is used to aggregate the weighted value V to obtain the proxy feature V. Through the proxy feature aggregation, the model gathers information from all features and generates a proxy feature V containing the global context. In this example, K and V are the location information feature vector and the prompt text feature vector.

[0056] Step 2: Information broadcast, using the proxy vector A as the key and the proxy feature V as the value, and performing a second attention calculation with the query Q. The calculation method is:

[0057]

[0058] Among them, d k is the latitude of the vector, and T is the matrix transpose. In this embodiment, Q may be the feature vector of the above-mentioned prompt image.

[0059] The proxy attention mechanism realizes the information exchange between each query and key through the proxy vector, avoiding the direct calculation of the similarity between all queries and keys. This method not only improves efficiency, but also enhances the expressiveness of the model. The proxy attention module redefines the way of attention calculation by introducing proxy vectors, so that the model can aggregate and broadcast information more efficiently when processing complex features. This innovative four-element attention paradigm provides a new solution for deep learning models in terms of performance and computational efficiency, especially for tasks that require processing multimodal data. . The feature vector of the prompt image, the feature vector of the position information, and the feature vector of the prompt text are processed by the proxy attention module to obtain the initial feature vector of the labeled object.

[0060] Based on the above embodiment, the training and use process of the proxy attention module can adopt the following formula:

[0061] O=σ(QA T +B 2 )σ(AK T +B 1 )V+DWC(V)

[0062]

[0063] Where O represents the output of the agent attention module, and σ represents the result of the above agent feature aggregation or the above information broadcast. 1 and B 2 It is a trainable bias that can use spatial information to enhance the attention of ordinary agents and help different Agent tokens focus on different areas. DWC is a deep convolution that can be used to enhance the diversity of features. This module can effectively improve the expressive power of ordinary agent attention. x represents the similarity between the calculated agent vector and all keys, such as the QK above. T A represents the above-mentioned agent vector, Q represents the query vector, and KV may represent the feature vector of the prompt image, the feature vector of the position information, and the feature vector of the prompt text.

[0064] In some embodiments of the present disclosure, the mask encoder (MaskDecod er) performs annotation processing on the image to be identified according to the feature vector of the image to be identified and the prompt vector of the annotated object, and obtains the annotation information of at least one annotated object in the object to be identified. Wherein, the annotation processing on the image to be identified according to the feature vector of the image to be identified and the prompt vector of the annotated object to obtain the annotation information of at least one annotated object in the object to be identified includes: performing cross-attention fusion processing on the feature vector of the image to be identified and the prompt vector of the annotated object to obtain the initial mask vector of the image to be identified; and performing annotation processing on the image to be identified according to the initial mask vector of the image to be identified to obtain the annotation information of at least one annotated object in the object to be identified. For example, the mask encoder is used to perform annotation processing according to the feature vector of the image to be identified and the prompt vector of the annotated object to generate the annotation information of at least one annotated object in the object to be identified. The process generally includes the following steps: the image to be identified is first subjected to feature extraction by an image encoder to generate a high-dimensional feature vector. These feature vectors contain the visual information of the object in the image. The prompt vector is generally generated by a text encoder and contains semantic information related to the annotated object. These prompt vectors provide contextual information for the annotation process to help the model understand the object to be identified. By calculating the similarity between the image feature and the prompt vector, the weight obtained after Softmax is used to weight the prompt vector. Through the cross-attention calculation, the output is an initial mask vector, which combines image features and prompt information and can effectively represent the features of the object to be identified. It is further processed by a feedforward network or other processing modules. The goal of this step is to convert the initial mask vector into more specific annotation information. By processing the initial mask vector, the model generates annotation information for at least one annotated object in the object to be identified. This information may include: Category label: the category of the annotated object (such as people, cars, animals, etc.). Position coordinates: the position of the annotated object in the image, usually expressed in the form of a bounding box or a segmentation mask.

[0065] In some embodiments of the present disclosure, the image to be identified is annotated according to the initial mask vector of the image to be identified, and the annotation information of at least one annotated object in the object to be identified is obtained, including: performing transposed convolution processing on the initial mask vector of the image to be identified to obtain the transposed convolution vector of the image to be identified; performing cross-attention processing on the initial mask vector of the image to be identified to obtain the cross-attention vector of the image to be identified; performing annotating processing on the image to be identified according to the transposed convolution vector of the image to be identified and the cross-attention vector of the image to be identified, and obtaining the annotation information of at least one annotated object in the object to be identified. For example, in the image annotation task, the mask encoder further processes the initial mask vector of the image to be identified to generate detailed annotation information. The following is a detailed description of the process: The initial mask vector is generated by fusing the feature vector of the image to be identified and the prompt vector of the annotated object through the cross-attention mechanism. This vector contains preliminary information about the object to be identified in the image, laying the foundation for subsequent annotation processing. The annotation processing flow mainly includes the following steps: Transposed convolution (also called deconvolution or upsampling convolution) is an operation commonly used in image generation and segmentation tasks. Its main function is to restore low-resolution feature maps to higher-resolution space. The initial mask vector is used as input and transposed convolution is performed to obtain the transposed convolution vector. This operation upsamples the initial mask vector through the learned convolution kernel, so that the spatial resolution of the feature map is improved. The initial mask vector also participates in the cross-attention calculation to extract more information from other input features. Through the cross-attention calculation, the cross-attention vector of the image to be identified reflects the relationship between the initial mask vector and other features, enhancing the understanding of the object to be identified. Combining the transposed convolution vector with the cross-attention vector The transposed convolution vector and the cross-attention vector are combined for annotation processing. The goal of this step is to comprehensively consider the information of both to generate the final annotation result.

[0066] Based on the above embodiments, through transposed convolution and cross-attention processing, the model can obtain richer feature representation and enhance the accuracy of annotation. The application of transposed convolution enables the model to annotate on high-resolution feature maps, which is suitable for fine-grained image segmentation tasks. The use of the cross-attention mechanism allows the model to fuse information between different features and improves the ability to understand complex scenes. Through transposed convolution and cross-attention processing of the initial mask vector, the mask encoder can effectively generate high-quality annotation information, which is suitable for various image annotation tasks.

[0067] In some embodiments of the present disclosure, the image to be identified is annotated according to the transposed convolution vector of the image to be identified and the cross-attention vector of the image to be identified, and the annotation information of at least one annotated object in the object to be identified is obtained, including: processing the cross-attention vector of the image to be identified by a multi-layer perceptron to obtain the multi-layer perceptual vector of the image to be identified; performing dot product calculation on the transposed convolution vector of the image to be identified and the multi-layer perceptual vector of the image to be identified to obtain the target mask vector of the image to be identified; converting the target mask vector of the image to be identified to obtain the annotation information of at least one annotated object in the object to be identified. For example, in an image annotation task, the mask encoder further processes the transposed convolution vector and the cross-attention vector of the image to be identified to generate detailed annotation information. The following is a detailed description of the process: The transposed convolution vector is a vector obtained by transposed convolution processing, which contains high-resolution image feature information. The cross-attention vector is a vector calculated by the cross-attention mechanism, which reflects the relationship between the object to be identified and other features. The multi-layer perceptron is a feedforward neural network composed of multiple fully connected layers and can learn complex nonlinear mappings. It is usually used for feature transformation and classification tasks. The cross-attention vector is input into the multi-layer perceptron for processing. The vector processed by the multi-layer perceptron. This process may include multiple linear transformations and activation functions (such as ReLU), so that the model can extract more discriminative features. Dot product is a common vector operation that can be used to calculate the similarity or correlation between two vectors. In this step, the dot product is used to calculate the target mask vector by combining the information of the transposed convolution vector and the multi-layer perceptron vector. The transposed convolution vector and the multi-layer perceptron vector are dot-producted to obtain the target mask vector, which combines the transposed convolution features and the features processed by the multi-layer perceptron to better represent the features of the object to be identified. The target mask vector is further transformed to generate the final annotation information. This process may include the use of convolutional layers, upsampling layers, or other transformation methods to convert the target mask vector into a specific annotation result. After the transformation process, the final output result contains the annotation information of the object to be identified, including: Category label: the identified object category (for example, cat, dog, car, etc.). Position coordinates: the position of the object in the image, usually expressed in the form of a bounding box or position point. Confidence score: The model's confidence in the annotation results, usually used for subsequent threshold judgment.

[0068] Based on the above embodiments, the use of multi-layer perceptron enables the model to effectively extract and transform features, enhancing the understanding of the object to be identified. Through the dot product operation, the model can combine information from different sources to generate a more discriminative target mask vector. This method is applicable to a variety of labeling tasks, such as object detection, instance segmentation, and semantic segmentation, and can adapt to different application scenarios. Through multi-layer perceptron processing, dot product calculation and conversion processing of transposed convolution vectors and cross-attention vectors, the mask encoder can effectively generate high-quality labeling information, which is applicable to various image labeling tasks.

[0069] Figure 4 It is a block diagram of an image annotation processing device provided by an embodiment of the present disclosure.

[0070] like Figure 4 As shown, the image annotation processing device 400 includes an acquisition module 410 , a first feature extraction module 420 , a second feature extraction module 430 , a third feature extraction module 440 , a fusion module 450 and an annotation module 460 .

[0071] Specifically, the acquisition module 410 is used to acquire the image to be recognized, the prompt image, the position information of the marked object in the prompt object, and the prompt text.

[0072] The first feature extraction module 420 is used to perform feature extraction processing on the image to be identified to obtain a feature vector of the image to be identified, and to perform feature extraction processing on the prompt image to obtain a feature vector of the prompt image.

[0073] The second feature extraction module 430 is used to perform feature extraction processing on the position information of the marked object in the prompt object to obtain a feature vector of the position information.

[0074] The third feature extraction module 440 is used to perform feature extraction processing on the prompt text to obtain a feature vector of the prompt text.

[0075] The fusion module 450 is used to fuse the feature vector of the prompt image, the feature vector of the position information, and the feature vector of the prompt text to obtain the prompt vector of the labeled object.

[0076] The labeling module 460 is used to label the image to be identified according to the feature vector of the image to be identified and the prompt vector of the labeled object, so as to obtain labeling information of at least one labeled object among the objects to be identified.

[0077] The image annotation processing device 400 can extract and fuse features of multiple information sources (images to be identified, prompt images, location information and prompt texts) to more comprehensively capture and understand the features of the annotation object, thereby improving the annotation accuracy, especially the recognition ability in complex scenes. Through the fusion of feature vectors, limited sample data can be effectively used for annotation, the detection accuracy in small sample cases can be improved, the challenges faced by traditional methods in small sample learning can be solved, and it can also be flexibly applied to different types of image annotation tasks, with strong adaptability and the ability to handle various scenes and annotation requirements. In addition, through the fusion of feature vectors, the dependence on a single information source is reduced, making the annotation process more efficient, and the annotation task can be completed in a shorter time to meet the needs of real-time applications. This technical solution reduces the dependence on manual intervention and has a high degree of automation, thereby reducing labor costs and the risk of human errors and improving overall work efficiency. This solution is not only suitable for image annotation, but can also be extended to multiple fields such as video analysis, intelligent monitoring, and automatic driving, and has broad application potential.

[0078] In some embodiments of the present disclosure, the fusion module 450 is configured to: process the feature vector of the prompt image, the feature vector of the position information, and the feature vector of the prompt text through a proxy attention module to obtain an initial feature vector of the annotated object; and process the initial feature vector of the annotated object through a feedforward network to obtain a prompt vector of the annotated object.

[0079] In some embodiments of the present disclosure, the labeling module 460 is configured to: perform cross-attention fusion processing on the feature vector of the image to be identified and the prompt vector of the labeled object to obtain an initial mask vector of the image to be identified; and label the image to be identified according to the initial mask vector of the image to be identified to obtain labeling information of at least one labeled object among the objects to be identified.

[0080] In some embodiments of the present disclosure, the image to be identified is annotated according to its initial mask vector to obtain annotation information of at least one annotated object among the objects to be identified, including: performing transposed convolution processing on the initial mask vector of the image to be identified to obtain the transposed convolution vector of the image to be identified; performing cross-attention processing on the initial mask vector of the image to be identified to obtain the cross-attention vector of the image to be identified; and annotating the image to be identified according to the transposed convolution vector of the image to be identified and the cross-attention vector of the image to be identified to obtain annotation information of at least one annotated object among the objects to be identified.

[0081] In some embodiments of the present disclosure, the image to be identified is annotated according to the transposed convolution vector of the image to be identified and the cross-attention vector of the image to be identified to obtain the annotation information of at least one annotated object among the objects to be identified, including: processing the cross-attention vector of the image to be identified by a multi-layer perceptual machine to obtain the multi-layer perceptual vector of the image to be identified; performing dot product calculation on the transposed convolution vector of the image to be identified and the multi-layer perceptual vector of the image to be identified to obtain the target mask vector of the image to be identified; and converting the target mask vector of the image to be identified to obtain the annotation information of at least one annotated object among the objects to be identified.

[0082] In some embodiments of the present disclosure, the first feature extraction module 420 is configured to: encode the image to be identified through an image encoder to obtain a feature vector of the image to be identified; encode the prompt image through the image encoder to obtain a feature vector of the prompt image.

[0083] In some embodiments of the present disclosure, the second feature extraction module 430 is configured to encode the position information of the marked object in the prompt object through a position encoder to obtain a feature vector of the position information.

[0084] In some embodiments of the present disclosure, the third feature extraction module 440 is configured to encode the prompt text through a text encoder to obtain a feature vector of the prompt text.

[0085] Figure 5 Schematic diagram of an electronic device 5 provided in an embodiment of the present disclosure. Figure 5 As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 501 executes the computer program 503, the functions of the modules in the above-mentioned device embodiments are implemented.

[0086] The electronic device 5 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 5 may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will appreciate that Figure 5 The electronic device 5 is merely an example and does not limit the electronic device 5 , and may include more or less components than those shown in the figure, or different components.

[0087] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0088] The memory 502 may be an internal storage unit of the electronic device 5, for example, a hard disk or memory of the electronic device 5. The memory 502 may also be an external storage device of the electronic device 5, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. The memory 502 may also include both an internal storage unit of the electronic device 5 and an external storage device. The memory 502 is used to store computer programs and other programs and data required by the electronic device.

[0089] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0090] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0091] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. An image annotation method, characterized in that: include: Acquire an image to be recognized, a prompt image, location information of annotated objects in the prompt object, and a prompt text; Performing feature extraction processing on the image to be identified to obtain a feature vector of the image to be identified, and performing feature extraction processing on the prompt image to obtain a feature vector of the prompt image; Performing feature extraction processing on the position information of the marked object in the prompt object to obtain a feature vector of the position information; Performing feature extraction processing on the prompt text to obtain a feature vector of the prompt text; fusing the feature vector of the prompt image, the feature vector of the position information, and the feature vector of the prompt text to obtain a prompt vector of the marked object; The image to be identified is annotated according to the feature vector of the image to be identified and the prompt vector of the annotated object, so as to obtain the annotation information of at least one annotated object among the objects to be identified.

2. The method according to claim 1, characterized in that: The feature vector of the prompt image, the feature vector of the position information, and the feature vector of the prompt text are fused to obtain the prompt vector of the marked object, including: Processing the feature vector of the prompt image, the feature vector of the position information, and the feature vector of the prompt text through a proxy attention module to obtain an initial feature vector of the annotated object; The initial feature vector of the labeled object is processed through a feedforward network to obtain a prompt vector of the labeled object.

3. The method according to claim 1, characterized in that The image to be identified is annotated according to the feature vector of the image to be identified and the prompt vector of the annotated object, and the annotated information of at least one annotated object in the object to be identified is obtained, including: Performing cross-attention fusion processing on the feature vector of the image to be identified and the prompt vector of the labeled object to obtain an initial mask vector of the image to be identified; The image to be identified is annotated according to the initial mask vector of the image to be identified, so as to obtain annotation information of at least one annotated object among the objects to be identified.

4. The method according to claim 3, characterized in that The image to be identified is annotated according to the initial mask vector of the image to be identified to obtain annotation information of at least one annotated object in the object to be identified, including: Performing a transposed convolution process on the initial mask vector of the image to be identified to obtain a transposed convolution vector of the image to be identified; Performing cross-attention processing on the initial mask vector of the image to be identified to obtain a cross-attention vector of the image to be identified; The image to be identified is annotated according to the transposed convolution vector of the image to be identified and the cross attention vector of the image to be identified, so as to obtain the annotation information of at least one annotated object among the objects to be identified.

5. The method according to claim 4, characterized in that The image to be identified is annotated according to the transposed convolution vector of the image to be identified and the cross attention vector of the image to be identified, and the annotation information of at least one annotated object in the object to be identified is obtained, including: Processing the cross attention vector of the image to be identified by a multi-layer perceptron to obtain a multi-layer perceptual vector of the image to be identified; Performing dot product calculation on the transposed convolution vector of the image to be identified and the multi-layer perception vector of the image to be identified to obtain a target mask vector of the image to be identified; The target mask vector of the image to be identified is converted to obtain labeling information of at least one labeled object in the object to be identified.

6. The method according to claim 1, characterized in that Performing feature extraction processing on the image to be identified to obtain a feature vector of the image to be identified includes encoding the image to be identified by an image encoder to obtain the feature vector of the image to be identified; Performing feature extraction processing on the prompt image to obtain the feature vector of the prompt image includes performing encoding processing on the prompt image through the image encoder to obtain the feature vector of the prompt image.

7. The method according to claim 1, characterized in that Performing feature extraction processing on the position information of the marked object in the prompt object to obtain the feature vector of the position information includes encoding the position information of the marked object in the prompt object through a position encoder to obtain the feature vector of the position information; Performing feature extraction processing on the prompt text to obtain a feature vector of the prompt text includes encoding the prompt text through a text encoder to obtain the feature vector of the prompt text.

8. An image annotation processing device, characterized in that: include: An acquisition module, used for acquiring an image to be recognized, a prompt image, location information of annotated objects in the prompt object, and a prompt text; A first feature extraction module is used to perform feature extraction processing on the image to be identified to obtain a feature vector of the image to be identified, and to perform feature extraction processing on the prompt image to obtain a feature vector of the prompt image; A second feature extraction module is used to perform feature extraction processing on the position information of the marked object in the prompt object to obtain a feature vector of the position information; A third feature extraction module is used to perform feature extraction processing on the prompt text to obtain a feature vector of the prompt text; A fusion module, used for fusing the feature vector of the prompt image, the feature vector of the position information, and the feature vector of the prompt text to obtain a prompt vector of the marked object; The labeling module is used to label the image to be identified according to the feature vector of the image to be identified and the prompt vector of the labeled object to obtain labeling information of at least one labeled object among the objects to be identified.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images