Multi-label identification cross-modal representation system and method based on semantic guidance

Abstract

The invention discloses a multi-label identification cross-modal representation system and method based on semantic guidance, and belongs to the field of artificial intelligence. The system is composed of a feature encoder, a graph attention network, a visual language feature reconstruction module, a visual-semantic feature matching module and a multi-label prediction module. The method comprises the following steps: S1, collecting image and text data; s2, a text feature encoder and an image feature encoder extract image features and text features; s3, performing semantic association enhancement among multiple tags on the text features; s4, fusing the image features and the text features; s5, fusing the fusion features and the improved text features; s6, a vision-semantic feature matching module aligns and reconstructs the fusion features and the text features, and cosine similarity is calculated; and S7, classifying the matching cosine similarity by using a multi-label classifier to obtain a multi-classification label corresponding to the image. According to the method, multi-label identification can be effectively carried out in an open scene, and the prediction accuracy is improved.

Description

Technical Field

[0001] This invention relates to a semantically guided multi-label recognition cross-modal representation system and method, belonging to the field of artificial intelligence, and particularly to a semantically guided multi-label recognition cross-modal representation. Background Technology

[0002] Traditional image recognition tasks typically employ single-label classification methods, where each image is assigned only one category label. However, in real-world scenarios, an object may be associated with multiple labels, which is the basis of multi-label classification (MLC). MLC aims to accurately identify all relevant categories within an image, with the challenge lying in effectively handling the relationships between multiple labels and reducing prediction ambiguity. Compared to single-label classification, MLC is better suited for tasks such as fine-grained image classification, autonomous driving, and scene understanding. The key to solving the MLC problem lies in fully leveraging the powerful learning capabilities of deep learning to better capture the dependencies between labels.

[0003] Multi-label classification is suitable for complex scenarios due to the complexity of input images and output label spaces. However, in real life, it is impossible to exhaustively list all possible categories. Therefore, enabling models to identify unknown categories is becoming increasingly crucial. Existing methods mostly rely on pre-trained text label embeddings (such as GloVe), but they only utilize knowledge from a single modality and fail to fully integrate visual semantics and image-text pair information. Therefore, there is an urgent need for a feature representation method that can effectively integrate multi-modal information, enabling the mapping of image and language representations to a shared feature space through pre-training on large-scale image-text pairs. This would achieve the alignment and fusion of cross-modal information, improving the accuracy of multi-label and unknown label prediction. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a semantically guided multi-label recognition cross-modal representation system and method for the recognition and prediction of multiple semantic labels in image and text data. This method aims to reconstruct and match cross-modal features of visual language in image and text data, thereby effectively performing multi-label image recognition in open scenes and improving prediction accuracy.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A semantically guided multi-label recognition cross-modal representation system, combined with Figure 1The system is characterized by comprising a feature encoder, a text feature improvement module, a visual-language feature reconstruction module, a visual-semantic feature matching module, and a multi-label prediction module. The feature encoder includes a text feature encoder and an image feature encoder. The text feature encoder uses a pre-trained context optimization model to extract features from text data. The image feature encoder uses a pre-trained visual Transformer model to extract features from image data. The text feature encoder is connected to the text feature improvement module, the visual-language feature reconstruction module, and the visual-semantic feature matching module, respectively used to generate new text features, achieve feature fusion of visual and linguistic features, and match text features with image features. The image feature encoder is connected to the visual-language feature reconstruction module to achieve feature fusion of visual and linguistic features. The text feature improvement module is based on GAT (Graph Attention). The system is implemented using a graph attention network (GAN) for enhancing semantic associations between multiple labels. The visual language feature reconstruction module includes a semantic decoupling attention module and a visual feature reconstruction module. The semantic decoupling attention module is an attention mechanism connected to the image feature encoder and the text feature encoder for fusing text and image features. The visual feature reconstruction module is implemented based on a deep learning network and connected to the image feature encoder and the text feature enhancement module for generating new text-image features. The visual language feature reconstruction module is connected to the visual-semantic feature matching module for matching text and image features. The multi-label prediction module is a multi-label classifier connected to the visual-semantic feature matching module for multi-label prediction.

[0007] Preferably, the text feature encoder uses a pre-trained CoOp (Context Optimization) model to extract text features; the image feature encoder uses a pre-trained visual Transformer model to extract image features.

[0008] Preferably, the visual-semantic feature matching module is implemented based on Top-K average pooling.

[0009] Preferably, the multi-label prediction module is implemented based on the L1 loss function.

[0010] A semantically guided multi-label recognition cross-modal representation method, combined with Figure 2 It includes the following steps:

[0011] S1: Acquire image and text data;

[0012] S2: The feature encoder extracts features from the image data and text data respectively through the text feature encoder and the image feature encoder to obtain image features. and text features ;in Indicates the first Zhang's image and its corresponding text data;

[0013] S3: Text Feature Improvement Module for Text Features Enhance the semantic associations between multiple labels to obtain improved text features. ;

[0014] S4: Semantic decoupling attention module for image features and text features By performing fusion, fusion characteristics are obtained. ;

[0015] S5: Visual feature reconstruction module for fused features and improve text features By performing fusion, the reconstructed fusion characteristics are obtained. ;

[0016] S6: Visual-Semantic Feature Matching Module Aligns, Reconstructs, and Fuses Features Text features Cosine similarity calculation is performed to match features between "image-text" pairs;

[0017] S7: The multi-label prediction module uses a multi-label classifier to classify the feature matching of "image-text" pairs and obtain the multi-class labels corresponding to the images.

[0018] Furthermore, the text feature encoder described in step S2 performs text feature processing. Extraction, specifically: for the first The image in the first Text feature vectors in each category , This is a matrix of text features for the image. The total number of categories, , Optimize the model for context. For text features; the image feature encoder performs image feature processing. Extraction, specifically: for the first Zhang's image was first divided using a sliding window. Then, the visual Transformer model is used to extract any nth image patch. Features of an image patch , This is a matrix of image features for "image-text pairs". The CoOp (Context Optimization) model transforms contextual words in the prompt into learnable vectors while keeping the parameters of the pre-trained model unchanged, thereby improving model performance with a small amount of labeled data.

[0019] Furthermore, step S3 specifically includes:

[0020] S301: The text feature improvement module constructs a fully connected graph for text features; specifically, categories are represented by graph nodes, the first... Nodes of each category With the Nodes of each category The edges between them are ,in, The attention vector to be trained. The weight matrix to be trained. This is a LeakyRectified Linear Unit (LER) activation function.

[0021] S302: The text feature improvement module generates node weights using a fully connected graph. ;in, , ,in Represents a node The set of neighboring nodes, ;

[0022] S303: Text Feature Improvement Module Generates New Text Features ,in, , It is a non-linear activation function. This is the weight matrix to be trained.

[0023] Furthermore, step S4 specifically includes:

[0024] S401: Image features Perform an expansion operation to obtain expanded image features. At the same time, text features Perform an expansion operation to obtain the expanded text features. Specifically, image features along Dimensional replication Next, make its dimension from Expand to of Text features along Dimensional replication Next, make its dimension from Expand to of ;

[0025] S402: The semantic decoupling attention module utilizes a fully connected layer to process the expanded image features. and extended text features Perform fusion to output a fully connected feature matrix of categories. , For fully connected layer operations, For element-wise multiplication, It is the hyperbolic tangent function;

[0026] S403: The semantic decoupling attention module uses an attention mechanism to obtain the attention coefficient matrix. ;in, , It is a normalized exponential function;

[0027] S404: The semantic decoupling attention module will extend the image features. Preliminary fused features are obtained by fusing features with attention scores. ;in, ;

[0028] S405: Semantic decoupling attention module based on preliminary fusion features Transpose matrix and text features Calculate cosine similarity; where, for image patches and text categories The cosine similarity is ;in, ;

[0029] S406: The semantic decoupling attention module uses normalization operations to obtain fused features. ;in, .

[0030] Furthermore, step S5 specifically includes:

[0031] S501: The fusion features obtained in step S4 Text features obtained by the text feature improvement module along By concatenating the dimensions, we obtain the dimension as follows: The fusion matrix is ​​obtained by first calculating the fusion matrix, and then using a fully connected operation to reduce the dimensionality of the fusion matrix, resulting in the dimensionality-reduced fusion matrix. ;

[0032] S502: For the fusion matrix Perform an expansion operation to obtain the expanded fusion matrix. Specifically, the fusion matrix along Dimensional replication Next, make its dimension from Expand to ;

[0033] S503: Expand the fusion matrix and extended image features Perform element-wise multiplication to obtain the similarity coefficient matrix. ;in, ;

[0034] S504: Obtaining Attention Coefficients Using Attention Mechanisms ;in, ;

[0035] S505: Feature fusion using image features and attention scores .

[0036] An electronic device includes at least one processor; and a memory communicatively connected to said at least one processor; wherein,

[0037] The memory stores a computer program that is executed by the at least one processor, which enables the at least one processor to execute the semantically guided multi-label recognition cross-modal representation method described above.

[0038] Finally, the present invention also discloses a computer-readable storage medium storing computer instructions for causing a processor to execute the aforementioned semantically guided multi-label recognition cross-modal representation method.

[0039] The beneficial effects of this invention are as follows: It provides a semantically guided multi-label recognition cross-modal representation system and method, which utilizes the semantic relationships between multiple labels by combining a trainable graph neural network in the text representation process through a text feature improvement module, and then integrates the semantic information into the visual image in the reconstruction process using a visual language feature reconstruction module; finally, the image and language representation are mapped to a shared feature space to achieve the alignment and fusion of cross-modal information, thereby improving the accuracy of multi-label prediction and recognition. Attached Figure Description

[0040] To make the objectives and technical solutions of this invention clearer, the following figures are provided for illustration:

[0041] Figure 1 This is a flowchart of the semantically guided multi-label recognition cross-modal representation method in this invention;

[0042] Figure 2 This is an architecture diagram of the semantically guided multi-label recognition cross-modal representation system of the present invention; where the arrows indicate the direction of data transmission.

[0043] Figure 3 This is a data processing flowchart of the semantically guided multi-label recognition cross-modal representation method in Embodiment 1 of the present invention;

[0044] Figure 4 This is a schematic diagram of the electronic device in Embodiment 2 of the present invention. Detailed Implementation

[0045] To make the objectives and technical solutions of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0046] Example 1: The acquired images have already been manually annotated with text descriptions. Now, it is necessary to utilize these images and text to perform multi-dimensional analysis and annotation of the images. This invention provides a "semantic-guided multi-label recognition cross-modal representation system and method." Details are as follows:

[0047] Combination Figure 1A semantically guided multi-label recognition cross-modal representation system is characterized by comprising a feature encoder (1), a text feature improvement module (2), a visual-language feature reconstruction module (3), a visual-semantic feature matching module (4), and a multi-label prediction module (5). The feature encoder (1) includes a text feature encoder (11) and an image feature encoder (12). The text feature encoder (11) uses a pre-trained context optimization model to extract features from text data. The image feature encoder (12) uses a pre-trained visual Transformer model to extract features from image data. The text feature encoder (11) is connected to the text feature improvement module (2), the visual-language feature reconstruction module (3), and the visual-semantic feature matching module (4), respectively used to generate new text features, realize feature fusion of visual and linguistic features, and match text features with image features. The image feature encoder (12) is connected to the visual-language feature reconstruction module (4). 3) Connected to achieve feature fusion of visual and linguistic features; the text feature improvement module (2) is based on the GAT graph attention network and is used to enhance the semantic association between multiple labels; the visual-linguistic feature reconstruction module (3) includes a semantic decoupling attention module (31) and a visual feature reconstruction module (32); the semantic decoupling attention module (31) is an attention mechanism, connected to the image feature encoder (12) and the text feature encoder (11), and is used for the fusion of text features and image features; the visual feature reconstruction module (32) is based on a deep learning network, connected to the image feature encoder (12) and the text feature improvement module (2), and is used to generate new text-image features; the visual-linguistic feature reconstruction module (32) is connected to the visual-semantic feature matching module (4), and is used to achieve the matching of text features and image features; the multi-label prediction module (5) is a multi-label classifier, connected to the visual-semantic feature matching module (4), and is used for multi-label prediction.

[0048] The text feature encoder (11) uses a pre-trained CoOp context optimization model to extract text features; the image feature encoder (12) uses a pre-trained visual Transformer model to extract image features.

[0049] The visual-semantic feature matching module (4) is implemented based on Top-K average pooling.

[0050] The multi-label prediction module (5) is implemented based on the L1 loss function.

[0051] Combination Figure 2 A semantically guided multi-label recognition cross-modal representation method includes the following steps:

[0052] S1: Collect image and text data.

[0053] S2: The feature encoder (1) extracts features from the image data and text data respectively through the text feature encoder (11) and the image feature encoder (12) to obtain image features. and text features ;in Indicates the first A single image and its corresponding text data.

[0054] Combination Figure 3 The text feature encoder (11) performs text feature processing. Extraction, specifically: for the first The image in the first Text feature vectors in each category , This is a matrix of text features for the image. The total number of categories, , Optimize the model for context. For text feature dimensions; the image feature encoder (12) performs image feature processing. Extraction, specifically: for the first Zhang's image was first divided using a sliding window. Then, the visual Transformer model is used to extract any nth image patch. Features of an image patch , This is a matrix of image features for "image-text pairs". The CoOp context optimization model converts context words in the prompt into learnable vectors while keeping the parameters of the pre-trained model unchanged, thereby improving model performance with a small amount of labeled data.

[0055] S3: Text Feature Improvement Module (2) for text features Enhance the semantic associations between multiple labels to obtain improved text features. Specifically:

[0056] S301: Text Feature Improvement Module (2) constructs a fully connected graph for text features; specifically, the categories are represented by graph nodes, the first... Nodes of each category With the Nodes of each category The edges between them are ,in, The attention vector to be trained. The weight matrix to be trained. It is a rectified activation function with leakage;

[0057] S302: Text Feature Improvement Module (2) Generates Node Weights Using Fully Connected Graphs ;in, , ,in Represents a node The set of neighboring nodes, ;

[0058] S303: Text Feature Improvement Module (2) generates new text features ,in, , It is a non-linear activation function. This is the weight matrix to be trained.

[0059] S4: Semantic decoupling attention module (31) for image features and text features By performing fusion, fusion characteristics are obtained. Specifically:

[0060] S401: Image features Perform an expansion operation to obtain expanded image features. At the same time, text features Perform an expansion operation to obtain the expanded text features. Specifically, image features along Dimensional replication Next, make its dimension from Expand to of Text features along Dimensional replication Next, make its dimension from Expand to of ;

[0061] S402: Semantic decoupling attention module (31) utilizes fully connected layers to process expanded image features and extended text features Perform fusion to output a fully connected feature matrix of categories. , For fully connected layer operations, For element-wise multiplication, It is the hyperbolic tangent function;

[0062] S403: Semantic Decoupling Attention Module (31) obtains the attention coefficient matrix using the attention mechanism. ;in, , It is a normalized exponential function;

[0063] S404: The semantic decoupling attention module (31) will extend the image features Preliminary fused features are obtained by fusing features with attention scores. ;in, ;

[0064] S405: Semantic decoupling attention module (31) based on preliminary fusion features Transpose matrix and text features Calculate cosine similarity; where, for image patches and text categories The cosine similarity is ;in, ;

[0065] S406: Semantic decoupling attention module (31) uses normalization operation to process and obtain fused features. ;in, .

[0066] S5: Visual feature reconstruction module (32) for fused features and improve text features By performing fusion, the reconstructed fusion characteristics are obtained. Specifically:

[0067] S501: The fusion features obtained in step S4 Text features obtained by the text feature improvement module along By concatenating the dimensions, we obtain the dimension as follows: The fusion matrix is ​​obtained by first calculating the fusion matrix, and then using a fully connected operation to reduce the dimensionality of the fusion matrix, resulting in the dimensionality-reduced fusion matrix. ;

[0068] S502: For the fusion matrix Perform an expansion operation to obtain the expanded fusion matrix. Specifically, the fusion matrix along Dimensional replication Next, make its dimension from Expand to ;

[0069] S503: Expand the fusion matrix and extended image features Perform element-wise multiplication to obtain the similarity coefficient matrix. ;in, ;

[0070] S504: Obtaining Attention Coefficients Using Attention Mechanisms ;in, ;

[0071] S505: Feature fusion using image features and attention scores .

[0072] S6: Visual-Semantic Feature Matching Module (4) Alignment, Reconstruction, and Fusion Features Text features Cosine similarity calculation is performed to match "image-text" pairs. .

[0073]

[0074] S7: Multi-label prediction module (5) uses random forest to build a multi-label classifier to classify the matching of "image-text" pairs and obtain the multi-class labels corresponding to the images.

[0075] To better demonstrate the effectiveness of the method of this invention in predicting known and unknown labels in multi-label scenarios, this embodiment conducted comparative experiments on a computing platform equipped with one NVIDIA GTX 4070 graphics card, using the PyTorch deep learning framework to implement the model. The experiments utilized multiple publicly available datasets, including:

[0076] The NUS-WIDE dataset (https: / / lms.comp.nus.edu.sg / wp-content / uploads / 2019 / research / nuswide / NUS-WIDE.html) is used for research on multi-label image classification and cross-modal retrieval.

[0077] The COCO dataset (http: / / images.cocodataset.org / annotations / annotations_trainval2014.zip --output coco_annotations.zip) is one of the most mainstream and authoritative benchmark datasets in the fields of object detection, instance segmentation, image captioning, and human keypoint detection.

[0078] The VOC dataset (http: / / host.robots.ox.ac.uk / pascal / VOC / voc2012 / VOCtrainval_11-May-2012.tar --output pascal_raw.tar) is the core evaluation dataset for object detection, image classification, and semantic segmentation tasks.

[0079] In the experiments, the method of this invention was compared with mainstream methods for existing open vocabulary (OV) tasks. The comparison metrics included average accuracy (mAP), F1 score, precision (P), and recall (R), and the results are shown in Table 1. Here, GZSL represents the prediction results for known and unknown labels, and ZSL represents the prediction results for unknown labels. As can be seen from the table, this invention outperforms existing mainstream methods in almost all metrics. Top-3 and Top-5 represent the sets of labels with the highest and highest probabilities in each category, respectively.

[0080] Under different loss functions IUN(2021), EM_APL(2022), and EM(2022), our proposed method is compared with mainstream existing Single Positive Multi-label Learning (SPML) methods. The comparison metrics include average accuracy (mAP), F1 score, precision (P), and recall (R). The results are shown in Tables 2 and 3. As can be seen from the tables, our proposed method outperforms mainstream methods in almost all metrics.

[0081] The references for the mainstream methods are as follows:

[0082] [1].Gupta A, Narayan S, Khan S, et al. Generative multi-label zero-shot learning[J]. IEEE Transactions on Pattern Analysis and MachineIntelligence, 2023, 45(12): 14611-14624. + Gen-MLZSL(2023).

[0083] [2].Ma Z, Ma R, Wang G, et al. Label correlation preserving visual-semantic joint embedding for multi-label zero-shot learning[J]. MultimediaTools and Applications, 2025, 84(20): 22251-22268. + SCP(2025).

[0084] [3].He S, Guo T, Dai T, et al. Open-vocabulary multi-labelclassification via multi-modal knowledge transfer[C] / / Proceedings of the AAAIconference on artificial intelligence. 2023, 37(1): 808-816. + MKT(2023).

[0085] [4].Cole E, Mac Aodha O, Lorieul T, et al. Multi-label learning from single positive labels[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition. 2021: 933-942. + IUN(2021).

[0086] [5].Zhou D, Chen P, Wang Q, et al. Acknowledging the unknown for multi-label learning with single positive labels[C] / / European Conference onComputer Vision. Cham: Springer Nature Switzerland, 2022: 423-440. + EM(2022), EM_APL(2022).

[0087] Table 1. Comparative experimental results on the NUS-WIDE dataset.

[0088]

[0089] Table 2. Comparative experimental results on the COCO dataset.

[0090]

[0091] Table 3. Comparative experimental results on the VOC dataset.

[0092]

[0093] Example 2: For the scenario in Example 1, Figure 4 A schematic diagram of an electronic device (90) that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.

[0094] Electronic devices can also refer to various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0095] like Figure 4 As shown, the electronic device (90) includes at least one processor (91) and a memory, such as a read-only memory (ROM) (92) or a random access memory (RAM) (93), which is communicatively connected to the at least one processor (91). The memory stores computer programs executable by the at least one processor. The processor (91) can perform various appropriate actions and processes based on the computer programs stored in the ROM (92) or loaded from storage units (98) into the RAM (93). The RAM (43) may also store various programs and data required for the operation of the electronic device (90). The processor (91), ROM (42), and RAM (43) are interconnected via a bus (94). An input / output (I / O) interface (95) is also connected to the bus (94).

[0096] Multiple components in the electronic device (90) are connected to the I / O interface (95), including: input units (96), such as keyboards, mice, etc.; output units (97), such as various types of displays, speakers, etc.; storage units (98), such as disks, optical disks, etc.; and communication units (99), such as network cards, modems, wireless transceivers, etc. The communication unit (99) allows the electronic device (90) to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0097] The processor (91) can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processors (91) include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor (91) performs the various methods and processes described above, such as semantically guided multi-label recognition cross-modal representation methods.

[0098] In some embodiments, the semantically guided multi-label recognition cross-modal representation method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as a storage unit (98). In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device (90) via a ROM (92) and / or a communication unit (99). When the computer program is loaded into RAM (93) and executed by a processor (91), one or more steps of the semantically guided multi-label recognition cross-modal representation method described above can be performed. Alternatively, in other embodiments, the processor (91) can be configured to perform the semantically guided multi-label recognition cross-modal representation method by any other suitable means (e.g., by means of firmware).

[0099] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0100] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0101] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0102] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0103] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0104] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0105] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.

Claims

1. A semantically guided multi-label recognition cross-modal representation system, characterized in that, It consists of a feature encoder (1), a text feature improvement module (2), a visual-language feature reconstruction module (3), a visual-semantic feature matching module (4), and a multi-label prediction module (5); the feature encoder (1) includes a text feature encoder (11) and an image feature encoder (12); the text feature encoder (11) uses a pre-trained context optimization model to extract features from text data; the image feature encoder (12) uses a pre-trained visual Transformer model to extract features from image data; the text feature encoder (11) is connected to the text feature improvement module (2), the visual-language feature reconstruction module (3), and the visual-semantic feature matching module (4), and is used to generate new text features, realize feature fusion of visual and linguistic features, and match text features with image features, respectively; the image feature encoder (12) is connected to the visual-language feature reconstruction module (3), and is used to realize feature fusion of visual and linguistic features; the text feature improvement module (2) is based on GAT (Graph Attention). The network is implemented using a graph attention network for enhancing semantic association between multiple labels; the visual language feature reconstruction module (3) includes a semantic decoupling attention module (31) and a visual feature reconstruction module (32); the semantic decoupling attention module (31) is an attention mechanism connected to the image feature encoder (12) and the text feature encoder (11) for fusing text features and image features; the visual feature reconstruction module (32) is implemented based on a deep learning network and connected to the image feature encoder (12) and the text feature improvement module (2) for generating new text-image features; the visual language feature reconstruction module (32) is connected to the visual-semantic feature matching module (4) for matching text features and image features; the multi-label prediction module (5) is a multi-label classifier connected to the visual-semantic feature matching module (4) for multi-label prediction.

2. The semantically guided multi-label recognition cross-modal representation system according to claim 1, characterized in that, The text feature encoder (11) uses a pre-trained CoOp (Context Optimization) model to extract text features; the image feature encoder (12) uses a pre-trained visual Transformer model to extract image features.

3. The semantically guided multi-label recognition cross-modal representation system according to claim 1, characterized in that, The visual-semantic feature matching module (4) is implemented based on Top-K average pooling.

4. The semantically guided multi-label recognition cross-modal representation system according to claim 1, characterized in that, The multi-label prediction module (5) is implemented based on the L1 loss function.

5. A semantically guided multi-label recognition cross-modal representation method, characterized in that, Includes the following steps: S1: Acquire image and text data; S2: The feature encoder (1) extracts features from the image data and text data respectively through the text feature encoder (11) and the image feature encoder (12) to obtain image features. and text features ;in Indicates the first Zhang's image and its corresponding text data; S3: Text Feature Improvement Module (2) for text features Enhance the semantic associations between multiple labels to obtain improved text features. ; S4: Semantic decoupling attention module (31) for image features and text features By performing fusion, fusion characteristics are obtained. ; S5: Visual feature reconstruction module (32) for fused features and improve text features By performing fusion, the reconstructed fusion characteristics are obtained. ; S6: Visual-Semantic Feature Matching Module (4) Aligns, Reconstructs, and Fuses Features Text features Cosine similarity calculation is performed to match "image-text" pairs; S7: Multi-label prediction module (5) uses a multi-label classifier to classify the matching of "image-text" pairs and obtain the multi-class label corresponding to the image.

6. The semantically guided multi-label recognition cross-modal representation method according to claim 5, characterized in that, The text feature encoder (11) described in step S2 performs text feature processing. Extraction, specifically: for the first The image in the first Text feature vectors in each category , This is a matrix of text features for the image. The total number of categories, , Optimize the model for context. For text feature dimensions; the image feature encoder (12) performs image feature processing. Extraction, specifically: for the first Zhang's image was first divided using a sliding window. Then, the visual Transformer model is used to extract any nth image patch. Features of an image patch , A matrix of image features for "image-text pairs".

7. The semantically guided multi-label recognition cross-modal representation method according to claim 5, characterized in that, Step S3 specifically involves: S301: Text Feature Improvement Module (2) constructs a fully connected graph for text features; specifically, the categories are represented by graph nodes, the first... Nodes of each category With the Nodes of each category The edges between them are ,in, The attention vector to be trained. The weight matrix to be trained. This is a LeakyRectified Linear Unit (LER) activation function. S302: Text Feature Improvement Module (2) Generates Node Weights Using Fully Connected Graphs ;in, , ,in Represents a node The set of neighboring nodes, ; S303: Text Feature Improvement Module (2) generates new text features ,in, , It is a non-linear activation function. This is the weight matrix to be trained.

8. The semantically guided multi-label recognition cross-modal representation method according to claim 5, characterized in that, Step S4 specifically involves: S401: Image features Perform an expansion operation to obtain expanded image features. At the same time, text features Perform an expansion operation to obtain the expanded text features. Specifically, image features along Dimensional replication Next, make its dimension from Expand to of Text features along Dimensional replication Next, make its dimension from Expand to of ; S402: Semantic decoupling attention module (31) utilizes fully connected layers to process expanded image features and extended text features Perform fusion to output a fully connected feature matrix of categories. , For fully connected layer operations, For element-wise multiplication, It is the hyperbolic tangent function; S403: Semantic Decoupling Attention Module (31) obtains the attention coefficient matrix using the attention mechanism. ;in, , It is a normalized exponential function; S404: The semantic decoupling attention module (31) will extend the image features Preliminary fused features are obtained by fusing features with attention scores. ;in, ; S405: Semantic decoupling attention module (31) based on preliminary fusion features Transpose matrix and text features Calculate cosine similarity; where, for image patches and text categories The cosine similarity is ;in, ; S406: Semantic decoupling attention module (31) uses normalization operation to process and obtain fused features. ;in, .

9. The semantically guided multi-label recognition cross-modal representation method according to claim 5, characterized in that, Step S5 specifically involves: S501: The fusion features obtained in step S4 Text features obtained by the text feature improvement module along By concatenating the dimensions, we obtain the dimension as follows: The fusion matrix is ​​obtained by first calculating the fusion matrix, and then using a fully connected operation to reduce the dimensionality of the fusion matrix, resulting in the dimensionality-reduced fusion matrix. ; S502: For the fusion matrix Perform an expansion operation to obtain the expanded fusion matrix. Specifically, the fusion matrix along Dimensional replication Next, make its dimension from Expand to ; S503: Expand the fusion matrix and extended image features Perform element-wise multiplication to obtain the similarity coefficient matrix. ;in, ; S504: Obtaining Attention Coefficients Using Attention Mechanisms ;in, ; S505: Feature fusion using image features and attention scores .

10. An electronic device and a computer-readable storage medium, characterized in that, The electronic device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the semantically guided multi-label recognition cross-modal representation method described above; the computer-readable storage medium stores computer instructions for causing the processor to implement the semantically guided multi-label recognition cross-modal representation method described above when executed.

Images