Multimodal image text multi-label classification method, system, device and storage medium

Abstract

The present invention provides a multimodal image text multi-label classification method, system, device and storage medium, the method comprising: constructing a multimodal multi-label binary classification model, performing weight merging reasoning optimization; performing data annotation based on a note database to generate a high-quality data set; inputting the high-quality data set into a picture-text base model in the multimodal multi-label binary classification model to extract features, obtain text information enhancement features and image information enhancement features; obtain enhanced image text features based on the text information enhancement features and image information enhancement features; and perform multi-label classification on images based on the enhanced image text features. The present invention can better understand the relationship between images and texts, thereby improving the model's ability to jointly represent picture and text information, enhancing the performance of the model in multimodal tasks, and improving the overall effectiveness of multi-label classification.

Description

Background Art

[0002] In the development of artificial intelligence, multimodal data processing plays a key role. The modal forms of data in the real world are complex and diverse, such as images, texts, etc. They are intertwined and contain rich semantics, which are the key elements of information understanding.

[0003] As a key task in multimodal data processing, multi-label classification is of great importance. Different from traditional single-label classification, it needs to deal with the complex situation of multiple labels for one sample. In the fields of medical imaging diagnosis, social media content analysis, industrial product quality inspection, etc., its results have a direct and significant impact on the quality of decision-making and the practical level of the system, and are related to the accuracy and effectiveness of the application.

[0004] The multimodal image-text model exhibits unique properties in the field of multi-label classification. It can cleverly integrate multimodal information, accurately and deeply mine the semantic relationships therein, and lay a solid foundation for multi-label classification work, thereby effectively coping with the challenges of complex labels and diverse data in multi-label classification, and improving the accuracy and efficiency of classification.

[0005] In terms of engineering practice, the multimodal image-text model has significant advantages. It is easy to transform into an engineering project and can be easily integrated with other system components. Its high scalability makes it easy to cope with new demands and data changes, whether it is adding label categories, expanding data scale or adjusting modal distribution. It can easily cope with it, greatly reducing the cost of system maintenance and upgrades, providing a cost-effective solution for engineering applications, and has extremely high practical value.

[0006] Therefore, the present invention provides a multi-modal image-text multi-label classification method, system, device and storage medium. Summary of the invention

[0007] In response to the problems in the prior art, the purpose of the present invention is to provide a multimodal image-text multi-label classification method, system, device and storage medium, which overcomes the difficulties of the prior art and can better understand the relationship between images and texts, thereby improving the model's ability to jointly represent image and text information, enhancing the model's performance in multimodal tasks, and improving the overall effectiveness of multi-label classification.

[0008] An embodiment of the present invention provides a multi-modal image text multi-label classification method, comprising the following steps:

[0009] S110, build a multi-modal multi-label binary classification model and perform weight merging reasoning optimization;

[0010] S120, perform data annotation based on the note database to generate high-quality data sets;

[0011] S130, inputting the high-quality data set into the image-text base model in the multimodal multi-label binary classification model to extract features, and obtaining text information enhancement features and image information enhancement features;

[0012] S140, obtaining enhanced image text features based on the text information enhancement features and the image information enhancement features;

[0013] S150: performing multi-label classification on the image based on the enhanced image text features.

[0014] Preferably, in step S110, model prediction is supervised based on training data and cross entropy loss, and cached model weights are updated through SGD.

[0015] Preferably, in step S110, the multimodal multi-label binary classification model is a key-value cache model.

[0016] Preferably, in the step S110, under the parallel matrix architecture formed by weight merging, the information is processed simultaneously in multiple dimensions.

[0017] Preferably, in step S110, the multimodal multi-label binary classification model includes a linear layer for transforming image features extracted by the image model, and the weight of the linear layer is set to the transpose of the linear layer.

[0018] Preferably, the step S110 further includes: in the process of training the multimodal multi-label binary classification model, for the input image data, firstly extracting features through the image base model, and then processing through linear layer fine-tuning to enhance the expression ability of the image features.

[0019] Preferably, the step S120 includes:

[0020] S121, collecting point of interest data;

[0021] S122, collecting web crawler data;

[0022] S123, generating random data;

[0023] S124, performing data cleaning and variance screening on the interest point data, web crawler data, and generated random data to obtain a high-quality data set.

[0024] An embodiment of the present invention further provides a multimodal image-text multi-label classification system, which is used to implement the above-mentioned multimodal image-text multi-label classification method. The multimodal image-text multi-label classification system includes:

[0025] Model building module, building a multi-modal multi-label binary classification model, and performing weight merging and reasoning optimization;

[0026] Data annotation module, which performs data annotation based on the note database to generate high-quality data sets;

[0027] A feature extraction module is used to input the high-quality data set into a picture-text base model in a multimodal multi-label binary classification model to extract features and obtain text information enhancement features and image information enhancement features;

[0028] An enhanced feature module, which obtains enhanced image text features based on the text information enhanced features and the image information enhanced features;

[0029] The label classification module performs multi-label classification on the image based on the enhanced image text features.

[0030] An embodiment of the present invention further provides a multi-modal image text multi-label classification device, comprising:

[0031] processor;

[0032] a memory storing executable instructions of the processor;

[0033] Wherein, the processor is configured to perform the steps of the above-mentioned multimodal image-text multi-label classification method by executing the executable instructions.

[0034] An embodiment of the present invention further provides a computer-readable storage medium for storing a program, which, when executed, implements the steps of the above-mentioned multi-modal image-text multi-label classification method.

[0035] The purpose of the present invention is to provide a multimodal image-text multi-label classification method, system, device and storage medium, which can better understand the relationship between images and texts, thereby improving the model's ability to jointly represent image and text information, enhancing the performance of the model in multimodal tasks, and improving the overall effectiveness of multi-label classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Other features, objectives and advantages of the present invention will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following accompanying drawings.

[0037] Figure 1 It is a flow chart of the multimodal image-text multi-label classification method of the present invention.

[0038] Figure 2 It is a structural schematic diagram of the multimodal image-text multi-label classification system of the present invention.

[0039] Figure 3 It is a structural schematic diagram of the multimodal image-text multi-label classification device of the present invention.

[0040] Figure 4 It is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the present application. The present application can also be implemented or applied through other different specific implementation methods, and the details in the present application can also be modified or changed in various ways according to different viewpoints and application systems without departing from the spirit of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0042] The following is a detailed description of the embodiments of the present application with reference to the accompanying drawings so that those skilled in the art can easily implement the present application. The present application can be embodied in a variety of different forms and is not limited to the embodiments described herein.

[0043] In the representations of the present application, the representations with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. mean that the specific features, structures, materials or characteristics represented in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the represented specific features, structures, materials or characteristics may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples represented in the present application and the features of the different embodiments or examples, unless they contradict each other.

[0044] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the representation of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0045] In order to clearly describe the present application, components irrelevant to the description are omitted, and the same reference symbols are given to the same or similar components throughout the specification.

[0046] Throughout the specification, when a device is said to be "connected" to another device, this includes not only the case of "direct connection" but also the case of "indirect connection" by placing other elements therebetween. In addition, when a device is said to "include" a certain component, unless otherwise stated, it does not exclude other components, but means that other components may be included.

[0047] When a device is said to be "on" another device, it may be directly on the other device, but there may also be other devices between it. In contrast, when a device is said to be "directly" on another device, there are no other devices between it.

[0048] Although the terms first, second, etc. are used to represent various elements in some examples, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first interface and the second interface, etc. are represented. Moreover, as used in this article, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising" and "including" indicate the existence of features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the existence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Only when the combination of elements, functions, steps or operations is inherently mutually exclusive in some way, will there be an exception to this definition.

[0049] The technical terms used herein are only used to refer to specific embodiments and are not intended to limit the present application. The singular form used herein also includes the plural form unless the sentence clearly indicates the contrary meaning. The meaning of "including" used in the specification is to specify specific characteristics, regions, integers, steps, operations, elements and / or components, and does not exclude the existence or addition of other characteristics, regions, integers, steps, operations, elements and / or components.

[0050] Although not defined differently, all terms, including technical and scientific terms used herein, have the same meaning as those generally understood by those skilled in the art to which this application belongs. Terms defined in commonly used dictionaries are additionally interpreted as having meanings consistent with the relevant technical literature and the contents of the current disclosure, and unless defined, shall not be overly interpreted as ideal or very formal meanings.

[0051] Multilabel classification means that the same instance can be assigned to multiple labels at the same time. Unlike multiclass classification, multilabel classification allows an instance to have multiple labels, which can coexist without mutual exclusion. In the field of multilabel classification, previous models have many shortcomings. For example, some models are complex and costly to transform, have poor performance in few-sample scenarios, poor scalability, inconvenient engineering applications, and the transformation of a single label is likely to affect other labels. The multimodal image-text model aims to overcome these problems. It has the advantages of low transformation cost, excellent few-sample performance, and strong scalability. It is easy to use in engineering, and can independently transform a single label without affecting other labels, thereby improving the overall performance of multi-label classification.

[0052] Figure 1 Flowchart of the multi-modal image text multi-label classification method of the present invention. Figure 1 As shown, the multi-modal image text multi-label classification method of the present invention includes:

[0053] S110, build a multi-modal multi-label binary classification model and perform weight merging reasoning optimization;

[0054] S120, perform data annotation based on the note database to generate high-quality data sets;

[0055] S130, inputting the high-quality data set into the image-text base model in the multimodal multi-label binary classification model to extract features, and obtaining text information enhancement features and image information enhancement features;

[0056] S140, obtaining enhanced image text features based on the text information enhancement features and the image information enhancement features;

[0057] S150, performing multi-label classification on the image based on the enhanced image text features.

[0058] In multi-label classification, traditional models either have high transformation costs and poor scalability, or have insufficient performance in the case of few samples, and are complex to apply in engineering. Transforming a single label can easily affect other labels. A new method is needed that has the training-free advantage of CLIP, while having strong performance in the case of few-sample classification, and is easy to transform and expand in engineering, and can independently process a single label without interfering with other labels.

[0059] In a preferred embodiment, in step S110, model prediction is supervised based on training data and cross entropy loss, and cached model weights are updated through SGD, but not limited thereto.

[0060] In a preferred embodiment, in step S110, the multimodal multi-label binary classification model is a key-value cache model, but is not limited thereto.

[0061] In a preferred embodiment, in step S110, under the parallel matrix architecture formed by weight merging, information is processed simultaneously in multiple dimensions, but not limited thereto.

[0062] In a preferred embodiment, in step S110, the multimodal multi-label binary classification model includes a linear layer that transforms the image features extracted by the image model, and the weight of the linear layer is set to the transpose of the linear layer, but is not limited thereto.

[0063] In a preferred embodiment, step S110 also includes: in the process of training the multimodal multi-label binary classification model, for the input image data, firstly extracting features through the image base model, and then processing through linear layer fine-tuning to enhance the expression ability of the image features, but not limited to this.

[0064] In a preferred embodiment, step S120 includes:

[0065] S121, collecting point of interest data;

[0066] S122, collecting web crawler data;

[0067] S123, generating random data;

[0068] S124, performing data cleaning and variance screening on the point of interest data, web crawler data, and generated random data to obtain a high-quality data set, but not limited to this.

[0069] The specific implementation of the present invention is as follows:

[0070] Continue to refer Figure 1 As shown, a specific embodiment of the present invention

[0071] First, a multimodal multi-label binary classification model is constructed, and weight merging inference optimization is performed. In the process of training the multimodal multi-label binary classification model, for the input image data, features are first extracted through the image base model, and then processed by linear layer fine-tuning to enhance the expressiveness of image features. The multimodal multi-label binary classification model includes a linear layer that transforms the image features extracted by the image model, and the weight of the linear layer is set to the transpose of the linear layer. The model prediction is supervised based on training data and cross entropy loss, and the cache model weights are updated through SGD. The multimodal multi-label binary classification model is a key-value cache model. Under the parallel matrix architecture formed by weight merging, information is processed simultaneously in multiple dimensions.

[0072] Data annotation is performed based on the note database to generate a high-quality data set, including collecting point of interest data; collecting web crawler data; generating random data; and performing data cleaning and variance screening on point of interest data, web crawler data, and generated random data to obtain a high-quality data set.

[0073] The high-quality dataset is input into the image-text base model in the multimodal multi-label binary classification model to extract features and obtain text information enhancement features and image information enhancement features.

[0074] The enhanced image text features are obtained based on the text information enhancement features and the image information enhancement features.

[0075] Finally, multi-label classification is performed on the image based on the enhanced image text features.

[0076] In multi-label classification, traditional models either have high transformation costs and poor scalability, or have insufficient performance in the case of few samples, and are complex to apply in engineering. Transforming a single label can easily affect other labels. A new method is needed that has the training-free advantage of CLIP, while having strong performance in the case of few-sample classification, and is easy to transform and expand in engineering, and can independently process a single label without interfering with other labels.

[0077] To achieve the above objectives, the present invention proposes a multimodal image-text model. It draws on relevant successful architectural ideas and constructs a special structure for a small number of training sets. A key-value cache model is constructed through a specific design method, and the cache is converted into the weight of the adapter in a non-parametric manner, without the need for complex fine-tuning operations. It is worth noting that the weights constructed through reasonable design can enable this method to perform well without fine-tuning. If fine-tuning is allowed, further adjustment of the weights during network initialization can achieve higher accuracy and extremely fast convergence speed, effectively improving model performance. At the same time, the model is designed with full consideration of engineering practicality, reduces transformation costs, enhances scalability, and ensures that the classification effect of other labels is not affected when a single label is transformed.

[0078] 1. Build a multimodal multi-label binary classification model

[0079] In the field of deep learning for multi-label classification, model innovation is crucial to improving classification efficiency and accuracy. The multimodal image-text multi-label binary classification model proposed in this invention is based on a deep understanding of multi-label classification tasks. This invention uses a large amount of scene data and trains a multimodal image-text base model similar to CLIP through a complex deep learning algorithm. This base model contains rich feature information and lays a solid foundation for subsequent label classification. On top of this base, a binary classification linear layer is carefully constructed for each label, such as Figure 1In practice, the present invention has carried out such a design for 85 labels. This hierarchical construction method enables the model to handle classification problems of different labels more specifically.

[0080] The model exhibits many significant advantages. In terms of data utilization, only a small amount of data is needed to achieve good classification results for a single label. This is due to the clever parameter adjustment and feature extraction mechanism of the model during the training process, which effectively avoids overfitting. At the same time, the present invention uses a small amount of training data and cross entropy loss to supervise model predictions, and updates the cache model weights through SGD. Thanks to the good initialization of the cache model, only a small amount of data fine-tuning is needed to surpass the CLIP fine-tuning model with a longer training time. This training method greatly reduces the dependence on the amount of data, thereby reducing the risk of high resource consumption and improving data utilization efficiency. In terms of operating efficiency, the model has a fast inference speed and can meet real-time requirements. This is because the model structure fully considers the balance between computational complexity and time cost when designing, ensuring that accurate classification results can be quickly output when processing large amounts of data.

[0081] In addition, the model shows excellent flexibility and stability in the face of complex situations where labels change dynamically. When a new label is needed, a single label can be optimized in a targeted manner, and this local adjustment will not have any negative impact on the performance of other existing labels. This feature ensures that the entire multi-label classification system remains stable and efficient during long-term operation and continuous expansion, providing a valuable and high-quality solution for the field of multi-label classification.

[0082] 2. Weight Merging Inference Optimization

[0083] In the structure of the multimodal image text model, the fine-tuning linear layer is a crucial component. The present invention performs special processing on all the fine-tuning linear layers in the model, and through careful design and construction, it forms a parallel matrix. The structure of this parallel matrix has unique advantages and plays a key role in the reasoning process. When the model performs reasoning operations, it is necessary to obtain key information, and this parallel matrix structure is like an efficient information retrieval and processing center. It uses its own parallel characteristics to process multiple data paths at the same time, greatly improving the speed of obtaining key information. Originally, under the traditional structure, obtaining key information may require searching and matching in each linear layer in turn. This process involves a lot of calculations and data transmission, which often consumes more time. However, under this parallel matrix architecture formed by weight merging, information can be processed simultaneously in multiple dimensions, without the need to search one by one in the linear layer as in the traditional way, thereby bypassing many unnecessary intermediate steps and calculation overheads. In this way, the reasoning time is greatly reduced, so that the model can output accurate results more efficiently and quickly when facing a large amount of data for reasoning, effectively improving the performance and efficiency of the entire model, and providing a strong guarantee for its real-time and accuracy in various complex application scenarios.

[0084] 3. Data and graphic information enhancement

[0085] In this patent, the inventor proposes a series of innovative data and graphic information enhancement methods, which significantly improve the performance and generalization ability of the model. The overall architecture is shown in the figure Figure 2 .

[0086] In order to solve the problems of insufficient data and limited generalization ability that may exist in real data sets, the present invention uses an image generation model to generate an enhanced data set. In this way, the present invention can obtain more diverse data that covers different scenes, angles, and styles, enriching the distribution of training data. During the training process, these enhanced data are used together with the original data to train the model, so that the model can learn a wider range of patterns and features, thereby significantly improving the generalization ability of the model. This data enhancement method effectively alleviates the problem of model overfitting caused by the size and diversity limitations of the real data set, allowing the model to more accurately predict and classify when faced with a variety of unseen data.

[0087] In terms of image data processing, the present invention uses advanced visual models to extract features and optimizes them through a linear fine-tuning mechanism. Specifically, this linear layer is created in the code, and its weights are set to the transpose of the linear layer. The role of this linear layer is to transform the image features extracted by the image model so that they are more in line with the needs of subsequent model processing. During the training process, for the input image data, the features are first extracted through the image base model, and then processed by the linear layer fine-tuning, which effectively enhances the expressiveness of the image features. This enhancement method helps the model better capture the complex information in the image, thereby improving the performance of the model in image-related tasks.

[0088] For text information, the present invention introduces a large model to obtain a description of the label dimension. In the model, these text information are used to provide prior information for the multimodal image text encoder. Specifically, in the process of obtaining text features as a classifier for multimodal image text, by utilizing the powerful language understanding ability of the large model, richer semantic information is provided to the model. The input of this prior information enables the multimodal image text encoder to better understand the relationship between images and texts when processing image-text pairs, thereby improving the model's ability to jointly represent image and text information and enhancing the model's performance in multimodal tasks.

[0089] The present invention implements the method on a dataset of about 100 labels in the note-taking scenario. The model is fine-tuned after initializing the adapter weights, trained with a small number of training sets of 64, 128, and 256, and tested on the complete test set. The prompt inputs the two templates "This is a picture of [CLASS], this is another picture" and the caption generated by gpt (a detailed description of CLASS) into the CLIP text encoder for processing. Fine-tuning uses a batch of 256, SGD, a learning rate of 0.001, and a cosine schedule of 0.001. Hyperparameters include the values ​​of search_hp, search_scale, and search_step, init_beta is 1, init_alpha is 3 or 10, and gpt3_prompt_file has a specified path. The basic configuration includes the type of dataset, the shots value is 64 / 128 / 256, the backbone types of clip and dino are Tlip and Vit respectively, and the training cycles augment_epoch and train_epoch are both 10. In the note labeling scenario, the average precision and recall of 85-label few-sample classification predictions reached 85% / 80%. Training 256 samples took about 10 minutes on the V100. It was easy to add and tune samples, and the inference time for outputting 85-label results was only 500ms.

[0090] Figure 2 Schematic diagram of the structure of the multi-modal image text multi-label classification system of the present invention. Figure 2 As shown, an embodiment of the present invention further provides a multimodal image text multi-label classification system, which is used to implement the above-mentioned multimodal image text multi-label classification method. The multimodal image text multi-label classification system 5 includes:

[0091] Model building module 51, building a multi-modal multi-label binary classification model, and performing weight merging reasoning optimization;

[0092] A data annotation module 52 performs data annotation based on the note database to generate a high-quality data set;

[0093] A feature extraction module 53 inputs the high-quality data set into the image-text base model in the multimodal multi-label binary classification model to extract features, thereby obtaining text information enhancement features and image information enhancement features;

[0094] An enhanced feature module 54 is configured to obtain enhanced image text features based on the text information enhanced features and the image information enhanced features;

[0095] The label classification module 55 performs multi-label classification on the image based on the enhanced image text features.

[0096] In a preferred embodiment, the model building module 51 is configured to supervise model prediction based on training data and cross entropy loss, and update cached model weights through SGD, but is not limited thereto.

[0097] In a preferred embodiment, in step S110, the multimodal multi-label binary classification model is a key-value cache model, but is not limited thereto.

[0098] In a preferred embodiment, in step S110, under the parallel matrix architecture formed by weight merging, information is processed simultaneously in multiple dimensions, but not limited thereto.

[0099] In a preferred embodiment, in step S110, the multimodal multi-label binary classification model includes a linear layer that transforms the image features extracted by the image model, and the weight of the linear layer is set to the transpose of the linear layer, but is not limited thereto.

[0100] In a preferred embodiment, step S110 also includes: in the process of training the multimodal multi-label binary classification model, for the input image data, firstly extracting features through the image base model, and then processing through linear layer fine-tuning to enhance the expression ability of the image features, but not limited to this.

[0101] In a preferred embodiment, step S120 includes:

[0102] S121, collecting point of interest data;

[0103] S122, collecting web crawler data;

[0104] S123, generating random data;

[0105] S124, performing data cleaning and variance screening on the point of interest data, web crawler data, and generated random data to obtain a high-quality data set, but not limited to this.

[0106] The multimodal image-text multi-label classification system of the present invention can better understand the relationship between images and texts, thereby improving the model's ability to jointly represent image and text information, enhancing the performance of the model in multimodal tasks, and improving the overall effectiveness of multi-label classification.

[0107] The embodiment of the present invention further provides a multi-modal image text multi-label classification device, comprising a processor and a memory, wherein executable instructions of the processor are stored. The processor is configured to execute the steps of the multi-modal image text multi-label classification method by executing the executable instructions.

[0108] As shown above, the multimodal image-text multi-label classification device of this embodiment of the present invention can better understand the relationship between images and texts, thereby improving the model's ability to jointly represent image and text information, enhancing the performance of the model in multimodal tasks, and improving the overall effectiveness of multi-label classification.

[0109] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0110] Figure 3 Schematic diagram of the structure of the multi-modal image text multi-label classification device of the present invention. Figure 3 The electronic device 600 according to this embodiment of the present invention is described. Figure 3 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0111] like Figure 3 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0112] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0113] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0114] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0115] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0116] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0117] The embodiment of the present invention also provides a computer-readable storage medium for storing a program, and the steps of the multimodal image text multi-label classification method are implemented when the program is executed. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above method section of this specification.

[0118] As shown above, the multimodal image-text multi-label classification system of this embodiment of the present invention can better understand the relationship between images and texts, thereby improving the model's ability to jointly represent image and text information, enhancing the performance of the model in multimodal tasks, and improving the overall effectiveness of multi-label classification.

[0119] Figure 4 Schematic diagram of the structure of the computer-readable storage medium of the present invention. Figure 4 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0120] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0121] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0122] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0123] In summary, the purpose of the present invention is to provide a multimodal image-text multi-label classification method, system, device and storage medium, which can better understand the relationship between images and texts, thereby improving the model's ability to jointly represent image and text information, enhancing the performance of the model in multimodal tasks, and improving the overall performance of multi-label classification.

[0124] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A multi-modal image text multi-label classification method, characterized in that: The following steps are involved: S110, build a multi-modal multi-label binary classification model and perform weight merging reasoning optimization; S120, perform data annotation based on the note database to generate high-quality data sets; S130, inputting the high-quality data set into the image-text base model in the multimodal multi-label binary classification model to extract features, and obtaining text information enhancement features and image information enhancement features; S140, obtaining enhanced image text features based on the text information enhancement features and the image information enhancement features; S150: performing multi-label classification on the image based on the enhanced image text features.

2. The multi-modal image-text multi-label classification method according to claim 1, characterized in that: In step S110, model prediction is supervised based on training data and cross entropy loss, and cached model weights are updated through SGD.

3. The multi-modal image-text multi-label classification method according to claim 1, characterized in that: In step S110, the multimodal multi-label binary classification model is a key-value cache model.

4. The multi-modal image-text multi-label classification method according to claim 1, characterized in that: In the step S110, under the parallel matrix architecture formed by weight merging, the information is processed simultaneously in multiple dimensions.

5. The multi-modal image-text multi-label classification method according to claim 1, characterized in that: In step S110, the multimodal multi-label binary classification model includes a linear layer for transforming image features extracted by the image model, and the weight of the linear layer is set to the transpose of the linear layer.

6. The multi-modal image-text multi-label classification method according to claim 1, characterized in that: The step S110 also includes: in the process of training the multimodal multi-label binary classification model, for the input image data, firstly extracting features through the image base model, and then processing through linear layer fine-tuning to enhance the expression ability of the image features.

7. The multi-modal image-text multi-label classification method according to claim 1, characterized in that: The step S120 includes: S121, collecting point of interest data; S122, collecting web crawler data; S123, generating random data; S124, performing data cleaning and variance screening on the interest point data, web crawler data, and generated random data to obtain a high-quality data set.

8. A multimodal image-text multi-label classification system, used to implement the multimodal image-text multi-label classification method according to claim 1, characterized in that: include: Model building module, building a multi-modal multi-label binary classification model, and performing weight merging and reasoning optimization; Data annotation module, which performs data annotation based on the note database to generate high-quality data sets; A feature extraction module is used to input the high-quality data set into a picture-text base model in a multimodal multi-label binary classification model to extract features and obtain text information enhancement features and image information enhancement features; An enhanced feature module, which obtains enhanced image text features based on the text information enhanced features and the image information enhanced features; The label classification module performs multi-label classification on the image based on the enhanced image text features.

9. A multi-modal image text multi-label classification device, characterized in that: include: processor; a memory storing executable instructions of the processor; Wherein, the processor is configured to perform the steps of the multimodal image-text multi-label classification method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the multimodal image-text multi-label classification method according to any one of claims 1 to 7 are implemented.

Images