Feature extraction method, feature extraction model training method, and related devices

By training a feature extraction model using a large-scale multimodal generative language model, the problems of cross-space fusion and bias in multimodal information embedding feature extraction are solved, high-quality embedded feature generation is achieved, and the representation ability of multimodal information is improved.

CN123068909APending Publication Date: 2026-11-10TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510593176.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-09
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing technologies for embedding features of multimodal information suffer from poor cross-space fusion and insufficient fusion. Furthermore, feature extraction models are subject to bias and text sequence truncation issues, resulting in insufficient accuracy and representational ability of embedded features.

Method used

A multimodal large-scale generative language model is used for training. Through contrastive learning and cue learning, a feature extraction model is generated. The rich knowledge of the multimodal large-scale generative language model is used to map multimodal information to a unified semantic space and output a single embedded feature.

Benefits of technology

It achieves full fusion of multimodal information across spaces, enhances the representational ability of embedded features, avoids the decline in representational ability caused by cross-space fusion, and corrects biases through the capabilities of the model itself and human feedback, generating more objective and scientific embedded features.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123068909A_ABST
    Figure CN123068909A_ABST
Patent Text Reader

Abstract

The application discloses a feature extraction method, a feature extraction model training method and related devices, and belongs to the technical field of artificial intelligence. The feature extraction method comprises: obtaining business data, the business data comprising data of at least two modalities, the content of the data of the at least two modalities having a correlation relationship; obtaining prompt information, the prompt information comprising a single target unit, the prompt information being used to instruct a feature extraction model to output embedding features corresponding to the business data to a preset information carrier, the preset information carrier being an information carrier corresponding to a word in a word sequence corresponding to the target unit, the word sequence being obtained by performing word segmentation processing on the prompt information; based on the business data and the prompt information, triggering the feature extraction model to output the embedding features corresponding to the business data; and the feature extraction model being a model obtained by training a multi-modal large generative language model based on contrast learning. The method improves the representation ability of the extracted embedding features for multi-modal information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a feature extraction method, a feature extraction model training method, and related apparatus. Background Technology

[0002] With the development of multimodal information, the demand for extraction of embedded features of multimodal information by related technologies is becoming increasingly urgent. High-quality embedded features of multimodal information can be used to perform a variety of tasks, such as information aggregation, text understanding, cross-modal retrieval, intelligent recommendation, cold start recommendation, etc. These tasks require that the embedded features of multimodal information have consistency and comparability in a unified vector space. Therefore, the quality of the embedded features of multimodal information has a significant impact on the performance of these tasks.

[0003] However, since the data of different modalities in multimodal information come from different sources, their inherent feature representations and semantic structures often have significant differences. This leads to alignment problems and insufficient fusion due to cross-space fusion of different modal information during the extraction of embedded features of multimodal information. Therefore, there is currently a lack of effective methods in related technologies that can directly extract multimodal information into a single embedded feature. Summary of the Invention

[0004] This application provides a feature extraction method, a feature extraction model training method, and related apparatus, which can perform single embedded feature extraction on multimodal information. The extraction process fully solves the alignment problem and the problem of insufficient fusion, and significantly improves the ability of the extracted embedded features to represent the corresponding multimodal information.

[0005] According to one aspect of the embodiments of this application, a feature extraction method is provided, the method comprising:

[0006] Acquire business data, which includes data from at least two modalities, and the content of the data from the at least two modalities is related.

[0007] Obtain prompt information, the prompt information including a single target unit, the prompt information being used to instruct the feature extraction model to output the embedded features corresponding to the business data to a preset information carrier, the preset information carrier being the information carrier corresponding to the word in the word sequence corresponding to the target unit, the word sequence being obtained by word segmentation of the prompt information;

[0008] Based on the business data and the prompt information, the feature extraction model is triggered to output the embedded features corresponding to the business data;

[0009] The feature extraction model is a model obtained by training a multimodal large-scale generative language model based on contrastive learning.

[0010] According to another aspect of the embodiments of this application, a method for training a feature extraction model is provided, the method comprising:

[0011] Multiple samples are acquired, each sample including data from at least two modalities, and the contents of the data from the at least two modalities are related.

[0012] The prompt information includes a single target unit. The prompt information is used to instruct the multimodal large-scale generative language model to output the sample embedding features corresponding to the sample to a preset information carrier. The preset information carrier is the information carrier corresponding to the word in the word sequence that corresponds to the target unit. The word sequence is obtained by segmenting the prompt information.

[0013] Extract the sample embedding features corresponding to each sample, wherein the sample embedding features are the information corresponding to the preset information carrier in the output result generated by the multimodal large-scale generative language model based on the sample and the prompt information;

[0014] Based on each sample and its corresponding sample embedding features, the parameters of the multimodal large-scale generative language model are adjusted through contrastive learning to obtain a feature extraction model.

[0015] According to one aspect of the embodiments of this application, a feature extraction apparatus is provided, the apparatus comprising:

[0016] A business data acquisition module is used to acquire business data, which includes data in at least two modalities, and the contents of the at least two modalities are related.

[0017] The prompt information acquisition module is used to acquire prompt information, which includes a single target unit. The prompt information is used to instruct the feature extraction model to output the embedded features corresponding to the business data to a preset information carrier. The preset information carrier is the information carrier corresponding to the word in the word sequence that corresponds to the target unit. The word sequence is obtained by segmenting the prompt information.

[0018] The feature extraction module is used to trigger the feature extraction model to output the embedded features corresponding to the business data based on the business data and the prompt information.

[0019] The feature extraction model is a model obtained by training a multimodal large-scale generative language model based on contrastive learning.

[0020] According to another aspect of the embodiments of this application, a feature extraction model training apparatus is provided, the apparatus comprising:

[0021] A sample acquisition module is used to acquire multiple samples, each sample including data from at least two modalities, the contents of the data from the at least two modalities having a correlation relationship;

[0022] The training module is used to perform the following operations:

[0023] The prompt information includes a single target unit. The prompt information is used to instruct the multimodal large-scale generative language model to output the sample embedding features corresponding to the sample to a preset information carrier. The preset information carrier is the information carrier corresponding to the word in the word sequence that corresponds to the target unit. The word sequence is obtained by segmenting the prompt information.

[0024] Extract the sample embedding features corresponding to each sample, wherein the sample embedding features are the information corresponding to the preset information carrier in the output result generated by the multimodal large-scale generative language model based on the sample and the prompt information;

[0025] Based on each sample and its corresponding sample embedding features, the parameters of the multimodal large-scale generative language model are adjusted through contrastive learning to obtain a feature extraction model.

[0026] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the above-described feature extraction method or feature extraction model training method.

[0027] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored in the storage medium, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the above-described feature extraction method or feature extraction model training method.

[0028] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform an action to implement the above-described feature extraction method or feature extraction model training method.

[0029] The technical solution provided in this application can bring the following beneficial effects:

[0030] This application proposes a feature extraction method, a feature extraction model training method, and related apparatus. This feature extraction method fully utilizes the rich knowledge contained in large-scale multimodal generative language models, mapping multimodal information covering multiple modalities to a unified semantic space to obtain a single embedded feature that can represent the multimodal information. This embedded feature can be directly used to measure the differences and correlations between different multimodal information. In this way, the problem of cross-space fusion of features extracted from data of different modalities is effectively solved, avoiding the problem of decreased representational capability of the fusion results caused by cross-space fusion in related technologies.

[0031] This application embodiment outputs embedded features representing multimodal information through a large-scale multimodal generative language model. Compared with other models in related technologies for processing multimodal information, the large-scale multimodal generative language model has the characteristic of fairness. It can correct various biases generated during knowledge learning through the model's own capabilities and reinforcement learning based on human feedback. This makes the feature extraction model built based on the large-scale multimodal generative language model in this application embodiment more objective and scientific, and has stronger representational capabilities than other models in related technologies that can extract features from multimodal information.

[0032] The feature extraction model trained based on the multimodal large-scale generative language model in this embodiment has a stronger ability to support the amount of information in multimodal information, and will not easily truncate, effectively enhancing the feature extraction model's ability to characterize multimodal information. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of an application runtime environment provided in one embodiment of this application;

[0035] Figure 2 This is a flowchart of a feature extraction method provided in one embodiment of this application;

[0036] Figure 3 This is a flowchart of a feature extraction model training method provided in one embodiment of this application;

[0037] Figure 4This is a schematic diagram of the training process provided in one embodiment of this application. Figure 1 ;

[0038] Figure 5 This is a schematic diagram of the training process provided in one embodiment of this application. Figure 2 ;

[0039] Figure 6 This is a flowchart of a sample embedding feature extraction method provided in one embodiment of this application;

[0040] Figure 7 This is a schematic diagram of a sample embedding feature extraction method provided in one embodiment of this application;

[0041] Figure 8 This is a block diagram of a feature extraction apparatus provided in one embodiment of this application;

[0042] Figure 9 This is a block diagram of a feature extraction model training apparatus provided in one embodiment of this application;

[0043] Figure 10 This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0044] Before introducing the method embodiments provided in this application, a brief introduction to the relevant background of the method embodiments in this application will be given.

[0045] In the context of the internet age, content production has grown rapidly. Numerous content creators can share content and interact with consumers on various content interaction platforms, forming a vast content community. This community contains data of multiple modalities, generating a large amount of multimodal information, including images, videos, audio, and text. This data is not only massive in quantity but also diverse in modality, posing challenges to content understanding and processing. To effectively represent multimodal information, related technologies typically use corresponding feature extraction methods for different modalities within the multimodal information, and then fuse the features corresponding to each modality across space. However, this fusion process suffers from poor spatial alignment and insufficient fusion capability, resulting in significant deficiencies in the representational power of the embedded features corresponding to the multimodal information generated by these technologies.

[0046] Related technologies also include methods for feature extraction from multimodal information of specific modalities, such as CLIP (Contrastive Language–Image Pre-training), a multimodal visual and text learning method capable of extracting embedding features from multimodal information covering both image and text modalities. The CLIP model was trained using over 400 million unfiltered network datasets, containing significant noise, which caused it to learn many social biases, impacting the accuracy of its output embedding features. Furthermore, due to their underlying design, models in related technologies, including CLIP, that extract features from multimodal information are limited to a maximum input text sequence length of 76 characters; any exceeding this length is truncated. This means these feature extraction models can only extract information from short texts; if long texts are present in the multimodal information, truncation occurs, resulting in inaccurate extracted embedding features.

[0047] In summary, the relevant technologies face the following problems when extracting embedded features for multimodal information covering multiple modalities:

[0048] (1) The technical solution of extracting features for each modality separately and then fusing the extracted results across space will result in poor spatial alignment and insufficient fusion capability.

[0049] (2) Existing feature extraction models in related technologies suffer from bias and text sequence truncation problems, which reduce the accuracy of the extracted embedded features.

[0050] In view of this, this application proposes a novel method for extracting embedded features of multimodal information. This method is based on a large multimodal generative language model. By training the large multimodal generative language model in a targeted manner, it can output embedded features of multimodal information, thus obtaining a feature extraction model.

[0051] This application provides a training method based on a multimodal large-scale generative language model for the fields of artificial intelligence and machine learning. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making functions. Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to give computers intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0052] Large Language Models (LLMs) are computer models capable of processing and generating natural language. Based on deep learning techniques and a Transformer architecture, LLMs possess hundreds of millions of parameters. They can predict the next word or sentence by learning the statistical patterns and semantic information of language data. The capabilities of LLMs increase as the input dataset and parameter space expand. They are used in various application areas such as robotics, machine learning, machine translation, speech recognition, and image processing. Multimodal Large Language Models (MLLMs) can not only understand and generate natural language but also integrate multimodal information such as images and sound, giving them a significant advantage in handling complex data scenarios. A Transformer is a neural network that learns context and thus meaning by extracting relationships from sequential data. Transformers apply attention or self-attention mechanisms to encode the mutual influence and interdependence of data in distant locations.

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0054] Please refer to Figure 1This diagram illustrates an application runtime environment provided in one embodiment of this application. The application runtime environment may include: terminal 10 and server 20.

[0055] Terminal 10 includes, but is not limited to, electronic devices such as mobile phones, computers, smart voice interaction devices, smart home appliances, in-vehicle terminals, game consoles, e-book readers, multimedia playback devices, and wearable devices. Application clients can be installed on terminal 10.

[0056] In this embodiment, the application described above can be any application capable of using the feature extraction method of this embodiment. Typically, the application is a video application. Of course, besides video applications, other types of applications can also use the feature extraction method of this embodiment. For example, news applications, social applications, interactive entertainment applications, browser applications, shopping applications, content sharing applications, virtual reality (VR) applications, augmented reality (AR) applications, etc., are not limited in this embodiment. Optionally, the terminal 10 runs a client of the above-mentioned application.

[0057] Server 20 provides background services for the clients of the applications in terminal 10. For example, server 20 can be the background server for the aforementioned applications. Server 20 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud technology-related services such as cloud databases, cloud computing, cloud functions, cloud storage, and cloud communication. It can also provide basic cloud computing services such as network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, server 20 can simultaneously provide background services for applications in multiple terminals 10. Cloud technology refers to a hosting technology that unifies hardware, software, network, and other resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology applied based on the cloud computing business model. It can form resource pools, be used on demand, and is flexible and convenient.

[0058] Optionally, terminal 10 and server 20 can communicate with each other via network 30. Terminal 10 and server 20 can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0059] Please refer to Figure 2 The diagram illustrates a flowchart of a feature extraction method provided in one embodiment of this application. This method can be applied to computer devices, which refer to electronic devices capable of data computation and processing. For example, the entity executing each step may be... Figure 1 The application runtime environment shown is server 20. This method may include the following steps:

[0060] Step 210. Obtain business data, which includes data from at least two modalities, and the content of the data from the at least two modalities is related.

[0061] This application does not limit the scope to at least two modalities; alternatively, it may include, but is not limited to, text modalities, image modalities, and audio modalities. This application does not limit the domain of the business data; for example, it may involve various natural technology fields in the current information world, including everyday and professional fields.

[0062] This application does not limit the source of business data. For example, it can come from posts in content communities, which may include data in various modalities such as images, text, and videos. It can also come from comments in video interaction applications; by extracting frames from the video corresponding to the comment, image modal data can be obtained, and the comment itself can contain both text modal data and image modal data. Business data can also come from comments in music interaction applications; the music corresponding to the comment contains audio modal data, and the comment itself can contain both text modal data and image modal data.

[0063] This application does not limit the correlation between the content of at least two modalities of data in the business data; a certain degree of correlation is sufficient. For example, if a post in a content community includes both image and text modal data, and these data originate from the same post, then their content is considered correlated and can be used to construct business data. Similarly, if a video comment in a video interaction application includes text modal data, image modal data can be extracted from the video, and this data corresponds to the text modal data in the comment; therefore, their content is considered correlated and can be used to construct business data. Likewise, if a music comment in a music interaction application includes text modal data, audio modal data can be obtained from the music, and this audio modal data corresponds to the text modal data in the comment; therefore, their content is considered correlated and can be used to construct business data.

[0064] Step 220. Obtain prompt information, the prompt information including a single target unit, the prompt information being used to instruct the feature extraction model to output the embedded features corresponding to the business data to a preset information carrier, the preset information carrier being the information carrier corresponding to the word in the word sequence corresponding to the target unit, the word sequence being obtained by word segmentation of the prompt information; wherein, the feature extraction model is a model obtained by training a multimodal large-scale generative language model based on contrastive learning.

[0065] The feature extraction model is obtained by training a multimodal large generative language model (MLLM) based on contrastive learning. The cue information refers to the information used for cue learning in the MLLM, and therefore can be used to guide the output of the feature extraction model. Cue learning, or prompt tuning, is a learning method in machine learning that significantly improves the performance of the MLLM by adding "cue information" to the input as an information enhancement technique without significantly changing the structure and parameters of the MLLM. This cue information can be seen as an instruction for the task.

[0066] In order for the feature extraction model to output embedded features corresponding to business data, this application embodiment needs to explicitly include a single target unit for carrying a single embedded feature in the prompt information input to the feature extraction model. This enhances the multimodal generative large language model's focus on and processing capabilities for multimodal information fusion. For example, the prompt information may include "Pack all the input information into one word: <embed> ",in <embed> A prompt is a single target unit used to carry a single embedded feature. Before inputting the prompt into the feature extraction model, it can be segmented into a word sequence, where each word corresponds to a token. The token corresponding to the word of the target unit can be understood as the carrier of the predefined information. In NLP tasks, a token usually refers to the smallest predictive target corresponding to the smallest meaningful part or unit of the text. NLP tasks refer to Natural Language Processing tasks, which involve computers' understanding and processing of human language. The task of generating embedded features based on MLLM can be understood as a type of NLP task.

[0067] Step 230. Based on the business data and the prompt information, trigger the feature extraction model to output the embedded features corresponding to the business data.

[0068] By inputting business data and prompts into the feature extraction model, or by inputting information obtained from further processing of business data and prompts into the feature extraction model, the model can be triggered to output corresponding results. The token information corresponding to the target unit in the output results can then be extracted as the embedded features of the business data. Continuing with the previous example, <embed> The information in the corresponding token is the embedded feature of the business data.

[0069] This application proposes a feature extraction method that fully utilizes the rich knowledge contained in large-scale generative multimodal language models. It maps multimodal information covering multiple modalities to a unified semantic space, obtaining a single embedded feature that can represent the multimodal information. This embedded feature can be directly used to measure the differences and correlations between different multimodal information. This effectively solves the problem of cross-space fusion of features extracted from different modalities, avoiding the degradation of the representational ability of the fusion results caused by cross-space fusion in related technologies.

[0070] This application embodiment outputs embedded features representing multimodal information through a large-scale multimodal generative language model. Compared with other models in related technologies for processing multimodal information, the large-scale multimodal generative language model has the characteristic of fairness. It can correct various biases generated during knowledge learning through the model's own capabilities and reinforcement learning based on human feedback. This makes the feature extraction model built based on the large-scale multimodal generative language model in this application embodiment more objective and scientific, and has stronger representational capabilities than other models in related technologies that can extract features from multimodal information.

[0071] The feature extraction model trained based on the multimodal large-scale generative language model in this embodiment has a stronger ability to support the amount of information in multimodal information, and will not easily truncate, effectively enhancing the feature extraction model's ability to characterize multimodal information.

[0072] In one exemplary implementation, the embedded features of the business data are latent vectors for the target unit output by the last network layer in the multimodal large-scale generative language model. As an internal representation of the network layer, the latent vector captures the complex features of the input data. Since this output latent vector is from the last network layer of the multimodal large-scale generative language model, it contains a comprehensive understanding of the input data by the MLLM, sufficient cross-space alignment, and full fusion, thus possessing high information content and representational power.

[0073] In an exemplary embodiment, the method further includes: acquiring a preset dimension corresponding to the business data; and inputting the embedded features into a multilayer perceptron to obtain target embedded features with the preset dimensions. The preset dimensions are pre-set according to the actual application scenario and requirements to ensure that the target embedded features can meet the requirements of subsequent processing or analysis. As a feedforward neural network, the multilayer perceptron can map the embedded features to a space of preset dimensions through a nonlinear transformation, thereby obtaining the target embedded features. This process not only preserves the key information in the original embedded features but also further enhances the representational ability and applicability of the features. The multilayer perceptron can map the dimensions of the embedded features from the fixed dimensions of the original output of the feature extraction model, such as 4096 dimensions, to the dimensions required by downstream tasks, such as 512 dimensions. It can also be set to different dimensions such as 64, 128, or 256 dimensions, allowing for a more flexible and cost-effective way to obtain a better characterization of the embedded features.

[0074] In one exemplary embodiment, the data of the at least two modalities includes text data and image data. The step of triggering the feature extraction model to output the embedded features corresponding to the business data based on the business data and the prompt information includes: fusing the text information corresponding to the text data and the image data with the prompt information to obtain target prompt information; performing word segmentation on the target prompt information to obtain a word sequence; performing feature extraction on the word sequence to obtain a text feature sequence; performing visual feature extraction on the image data to obtain visual features; inputting the text feature sequence and the visual features into the feature extraction model to obtain an output result; and extracting information corresponding to the preset information carrier from the output result to obtain the embedded features corresponding to the business data.

[0075] This application does not limit the text information corresponding to the image data. For example, it may be at least one of the image's address information, name information, size information, etc.

[0076] Word segmentation is a fundamental task in natural language processing, aiming to divide continuous text information into independent word units. This application does not limit the word segmentation method; for example, rule-based, statistical, or deep learning-based word segmentation methods can be used. The process of feature extraction from the word sequence can be based on various methods and techniques, including but not limited to bag-of-words models, word embedding models, or more complex deep learning models. This application does not limit the specific feature extraction method; the key is to transform the segmented word sequence into a text feature sequence that reflects the key information and semantic features of the text. The text feature sequence will serve as the basis for feature extraction model analysis. By extracting visual features from image data, key information such as color, texture, and shape can be captured. This application does not limit the visual feature extraction method; convolutional neural networks, residual networks, or other image recognition techniques can be used to perform visual feature extraction.

[0077] During the word segmentation stage, a unique token corresponding to the target unit is directly generated. At the final output of the feature extraction model, the latent vector of the last layer corresponding to this token is extracted as the information corresponding to the token, i.e., the information corresponding to the preset information carrier. This information is a comprehensive single embedded feature of the input image and text data. The feature extraction model is based on the Multimodal Large Language Model (MLLM), fully utilizing the rich knowledge and powerful logical reasoning and multimodal content understanding capabilities contained in the MLLM to obtain high-quality single embedded features.

[0078] Please refer to Figure 3 The diagram illustrates a flowchart of a feature extraction model training method proposed in an embodiment of this application. This method allows the aforementioned feature extraction model to be trained. The training method includes:

[0079] S310. Obtain multiple samples, each sample including data from at least two modalities, the contents of the data from the at least two modalities having a correlation.

[0080] This application does not limit the number of modalities to at least two; alternatively, it may include, but is not limited to, text modalities, image modalities, audio modalities, etc. This application does not limit the field of the samples; for example, it may involve various natural technology fields in the current information world, including everyday and professional fields.

[0081] This application does not limit the source of the samples. For example, they can come from posts in content communities, which may include data in various modalities such as images, text, and videos. They can also come from comments in video interaction applications; by extracting frames from the video corresponding to the comment, image modal data can be obtained, and the comment itself may contain both text modal data and image modal data. They can also come from comments in music interaction applications; the music corresponding to the comment contains audio modal data, and the comment itself may contain both text modal data and image modal data.

[0082] This application does not limit the correlation between the content of at least two modalities of data in the sample; a certain degree of correlation is sufficient. For example, if a post in a content community includes both image and text modal data, and these data originate from the same post, then their content is considered correlated and can be used to construct a sample. Similarly, if a video comment in a video interaction application includes text modal data, image modal data can be extracted from the video, and if this data corresponds to the text modal data in the comment, then their content is considered correlated and can be used to construct a sample. Likewise, if a music comment in a music interaction application includes text modal data, audio modal data can be obtained from the music, and if this audio corresponds to the text modal data in the comment, then their content is considered correlated and can be used to construct a sample.

[0083] Step 320. Determine the prompt information, which includes a single target unit. The prompt information is used to instruct the multimodal large-scale generative language model to output the sample embedding features corresponding to the sample to a preset information carrier. The preset information carrier is the information carrier corresponding to the word in the word sequence that corresponds to the target unit. The word sequence is obtained by segmenting the prompt information.

[0084] Prompt information refers to the information used for prompt learning in large-scale multimodal generative language models. Prompt learning, or prompt tuning, is a learning method in machine learning that significantly improves the performance of MLLMs by adding "prompt information" to the input as an information enhancement technique without significantly changing the structure and parameters of the MLLM. This prompt information can be seen as an instruction for the task and also as a reuse of the pre-training objective. Its essence is to enhance the training of parameter effectiveness. By generating prompts separately, the MLLM is fine-tuned based on the prompt information to achieve the training objective.

[0085] In the embodiments of this application, to enable the MLLM to output the sample embedding feature corresponding to a sample, it is necessary to explicitly include a single target unit for carrying a single sample embedding feature in the prompt information input to the MLLM, so as to improve the multimodal generative large language model's attention and processing capability for multimodal information fusion. For example, the prompt information may include "Pack all input information into one word: <EMBED", where <embed> is the single target unit for carrying a single sample embedding feature. Before inputting the prompt information into the MLLM, the prompt may be subjected to word segmentation processing to obtain a word sequence, each word in the word sequence corresponds to a token, and the token corresponding to the word of the target unit may be understood as the preset information carrier. In NLP tasks, a token generally refers to the minimum prediction target corresponding to the smallest meaningful part or unit obtained by segmenting text. The NLP tasks refer to natural language processing tasks, which involve computers understanding and processing human languages. The task of generating embedding features based on the MLLM may be regarded as a type of NLP task.

[0086] Step S330. Extract the sample embedding feature corresponding to each of the samples, where the sample embedding feature is information corresponding to the preset information carrier in the output result generated by the multimodal large generative language model based on the sample and the prompt information.

[0087] By inputting the sample and the prompt information into the MLLM, or inputting information obtained by further processing the sample and the prompt information into the MLLM, the MLLM can be triggered to output a corresponding result, and the information of the token corresponding to the target unit in the output result is extracted as the sample embedding feature. Following the example above, <embed> the information of the corresponding token is the sample embedding feature.

[0088] In an exemplary embodiment, the sample embedding feature is a hidden vector for the target unit output by the last network layer in the multimodal large generative language model. As an internal representation of the network layer, the hidden vector can capture complex features of the input data. Since this output hidden vector is output from the last network layer of the multimodal large generative language model, it contains the result of the MLLM's comprehensive understanding of the input data, sufficient cross-space alignment and sufficient fusion, and thus has very high information content and representation capability.

[0089] Step S340. Adjust parameters of the multimodal large generative language model through contrastive learning based on each of the samples and the corresponding sample embedding feature, to obtain a feature extraction model.

[0090] Based on the embedded features of each sample, a contrastive learning loss is calculated for the contrastive learning process. This loss allows for the adjustment of the MLLM parameters, thereby optimizing the MLLM's feature extraction capability for multimodal information in the samples. Specifically, by minimizing the contrastive learning loss, the sample embedded features generated by the MLLM can more accurately represent multimodal information and exhibit better performance in similarity-based tasks, such as retrieval, recommendation, clustering, and cold start tasks. This application does not limit the method of parameter adjustment; for example, gradient descent can be used. Nor does it limit the stopping condition for parameter tuning; for example, parameter tuning can be stopped based on the convergence of the contrastive learning loss or after adjusting the parameters a fixed number of times. Through the above method, a feature extraction model can be obtained that can more accurately extract single embedded features from multimodal information.

[0091] This application proposes a feature extraction model training method. This method can fully utilize the rich knowledge contained in large-scale multimodal generative language models to map multimodal information covering multiple modalities to a unified semantic space, obtaining a single embedded feature that can represent multimodal information. This embedded feature can be directly used to measure the differences and correlations between different multimodal information. In this way, the problem of cross-space fusion of features extracted from data of different modalities is effectively solved, avoiding the problem of reduced representational ability of fusion results caused by cross-space fusion in related technologies.

[0092] This application embodiment outputs embedded features representing multimodal information through a large-scale multimodal generative language model. Compared with other models in related technologies for processing multimodal information, the large-scale multimodal generative language model has the characteristic of fairness. It can correct various biases generated during knowledge learning through the model's own capabilities and reinforcement learning based on human feedback. This makes the feature extraction model built based on the large-scale multimodal generative language model in this application embodiment more objective and scientific, and has stronger representational capabilities than other models in related technologies that can extract features from multimodal information.

[0093] The feature extraction model trained based on the multimodal large-scale generative language model in this embodiment has a stronger ability to support the amount of information in multimodal information, and will not easily truncate, effectively enhancing the feature extraction model's ability to characterize multimodal information.

[0094] In one exemplary embodiment, each sample corresponds to at least one level of category information, and the prompt information is further used to instruct the multimodal large-scale generative language model to output at least one level of predicted classification based on the sample. For example, the prompt information may also include "Please output multi-level classification information describing the type, such as first and second level classifications being: XX, XX". Please refer to [reference needed]. Figure 4 It illustrates the training process of an embodiment of this application. Figure 1 The step of adjusting the parameters of the multimodal large-scale generative language model through contrastive learning based on each sample and its corresponding sample embedding features includes:

[0095] Step 410. Based on at least one level of category information and at least one level of predicted classification for each sample, determine the corresponding classification loss.

[0096] This classification loss can be used to quantify the difference between each predicted classification of a sample and the category information corresponding to that predicted classification. Based on the classification loss, the parameters of the multimodal large-scale generative language model can be adjusted using the backpropagation algorithm to optimize the performance of the feature extraction model. In this process, the multimodal large-scale generative language model can learn more accurate feature representations, resulting in better performance of the embedded features in classification tasks. This application does not limit the quantification method of the classification loss; for example, cross-entropy loss, mean squared error loss, or other suitable loss functions can be used to calculate the classification loss. By reasonably selecting and adjusting the quantification method of the classification loss, the performance of the multimodal large-scale generative language model in feature extraction and classification tasks can be further improved.

[0097] Step 420. Adjust the parameters of the multimodal large-scale generative language model based on the classification loss and the contrastive learning loss, wherein the contrastive learning loss is the loss generated during the contrastive learning process.

[0098] This application does not limit the specific method for adjusting the parameters of the multimodal large-scale generative language model based on the classification loss and contrastive learning loss. For example, gradient descent or other optimization algorithms can be used to iteratively update the parameters of the MLLM to minimize the weighted sum of the classification loss and contrastive learning loss. This application does not limit the weights and can set them according to actual needs. During this process, an appropriate learning rate and number of iterations can be selected according to the actual situation to ensure that the MLLM can effectively converge and achieve optimal performance. By comprehensively considering the classification loss and contrastive learning loss, this application can more comprehensively optimize the embedding feature extraction performance of the MLLM, making the extracted embedding features more accurate and showing better performance in classification tasks. According to actual training experience, adding classification tasks can make the training of the MLLM more stable and improve the accuracy of embedding feature extraction for multimodal information.

[0099] In one exemplary embodiment, each sample corresponds to at least one label, and the prompt information is further used to instruct the multimodal large-scale generative language model to output at least one labeling result based on the sample. For example, the prompt information may also include "Please label the full amount of input information; the labeling result can be XX, XX". The difference between labels and the category information mentioned above is that category information can represent the result of content classification based on the content of multimedia information, such as "era," "Republic of China," "suspense," and "comedy," which are all content-based classification results. Labels can be markers representing other information besides content classification results, such as "exciting" indicating degree information or "popular" indicating interaction popularity.

[0100] Please refer to Figure 5 It illustrates the training process of an embodiment of this application. Figure 2 The step of adjusting the parameters of the multimodal large-scale generative language model through contrastive learning based on each sample and its corresponding sample embedding features includes:

[0101] Step S510. Based on at least one label and at least one labeling result corresponding to each sample, determine the corresponding label prediction loss.

[0102] The label prediction loss can be used to quantify the difference between each label of a sample and the corresponding labeling result. Based on the label prediction loss, the parameters of the multimodal large-scale generative language model can be adjusted through backpropagation to optimize the performance of the feature extraction model. In this process, the multimodal large-scale generative language model can learn more accurate feature representations, and the resulting embedded features perform better in the label prediction task. This application does not limit the quantification method of the label prediction loss; for example, cross-entropy loss function, mean squared error loss function, or other suitable loss functions can be used to calculate the label prediction loss. By reasonably selecting and adjusting the quantification method of the label prediction loss, the performance of the multimodal large-scale generative language model in feature extraction and label prediction tasks can be further improved.

[0103] Step S520. Adjust the parameters of the multimodal large-scale generative language model based on the label prediction loss and the contrastive learning loss, wherein the contrastive learning loss is the loss generated during the contrastive learning process.

[0104] This application does not limit the specific method for adjusting the parameters of the multimodal large-scale generative language model based on the label prediction loss and contrastive learning loss. For example, gradient descent or other optimization algorithms can be used to iteratively update the parameters of the MLLM to minimize the weighted sum of the label prediction loss and contrastive learning loss. This application does not limit the weights and can set them according to actual needs. During this process, an appropriate learning rate and number of iterations can be selected according to the actual situation to ensure that the MLLM can effectively converge and achieve optimal performance. By comprehensively considering the label prediction loss and contrastive learning loss, this application can more comprehensively optimize the embedding feature extraction performance of the MLLM, making the extracted embedding features more accurate and showing better performance in the label prediction task. According to actual training data, adding a label prediction task can make the training of the MLLM more stable and improve the accuracy of embedding feature extraction for multimodal information.

[0105] In some implementations, the relevant content from S410-420 and S510-S520 above can be combined, allowing the prompt information to trigger MLLM to perform classification and label prediction tasks, and to label each sample with classification information and labels. This enables the parameters of the multimodal large-scale generative language model to be adjusted based on the classification loss, the label prediction loss, and the contrastive learning loss. The parameter tuning process is as described above and will not be repeated here. The weights of the classification loss, the label prediction loss, and the contrastive learning loss are not limited. This approach simplifies the training process and improves training efficiency. Specifically, MLLM can handle classification and label prediction tasks simultaneously. Based on these two tasks, MLLM's ability to extract embedded features for multimodal information is significantly enhanced, the training process is more stable, the convergence speed is faster, and the extracted embedded features simultaneously meet the requirements of both label prediction and classification tasks, resulting in higher accuracy.

[0106] This application embodiment trains a multimodal large-scale generative language model through contrastive learning, which requires forming positive and negative sample pairs based on the aforementioned multiple samples. Since large-scale sample pairs are difficult to obtain, this application embodiment proposes to adopt a self-supervised approach to obtain positive and negative sample pairs.

[0107] Specifically, multiple multimodal media contents can be acquired first; each multimodal media content can be data augmented twice to obtain two corresponding samples. This application does not limit the specific data augmentation method; taking multimodal media content covering both image and text modalities as an example, the data augmentation method is explained. This application does not limit the source of the image and text content; for example, it can come from posts in a content community. Data augmentation of the image portion of the post includes at least one of random cropping, random color perturbation, random grayscale conversion, random Gaussian blurring, and random scaling. For the text portion of the post, data augmentation includes at least one of random synonym replacement, random synonym insertion, random deletion, and random shuffling. Alternatively, the image or text portion of a post can be randomly discarded, as long as at least one of the image or text modal information is present, samples can still be generated.

[0108] Next, positive and negative sample pairs can be determined based on the data-augmented samples, and then the MLLM can be optimized through contrastive learning. Specifically, the step of adjusting the parameters of the multimodal large-scale generative language model through contrastive learning based on each sample and its corresponding sample embedding features includes: determining samples corresponding to the same multimodal media content as positive sample pairs and samples not corresponding to the same multimodal media content as negative sample pairs. A contrastive learning loss is obtained based on the sample embedding features corresponding to the positive sample pairs and the sample embedding features corresponding to the negative sample pairs; the parameters of the multimodal large-scale generative language model are then adjusted based on the contrastive learning loss.

[0109] Contrastive learning loss is formed by calculating the similarity between positive sample pairs and the similarity between negative sample pairs. Specifically, the higher the similarity between positive sample pairs, the smaller the contrastive learning loss; the lower the similarity between negative sample pairs, the smaller the contrastive learning loss. The design of the contrastive learning loss function aims to enable the model to better learn the correlation and differences between multimodal media content. This application does not limit the loss function used for contrastive learning loss; for example, a triplet loss function or a contrastive loss function can be used. The design idea of ​​these loss functions is to calculate the similarity or distance between sample pairs, making positive sample pairs as close as possible in the feature space and negative sample pairs as far apart as possible. In practical applications, an appropriate loss function can be selected according to specific needs and scenarios, and its effect can be verified through experiments. This application does not limit the specific method of adjusting the parameters of the multimodal large-scale generative language model based on contrastive learning loss; please refer to the preceding text, which will not be repeated here.

[0110] In one exemplary implementation, the media data of the at least two modalities includes text data and image data. Please refer to... Figure 6 The flowchart illustrates a sample embedding feature extraction method according to an embodiment of this application. The extraction of sample embedding features corresponding to each sample includes:

[0111] S610. The text data, the text information corresponding to the image data, and the prompt information are fused together to obtain the target prompt information.

[0112] This application does not limit the text information corresponding to the image data; for example, it can be at least one of the image's address information, name information, or size information. Please refer to... Figure 7The diagram illustrates a sample embedding feature extraction method according to an embodiment of this application. The content corresponding to the "Image" field is the text information corresponding to the image data, which reads "It's an operation. It's so exciting, don't miss a second." The content input to the word segmenter is all target prompt information. All information other than the "text information corresponding to the image data" and "text data" explained earlier is prompt information, i.e., the prompt mentioned earlier.

[0113] S620. Perform word segmentation on the target prompt information to obtain a word sequence.

[0114] Word segmentation is a fundamental task in natural language processing, aiming to divide continuous text information into individual word units. This application does not limit the word segmentation method; for example, rule-based segmentation, statistical segmentation, or deep learning-based segmentation methods can be used. Please refer to... Figure 7 A word segmenter can be used for word segmentation. This application does not limit the word segmenter used; for example, the InternLM2-chat-7b word segmenter can be used. The InternLM2-chat-7b word segmenter is a deep learning-based word segmentation tool that employs an advanced neural network model, enabling accurate and efficient processing of Chinese text word segmentation tasks. This word segmenter can efficiently process Chinese text, accurately dividing continuous text information into independent word units. Using the InternLM2-chat-7b word segmenter ensures the accuracy and stability of the segmentation results, providing strong support for subsequent text processing tasks.

[0115] S630. Extract features from the word sequence to obtain a text feature sequence.

[0116] Feature extraction can be based on various methods and techniques, including but not limited to bag-of-words models, word embedding models, or more complex deep learning models. In this embodiment, the specific method of feature extraction is not limited; the key is to transform the word sequence obtained after word segmentation into a feature sequence that reflects the key information and semantic features of the text. These feature sequences will serve as the basis for subsequent model training or analysis, helping to improve the model's performance and accuracy. Effective feature extraction from word sequences can provide richer and more valuable information for subsequent text processing and analysis tasks.

[0117] Please refer to Figure 7Feature extraction can be performed using word embeddings, which can be implemented using BERT or FastText encoding models. BERT stands for Bidirectional Encoder Representations from Transformers. It is a pre-trained language representation model that captures rich linguistic features, including semantics, syntax, and contextual information, through unsupervised learning on large amounts of text data. FastText performs exceptionally well when processing large-scale text data. By selecting a suitable model, word embeddings can generate high-quality word vectors.

[0118] S640. Visual features are extracted from the image data to obtain visual features.

[0119] By extracting visual features from image data, key information such as color, texture, and shape can be captured. This application does not limit the method of visual feature extraction; convolutional neural networks, residual networks, or other image recognition techniques can be used to perform visual feature extraction. Please refer to... Figure 7 VitBlock vision processors can be used for visual feature extraction. VitBlock is a hardware accelerator specifically designed for image processing and visual feature extraction. It employs advanced algorithms and architecture to efficiently process large-scale image data and extract key visual features from images. Using VitBlock can significantly improve the speed and accuracy of visual feature extraction, thereby optimizing the entire feature extraction model training process.

[0120] S650. Input the text feature sequence and the visual features into the multimodal large-scale generative language model to obtain the output result; extract the information in the output result corresponding to the preset information carrier.

[0121] Please refer to Figure 7 The target prompt message contains ' <embed> This string will directly correspond to a unique token in the token segmenter. When the MLLM outputs the final output, the latent vector of the last layer corresponding to this token will be taken out as the information corresponding to the token, that is, the information corresponding to the preset information carrier. This information is the comprehensive single embedding feature of the input image data and text data.

[0122] This application does not limit the architecture of MLLM; please refer to [the relevant documentation]. Figure 7MLLM can be built upon the Transformer. By jointly training the MLLM using contrastive learning, classification loss, and label prediction loss, a feature extraction model is obtained. This feature extraction model can include... Figure 7 The various components within.

[0123] During training, the embedding features output by MLLM can be represented by embeddings to capture the rich semantic meaning of multimodal information. Before inputting into MLLM, word segmentation and preliminary word feature extraction can be performed to map each word to vector tokens in a multidimensional space, facilitating MLLM's learning of complex relationships between words. Information from different modalities is usually transformed into vectors using different models; for example, images can be segmented and preliminarily feature extracted using the VitBlock model. However, text vector tokens and image features cannot be directly combined because they are learned and formed in different vector spaces, and their understanding of things differs. Related technologies do not have an ideal solution for aligning these differences. This application's embodiment is based on the multimodal large language model MLLM, fully utilizing the rich knowledge and powerful logical reasoning and multimodal content understanding capabilities contained in the multimodal large language model. It further trains the multimodal MLLM model through contrastive learning and predictive learning, thereby using MLLM training to solve the alignment difficulty problem and obtain high-quality single embedding features.

[0124] The following are embodiments of the apparatus of this application, which can be used to execute embodiments of the method of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method of this application.

[0125] Please refer to Figure 8 This diagram illustrates a block diagram of a feature extraction apparatus according to an embodiment of this application. The apparatus has the function of implementing the above-described feature extraction method; this function can be implemented in hardware or by hardware executing corresponding software. The apparatus can be a computer device or can be installed within a computer device. The apparatus may include:

[0126] The business data acquisition module 810 is used to acquire business data, which includes data in at least two modalities, and the contents of the at least two modalities are related.

[0127] The prompt information acquisition module 820 is used to acquire prompt information, which includes a single target unit. The prompt information is used to instruct the feature extraction model to output the embedded features corresponding to the business data to a preset information carrier. The preset information carrier is the information carrier corresponding to the word in the word sequence that corresponds to the target unit. The word sequence is obtained by segmenting the prompt information.

[0128] Feature extraction module 830 is used to trigger the feature extraction model to output the embedded features corresponding to the business data based on the business data and the prompt information;

[0129] The feature extraction model is a model obtained by training a multimodal large-scale generative language model based on contrastive learning.

[0130] In one exemplary implementation, the embedded feature is a latent vector for the target unit output by the last network layer in the multimodal large generative language model.

[0131] In one exemplary embodiment, the method further includes:

[0132] Obtain the preset dimension corresponding to the business data;

[0133] The embedded features are input into a multilayer perceptron to obtain target embedded features with the preset dimensions.

[0134] In one exemplary implementation, the data of at least two modalities includes text data and image data. The step of triggering the feature extraction model to output embedded features corresponding to the business data based on the business data and the prompt information includes:

[0135] The target prompt information is obtained by fusing the text data, the text information corresponding to the image data, and the prompt information.

[0136] The target prompt information is segmented into words to obtain a word sequence;

[0137] Feature extraction is performed on the word sequence to obtain the text feature sequence;

[0138] Visual features are extracted from the image data to obtain visual features;

[0139] The text feature sequence and the visual features are input into the feature extraction model to obtain the output result;

[0140] Extract the information corresponding to the preset information carrier from the output results to obtain the embedded features corresponding to the business data.

[0141] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0142] Please refer to Figure 9 This diagram illustrates a block diagram of a feature extraction model training apparatus according to an embodiment of this application. The apparatus has the function of implementing the aforementioned feature extraction model training method; this function can be implemented in hardware or by hardware executing corresponding software. The apparatus can be a computer device or can be installed within a computer device. The apparatus may include:

[0143] The sample acquisition module 910 is used to acquire multiple samples, each sample including data from at least two modalities, and the contents of the data from the at least two modalities are related.

[0144] Training module 920 is used to perform the following operations:

[0145] The prompt information includes a single target unit. The prompt information is used to instruct the multimodal large-scale generative language model to output the sample embedding features corresponding to the sample to a preset information carrier. The preset information carrier is the information carrier corresponding to the word in the word sequence that corresponds to the target unit. The word sequence is obtained by segmenting the prompt information.

[0146] Extract the sample embedding features corresponding to each sample, wherein the sample embedding features are the information corresponding to the preset information carrier in the output result generated by the multimodal large-scale generative language model based on the sample and the prompt information;

[0147] Based on each sample and its corresponding sample embedding features, the parameters of the multimodal large-scale generative language model are adjusted through contrastive learning to obtain a feature extraction model.

[0148] In one exemplary implementation, each sample corresponds to at least one level of category information, and the prompt information is further used to instruct the multimodal large-scale generative language model to output at least one level of predicted classification based on the sample.

[0149] The training module 920 is used to perform the following operations:

[0150] Based on at least one level of category information and at least one level of predicted classification for each sample, determine the corresponding classification loss;

[0151] The parameters of the multimodal large-scale generative language model are adjusted based on the classification loss and the contrastive learning loss, whereby the contrastive learning loss is the loss generated during the contrastive learning process.

[0152] In one exemplary implementation, each sample corresponds to at least one label, and the prompt information is further used to instruct the multimodal large-scale generative language model to output at least one labeling result based on the sample.

[0153] The training module 920 is used to perform the following operations:

[0154] Based on at least one label and at least one labeling result for each sample, determine the corresponding label prediction loss;

[0155] The parameters of the multimodal large-scale generative language model are adjusted based on the label prediction loss and the contrastive learning loss, where the contrastive learning loss is the loss generated during the contrastive learning process.

[0156] In one exemplary embodiment, the sample acquisition module 910 is configured to perform the following operations:

[0157] Acquire multiple multimodal media contents;

[0158] Each piece of multimodal media content is augmented twice to obtain two corresponding samples;

[0159] The step of adjusting the parameters of the multimodal large-scale generative language model through contrastive learning based on each sample and its corresponding sample embedding features includes:

[0160] Samples that correspond to the same multimodal media content are identified as positive sample pairs, and samples that do not correspond to the same multimodal media content are identified as negative sample pairs.

[0161] Based on the sample embedding features corresponding to the positive sample pairs and the sample embedding features corresponding to the negative sample pairs, the contrastive learning loss is obtained;

[0162] The parameters of the multimodal large-scale generative language model are adjusted based on the contrastive learning loss.

[0163] In one exemplary embodiment, the media data of the at least two modalities includes text data and image data, and the training module 920 is configured to perform the following operations:

[0164] The target prompt information is obtained by fusing the text data, the text information corresponding to the image data, and the prompt information.

[0165] The target prompt information is segmented into words to obtain a word sequence;

[0166] Feature extraction is performed on the word sequence to obtain the text feature sequence;

[0167] Visual features are extracted from the image data to obtain visual features;

[0168] The text feature sequence and the visual features are input into the multimodal large-scale generative language model to obtain the output result;

[0169] Extract the information corresponding to the preset information carrier from the output result.

[0170] In one exemplary implementation, the sample embedding feature is a latent vector for the target unit output by the last network layer in the multimodal large generative language model.

[0171] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0172] Please refer to Figure 10 This diagram illustrates a structural block diagram of a computer device provided in one embodiment of this application. The computer device may be a server for executing the aforementioned feature extraction method or feature extraction model training method. Specifically:

[0173] Computer device 1000 includes a central processing unit (CPU) 1001, a system memory 1004 including random access memory (RAM) 1002 and read-only memory (ROM) 1003, and a system bus 1005 connecting the system memory 1004 and the CPU 1001. Computer device 1000 also includes a basic input / output system (I / O system) 1006 to facilitate information transfer between various devices within the computer, and a mass storage device 1007 for storing the operating system 1013, application programs 1014, and other program modules 1015.

[0174] The basic input / output system 1006 includes a display 1008 for displaying information and an input device 1009 for user input, such as a mouse or keyboard. Both the display 1008 and the input device 1009 are connected to the central processing unit 1001 via an input / output controller 1010 connected to the system bus 1005. The basic input / output system 1006 may also include the input / output controller 1010 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1010 also provides output to a display screen, printer, or other types of output devices.

[0175] Mass storage device 1007 is connected to central processing unit 1001 via a mass storage controller (not shown) connected to system bus 1005. Mass storage device 1007 and its associated computer-readable media provide non-volatile storage for computer device 1000. That is, mass storage device 1007 may include computer-readable media (not shown) such as hard disk or CD-ROM (Compact Disc Read-Only Memory) drive.

[0176] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 1004 and mass storage device 1007 described above can be collectively referred to as memory.

[0177] According to various embodiments of this application, the computer device 1000 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1000 can be connected to the network 1012 via the network interface unit 1011 connected to the system bus 1005, or the network interface unit 1011 can be used to connect to other types of networks or remote computer systems (not shown).

[0178] The aforementioned memory also includes a computer program stored in the memory and configured to be executed by one or more processors to implement the aforementioned feature extraction model training method.

[0179] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, at least one program, code set, or instruction set is executed by a processor to implement the feature extraction method or feature extraction model training method.

[0180] Specifically, the feature extraction method includes:

[0181] Acquire business data, which includes data from at least two modalities, and the content of the data from the at least two modalities is related.

[0182] Obtain prompt information, the prompt information including a single target unit, the prompt information being used to instruct the feature extraction model to output the embedded features corresponding to the business data to a preset information carrier, the preset information carrier being the information carrier corresponding to the word in the word sequence corresponding to the target unit, the word sequence being obtained by word segmentation of the prompt information;

[0183] Based on the business data and the prompt information, the feature extraction model is triggered to output the embedded features corresponding to the business data;

[0184] The feature extraction model is a model obtained by training a multimodal large-scale generative language model based on contrastive learning.

[0185] In one exemplary implementation, the embedded feature is a latent vector for the target unit output by the last network layer in the multimodal large generative language model.

[0186] In one exemplary embodiment, the method further includes:

[0187] Obtain the preset dimension corresponding to the business data;

[0188] The embedded features are input into a multilayer perceptron to obtain target embedded features with the preset dimensions.

[0189] In one exemplary implementation, the data of at least two modalities includes text data and image data. The step of triggering the feature extraction model to output embedded features corresponding to the business data based on the business data and the prompt information includes:

[0190] The target prompt information is obtained by fusing the text data, the text information corresponding to the image data, and the prompt information.

[0191] The target prompt information is segmented into words to obtain a word sequence;

[0192] Feature extraction is performed on the word sequence to obtain the text feature sequence;

[0193] Visual features are extracted from the image data to obtain visual features;

[0194] The text feature sequence and the visual features are input into the feature extraction model to obtain the output result;

[0195] Extract the information corresponding to the preset information carrier from the output results to obtain the embedded features corresponding to the business data.

[0196] Specifically, the training method for this feature extraction model includes:

[0197] Multiple samples are acquired, each sample including data from at least two modalities, and the contents of the data from the at least two modalities are related.

[0198] The prompt information includes a single target unit. The prompt information is used to instruct the multimodal large-scale generative language model to output the sample embedding features corresponding to the sample to a preset information carrier. The preset information carrier is the information carrier corresponding to the word in the word sequence that corresponds to the target unit. The word sequence is obtained by segmenting the prompt information.

[0199] Extract the sample embedding features corresponding to each sample, wherein the sample embedding features are the information corresponding to the preset information carrier in the output result generated by the multimodal large-scale generative language model based on the sample and the prompt information;

[0200] Based on each sample and its corresponding sample embedding features, the parameters of the multimodal large-scale generative language model are adjusted through contrastive learning to obtain a feature extraction model.

[0201] In one exemplary implementation, each sample corresponds to at least one level of category information, and the prompt information is further used to instruct the multimodal large-scale generative language model to output at least one level of predicted classification based on the sample.

[0202] The step of adjusting the parameters of the multimodal large-scale generative language model through contrastive learning based on each sample and its corresponding sample embedding features includes:

[0203] Based on at least one level of category information and at least one level of predicted classification for each sample, determine the corresponding classification loss;

[0204] The parameters of the multimodal large-scale generative language model are adjusted based on the classification loss and the contrastive learning loss, whereby the contrastive learning loss is the loss generated during the contrastive learning process.

[0205] In one exemplary implementation, each sample corresponds to at least one label, and the prompt information is further used to instruct the multimodal large-scale generative language model to output at least one labeling result based on the sample.

[0206] The step of adjusting the parameters of the multimodal large-scale generative language model through contrastive learning based on each sample and its corresponding sample embedding features includes:

[0207] Based on at least one label and at least one labeling result for each sample, determine the corresponding label prediction loss;

[0208] The parameters of the multimodal large-scale generative language model are adjusted based on the label prediction loss and the contrastive learning loss, where the contrastive learning loss is the loss generated during the contrastive learning process.

[0209] In one exemplary implementation, acquiring multiple samples includes:

[0210] Acquire multiple multimodal media contents;

[0211] Each piece of multimodal media content is augmented twice to obtain two corresponding samples;

[0212] The step of adjusting the parameters of the multimodal large-scale generative language model through contrastive learning based on each sample and its corresponding sample embedding features includes:

[0213] Samples that correspond to the same multimodal media content are identified as positive sample pairs, and samples that do not correspond to the same multimodal media content are identified as negative sample pairs.

[0214] Based on the sample embedding features corresponding to the positive sample pairs and the sample embedding features corresponding to the negative sample pairs, the contrastive learning loss is obtained;

[0215] The parameters of the multimodal large-scale generative language model are adjusted based on the contrastive learning loss.

[0216] In one exemplary implementation, the media data of at least two modalities includes text data and image data, and the extraction of sample embedding features corresponding to each sample includes:

[0217] The target prompt information is obtained by fusing the text data, the text information corresponding to the image data, and the prompt information.

[0218] The target prompt information is segmented into words to obtain a word sequence;

[0219] Feature extraction is performed on the word sequence to obtain the text feature sequence;

[0220] Visual features are extracted from the image data to obtain visual features;

[0221] The text feature sequence and the visual features are input into the multimodal large-scale generative language model to obtain the output result;

[0222] Extract the information corresponding to the preset information carrier from the output result.

[0223] In one exemplary implementation, the sample embedding feature is a latent vector for the target unit output by the last network layer in the multimodal large generative language model.

[0224] Optionally, the computer-readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0225] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned feature extraction model training method.

[0226] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0227] In addition, in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0228] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims

1. A feature extraction method, characterized in that, The method includes: Acquire business data, which includes data from at least two modalities, and the content of the data from the at least two modalities is related. Obtain prompt information, the prompt information including a single target unit, the prompt information being used to instruct the feature extraction model to output the embedded features corresponding to the business data to a preset information carrier, the preset information carrier being the information carrier corresponding to the word in the word sequence corresponding to the target unit, the word sequence being obtained by word segmentation of the prompt information; Based on the business data and the prompt information, the feature extraction model is triggered to output the embedded features corresponding to the business data; The feature extraction model is a model obtained by training a multimodal large-scale generative language model based on contrastive learning.

2. The method according to claim 1, characterized in that, The embedded feature is the latent vector for the target unit output by the last network layer in the multimodal large generative language model.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Obtain the preset dimension corresponding to the business data; The embedded features are input into a multilayer perceptron to obtain target embedded features with the preset dimensions.

4. The method according to claim 1, characterized in that, The data of at least two modalities includes text data and image data. The step of triggering the feature extraction model to output the embedded features corresponding to the business data based on the business data and the prompt information includes: The target prompt information is obtained by fusing the text data, the text information corresponding to the image data, and the prompt information. The target prompt information is segmented into words to obtain a word sequence; Feature extraction is performed on the word sequence to obtain the text feature sequence; Visual features are extracted from the image data to obtain visual features; The text feature sequence and the visual features are input into the feature extraction model to obtain the output result; Extract the information corresponding to the preset information carrier from the output results to obtain the embedded features corresponding to the business data.

5. A method for training a feature extraction model, characterized in that, The method includes: Multiple samples are acquired, each sample including data from at least two modalities, and the contents of the data from the at least two modalities are related. The prompt information includes a single target unit. The prompt information is used to instruct the multimodal large-scale generative language model to output the sample embedding features corresponding to the sample to a preset information carrier. The preset information carrier is the information carrier corresponding to the word in the word sequence that corresponds to the target unit. The word sequence is obtained by segmenting the prompt information. Extract the sample embedding features corresponding to each sample, wherein the sample embedding features are the information corresponding to the preset information carrier in the output result generated by the multimodal large-scale generative language model based on the sample and the prompt information; Based on each sample and its corresponding sample embedding features, the parameters of the multimodal large-scale generative language model are adjusted through contrastive learning to obtain a feature extraction model.

6. The method according to claim 5, characterized in that, Each of the samples corresponds to at least one level of category information, and the prompt information is further used to instruct the multimodal large-scale generative language model to output at least one level of predicted classification based on the sample. The step of adjusting the parameters of the multimodal large-scale generative language model through contrastive learning based on each sample and its corresponding sample embedding features includes: Based on at least one level of category information and at least one level of predicted classification for each sample, determine the corresponding classification loss; The parameters of the multimodal large-scale generative language model are adjusted based on the classification loss and the contrastive learning loss, whereby the contrastive learning loss is the loss generated during the contrastive learning process.

7. The method according to claim 5 or 5, characterized in that, Each of the samples corresponds to at least one label, and the prompt information is also used to instruct the multimodal large-scale generative language model to output at least one labeling result based on the sample. The step of adjusting the parameters of the multimodal large-scale generative language model through contrastive learning based on each sample and its corresponding sample embedding features includes: Based on at least one label and at least one labeling result for each sample, determine the corresponding label prediction loss; The parameters of the multimodal large-scale generative language model are adjusted based on the label prediction loss and the contrastive learning loss, where the contrastive learning loss is the loss generated during the contrastive learning process.

8. The method according to claim 5, characterized in that, The acquisition of multiple samples includes: Acquire multiple multimodal media contents; Each piece of multimodal media content is augmented twice to obtain two corresponding samples; The step of adjusting the parameters of the multimodal large-scale generative language model through contrastive learning based on each sample and its corresponding sample embedding features includes: Samples that correspond to the same multimodal media content are identified as positive sample pairs, and samples that do not correspond to the same multimodal media content are identified as negative sample pairs. Based on the sample embedding features corresponding to the positive sample pairs and the sample embedding features corresponding to the negative sample pairs, the contrastive learning loss is obtained; The parameters of the multimodal large-scale generative language model are adjusted based on the contrastive learning loss.

9. The method according to claim 5, characterized in that, The sample embedding feature is the latent vector for the target unit output by the last network layer in the multimodal large-scale generative language model.

10. A feature extraction device, characterized in that, The device includes: A business data acquisition module is used to acquire business data, which includes data in at least two modalities, and the contents of the at least two modalities are related. The prompt information acquisition module is used to acquire prompt information, which includes a single target unit. The prompt information is used to instruct the feature extraction model to output the embedded features corresponding to the business data to a preset information carrier. The preset information carrier is the information carrier corresponding to the word in the word sequence that corresponds to the target unit. The word sequence is obtained by segmenting the prompt information. The feature extraction module is used to trigger the feature extraction model to output the embedded features corresponding to the business data based on the business data and the prompt information. The feature extraction model is a model obtained by training a multimodal large-scale generative language model based on contrastive learning.

11. A feature extraction model training device, characterized in that, The device includes: A sample acquisition module is used to acquire multiple samples, each sample including data from at least two modalities, the contents of the data from the at least two modalities having a correlation relationship; The training module is used to perform the following operations: The prompt information includes a single target unit. The prompt information is used to instruct the multimodal large-scale generative language model to output the sample embedding features corresponding to the sample to a preset information carrier. The preset information carrier is the information carrier corresponding to the word in the word sequence that corresponds to the target unit. The word sequence is obtained by segmenting the prompt information. Extract the sample embedding features corresponding to each sample, wherein the sample embedding features are the information corresponding to the preset information carrier in the output result generated by the multimodal large-scale generative language model based on the sample and the prompt information; Based on each sample and its corresponding sample embedding features, the parameters of the multimodal large-scale generative language model are adjusted through contrastive learning to obtain a feature extraction model.

12. A computer device, characterized in that, The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the feature extraction method as described in any one of claims 1 to 4, or the feature extraction model training method as described in any one of claims 5 to 9.

13. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the feature extraction method as described in any one of claims 1 to 4, or the feature extraction model training method as described in any one of claims 5 to 9.

14. A computer program product, characterized in that, The computer program product includes computer instructions, the processor of the computer device reads the computer instructions, and the processor of the computer device executes the computer instructions to implement the feature extraction method as described in any one of claims 1 to 4, or the feature extraction model training method as described in any one of claims 5 to 9.