Multimedia processing method, system, device, equipment, medium and product

Abstract

The embodiment of the invention provides a multimedia processing method, system and device, a model training method, system and device, equipment, a medium and a product. The method comprises the following steps: acquiring multimedia and a text related to the multimedia; performing feature processing on the multimedia and the text to obtain multimedia features and text features; performing semantic information extraction on the text features by utilizing the learnable latent variables to obtain semantic information; performing enhancement processing on the multimedia features by using the semantic information to obtain a text guidance visual identifier; and inputting the text guidance visual identification into the large language model, and generating a response result related to multimedia. Semantic information in the text is extracted and used as guidance information to perform enhancement processing on original multimedia features. According to the method, the problem that the coding process between different modes of a traditional multi-mode model is isolated is solved, meanwhile, the method better fits the thinking logic of human in reality, images are observed with problems and purposes, and then the multi-mode model outputs response results better meeting text requirements proposed by users.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to multimedia processing, model training methods, systems, devices, equipment, media and products. Background Art

[0002] With the development of multimodal model application technology, multimodal technology is used in more and more scenarios. For example, in image and text processing scenarios, multimodal model technology is used to improve image and text processing capabilities.

[0003] In the prior art, in order to improve the image and text processing capabilities of multimodal models, researchers have focused on improving the models of visual speech models, for example, optimizing the connectors and / or language models in the model, and have achieved certain improvements. However, the improvements achieved are still limited. When processing images and text, the combination between the two is not close enough, and it is impossible to analyze images and texts in the same way that humans process thoughts. Therefore, there is a need for a solution that can improve the multimedia processing capabilities of multimodal models. Summary of the invention

[0004] In order to solve or improve the problems existing in the prior art, the embodiments of the present application provide multimedia processing, model training methods, systems, devices, equipment, media and products.

[0005] In a first aspect, in one embodiment of the present application, a multimedia processing method is provided. Applied to a multimodal model, the method comprises:

[0006] Retrieving multimedia and text associated with the multimedia;

[0007] Performing feature processing on the multimedia and the text to obtain multimedia features and text features;

[0008] Extracting semantic information from the text features using learnable latent variables to obtain semantic information;

[0009] Using the semantic information to enhance the multimedia features, obtaining a text-guided visual identifier;

[0010] The text-guided visual identifier is input into a large language model to generate a response result related to the multimedia.

[0011] In a second aspect, in one embodiment of the present application, a model training method is provided. The method comprises:

[0012] Constructing multimodal training samples; wherein the multimodal training samples include multimedia training samples and text training samples related to the multimedia training samples;

[0013] While keeping the weights of the visual encoder and the large language model unchanged, the projector and the global mask module are trained using the multimodal training samples to obtain a first training result and a text-guided global mask training result; and

[0014] Using the first training result output by the projector training to train the detail sensor, and obtaining a text-guided detail perception token training result;

[0015] The text-guided global mask training result and the text-guided detail-aware token training result are fused into the large language model for processing to obtain a multimodal output result, so as to optimize the projector, the global mask module and the detail sensor according to the multimodal output result.

[0016] In a third aspect, in one embodiment of the present application, a multimedia processing system is provided, the system comprising:

[0017] A multimedia encoder is used to encode multimedia to obtain multimedia features;

[0018] A text encoder is used to encode the text to obtain text features;

[0019] A projector, configured to perform conversion processing on the multimedia features to obtain multimedia embedding tokens aligned with a large language model;

[0020] A global processing module, used for performing global enhancement processing on the multimedia features using the text features to obtain a text-guided global mask;

[0021] A detail processing module, used to perform detail enhancement processing on the multimedia features using the text features to obtain text-guided detail-aware tokens;

[0022] The large language model is used to input the text-guided global mask and the text-guided detail-aware token into the large language model to generate a response result related to the multimedia.

[0023] In a fourth aspect, in one embodiment of the present application, a multimedia processing device is provided, the device comprising:

[0024] The acquisition module is used to acquire multimedia and text related to the multimedia.

[0025] The feature processing module is used to perform feature processing on the multimedia and the text to obtain multimedia features and text features.

[0026] The extraction module is used to extract semantic information from the text features using learnable latent variables to obtain semantic information.

[0027] The enhancement processing module is used to enhance the multimedia features by using the semantic information to obtain a text-guided visual identifier.

[0028] A generating module is used to input the text-guided visual identifier into a large language model to generate a response result related to the multimedia.

[0029] In a fifth aspect, in one embodiment of the present application, an electronic device is provided, including a memory and a processor; wherein:

[0030] The memory is used to store programs;

[0031] The processor is coupled to the memory, and is used to execute the program stored in the memory to implement the method described in the first aspect.

[0032] In the sixth aspect, in one embodiment of the present application, a non-temporary machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the first aspect.

[0033] In a seventh aspect, in one embodiment of the present application, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the method described in the first aspect.

[0034] The technical solution provided by the embodiment of the present application is applied to a multimodal model, specifically a multimodal model capable of image and text processing. Assuming that the multimodal model for image and text processing has been pre-trained, the multimedia provided by the user or actively received, as well as the text related to the multimedia, can be obtained. Feature extraction is performed on the multimedia and text respectively to obtain multimedia features and text features. Further, the multimedia features are enhanced using the text features. Specifically as follows: the multimedia features are globally enhanced using the text features to obtain a global mask guided by the text, thereby realizing global attention to the multimedia with the help of the text; and the multimedia features are enhanced using the text features to obtain a detail perception token guided by the text, thereby realizing detail perception of the multimedia with the help of the text. Here, by establishing an interactive relationship between the text features and the multimedia features, the multimodal model can pay attention to the details and the global aspects of the multimedia after analyzing the text like a human being, thereby being able to obtain a more accurate model output result. After the obtained global mask guided by the text and the detail perception token guided by the text are input into the large language model, an accurate response result related to the multimedia is obtained. Through the above method, the semantic information in the text is refined and used as guiding information to enhance the original multimedia features. This overcomes the dilemma of isolated encoding processes between different modes in traditional multimodal models, and is more in line with the thinking logic of humans in reality, observing objects with questions and purposes, thereby making the multimodal model output a response result that is more in line with the text requirements put forward by users. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 A flowchart of a multimedia processing method provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of a flow chart of an enhanced processing method proposed in an embodiment of the present application;

[0038] Figure 3 A flowchart of a model training method provided in an embodiment of the present application;

[0039] Figure 4 A schematic diagram of the structure of a multimedia processing system provided in an embodiment of the present application;

[0040] Figure 5A schematic diagram of the processing flow of the global processing module provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of the processing flow of the detail processing module provided in the embodiment of the present application;

[0042] Figure 7 A schematic diagram of a multimedia processing device provided in an embodiment of the present application;

[0043] Figure 8 A schematic diagram of a model training device provided in an embodiment of the present application;

[0044] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0046] In some processes described in the specification, claims and the above-mentioned figures of the present application, multiple operations appearing in a specific order are included, and these operations may not be executed or executed in parallel in the order in which they appear in this article. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit "first" and "second" to different types. In addition, the embodiments described below are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0047] With the rise of artificial intelligence applications, more and more fields are empowered by artificial intelligence technology. In addition, different artificial intelligence models have been developed for different fields. For example, in order to realize the simultaneous processing of images and texts, a visual language model (Visual Language Models, VLMs) was proposed, which is a multimodal model. With the popularization of visual language model applications, more researchers have begun to try various optimization schemes, such as optimizing connectors, enhancing speech model components, and so on. However, although these optimization schemes can bring about certain improvements in model processing capabilities and processing effects, they still cannot give full play to the visual language model's ability to process images and text. Therefore, this application proposes a solution to optimize the visual language model using text-guided visual encoders.

[0048] Terminology explanation:

[0049] Multimodal model: A multimodal model refers to a deep learning model that effectively integrates data of different modalities (such as images, text, sound, etc.) to achieve more efficient information response and intelligent recognition. It can make full use of the complementary information between various modalities to improve the generalization ability and accuracy of the model. The English expression of multimodal model is "MultimodalModels". It can fuse and connect data from different perceptual modalities to achieve more comprehensive and accurate information understanding and reasoning capabilities. The model needs to have cross-modal feature extraction, fusion and reasoning capabilities, which puts higher requirements on the design and training of the model. Multimodal models can handle complex and diverse data, including various types of data such as text, images, audio and video.

[0050] Visual Language Models (VLMs): are artificial intelligence models that can process and understand both visual (image) and language (text) modal information. These models combine computer vision and natural language processing techniques, enabling them to perform well in a variety of complex tasks, such as visual question answering, image description generation, and text-to-image search.

[0051] TG-LLaVA: Text-guided LLaVA refers to the technology and method of using text information to guide or control the work of LLaVA (which may refer to a specific large language model or an AI system with similar functions). The "text guidance" here emphasizes the key role of text data in the model's working process, that is, text is not only the input of the model, but also an important signal to guide and control the model's behavior.

[0052] Embedding: Embedding is a vectorized mathematical representation of data such as text and images. It has a wide range of applications in the fields of artificial intelligence and machine learning. It effectively captures the intrinsic characteristics and relationships of data by converting high-dimensional, discrete or symbolic data into low-dimensional continuous vector representations, and promotes the performance improvement of various machine learning models and the optimization of practical application effects.

[0053] Token: Translated as a tag, the smallest unit used by large language models to understand and generate text. In large language models, tokens refer to the smallest unit processed in text. These units can be words, subwords (units between words and characters, such as "unbelievable" may be divided into "un", "believ", "able"), characters, or other forms of text fragments. Tokenization is the process of dividing text into these basic units and is the cornerstone of LLMs in processing text data. Among them, the role of tokens includes understanding and generating language: tokens are the basic elements for LLMs to understand and generate language. The model captures the semantic and structural information of the language by processing these tokens, thereby achieving understanding and generation of the language. As a numerical identifier: Inside LLMs, each token is assigned a numerical value or identifier and is input or output from the model in the form of a sequence. These numerical identifiers are the actual representation used by the model when processing and generating text. Bridge role: Tokens serve as a bridge between the original text data and the digital representation that LLMs can use, helping the model ensure the coherence and consistency of the text, thereby effectively handling various tasks such as writing, translation, and answering queries.

[0054] In order to solve the above technical problems, the present application proposes a multimedia processing solution.

[0055] The technical solution implemented in this application will be explained below in conjunction with specific embodiments.

[0056] like Figure 1 The flowchart of the multimedia processing method provided in the embodiment of the present application is shown in FIG. It should be noted that the method is implemented based on the multimodal model technology. The server of the method can be running in a computing device such as a local computer or a cloud computer. Figure 1 You can see that the specific steps include the following:

[0057] 101: Acquire multimedia and text related to the multimedia.

[0058] 102: Perform feature processing on the multimedia and the text to obtain multimedia features and text features.

[0059] 103: Extracting semantic information from the text features using learnable latent variables to obtain semantic information.

[0060] 104: Enhance the multimedia features using the semantic information to obtain a text-guided visual identifier.

[0061] 105: Inputting the target multimedia features obtained after correction by the text-guided visual identifier into a large language model to generate a response result related to the multimedia.

[0062] The multimedia mentioned here is a combination of multiple media, which can be images (including static images, dynamic images), text, video and other media forms. The text related to multimedia mentioned here can be understood as the text content and image content contained in the multimedia provided by the user, or the text content and image content in the multimedia that the user hopes to process. On the contrary, if the information contained in the text is irrelevant to the multimedia, for example, assuming that the multimedia is a static image of a car, if the text is related to the car, the color of the car, and the image processing, then the text can be considered to be related to the multimedia; for another example, assuming that the multimedia is a static image of a car, if the text is about asking about disease treatment methods, flight information, etc., then the text can be considered to be irrelevant to the multimedia. In actual applications, users can directly provide text information, or users can provide voice and convert it into text.

[0063] In order to improve processing efficiency, text features and multimedia features can be used for correlation analysis and judgment. For example, in an e-commerce scenario, multiple pictures are displayed in the current interface at the same time, and the user only asks a question (that is, text). At this time, it is necessary to first determine the correlation between the question text and multiple pictures, and select the picture with the highest correlation as the multimedia to be processed this time. There are many ways to perform correlation analysis, such as using text features and multimedia features to calculate similarity, and selecting related images based on the size of the similarity. Of course, there may be multiple images.

[0064] After feature processing of multimedia and text, multimedia features and text features can be obtained. Then, the process of extracting semantic information from the text features using learnable latent variables can be understood as assuming that there is a set of learnable latent variables (Latent Variables), which are continuously optimized during the training process to capture specific information or patterns related to the task. These latent variables can be regarded as query vectors, which play the role of active inquiry in the cross-attention operation. In the cross-attention mechanism, text features play the role of Key and Value. Key is used to calculate the similarity with the query vector (i.e., learnable latent variable), while Value provides information weighted according to these similarities. Specifically, each query vector (latent variable) is compared with all Keys (text features) to calculate a weight distribution that reflects the importance of each text feature for the current query. Then, these weights are used to aggregate information from Value (also a text feature) to generate a representation that combines query-specific requirements (that is, when the latent variable is used as a query vector, it represents the user's query-specific requirements) with comprehensive text information. In this way, efficient information interaction is achieved between the learnable latent variables and text features, enabling the model to dynamically adjust its internal representation based on the rich information of the text, thereby showing stronger adaptability and accuracy when processing various scenario tasks such as product consultation, question-answering systems, sentiment analysis, etc.

[0065] After obtaining accurate and comprehensive semantic information, the semantic information can be used to enhance multimedia features. This process is to optimize multimedia features using text information. The optimized multimedia features can focus on the global information guided by the text information and better perceive the details in the original image. Therefore, when performing image processing, multimedia can be observed purposefully like humans (that is, the purpose expressed by the text).

[0066] After the above enhancement processing, the text guidance visual identifier is input into the large language model (LLM) in the multimodal model. The text guidance visual identifier here is aligned with the large language model, that is, the two have input vectors of the same dimension. The perspective identifier obtained after the multimedia feature correction is input into the large language model, so that a response result related to multimedia can be obtained, which can more accurately and comprehensively meet the response requirements for multimedia.

[0067] In one or more embodiments of the present application, the feature processing of the multimedia and the text to obtain multimedia features and text features includes: feature processing the multimedia to obtain the multimedia features including multimedia global features and / or visual detail tokens; feature processing the text to obtain the text features including text global features and / or text detail features.

[0068] When using text encoders for feature processing, there are many processing methods, such as using pre-trained word embeddings, deep learning models (such as BERT, GPT, etc.), convolutional neural networks (CNN), or a combination of multiple methods.

[0069] For example, when using pre-trained word embeddings for feature extraction, each word in the text can be converted into a vector of fixed dimension using pre-trained word embedding models such as Word2Vec, GloVe, or FastText. After that, global features of the text can be extracted through pooling techniques (such as average pooling and maximum pooling). These pooling operations aggregate the information of all word vectors in the text into a single vector, which can be regarded as the global features of the text. When extracting detailed features of the text (detailed features usually refer to specific information of each part of the text (such as words, phrases, or sentences)), after using pre-trained word embeddings, each word has its own unique vector representation, which is the detailed feature itself. If more fine-grained features (such as phrase or sentence level) are required, consider using phrase embedding or sentence embedding models (such as BERT's [CLS] tag output as sentence representation).

[0070] When using deep learning models (such as BERT, GPT, etc.) for feature extraction, deep learning models such as BERT can process the entire text sequence and generate context-related representations (i.e., embeddings) for each word. For global features, the output of special tags in models such as BERT (such as BERT's [CLS] tag) can be used as the global representation of the entire text. This representation is obtained by synthesizing the encoding information of the entire text sequence. When extracting detailed features, models such as BERT generate context-related representations for each word (or sub-word unit), which are the detailed features of the text. In addition, features of different granularity can be obtained through different layers of the model. The shallow layer usually captures grammatical information, while the deep layer is more inclined to semantic information. Therefore, it is preferred to extract deep semantic information as the extracted text detail features.

[0071] Assuming that the multimedia mentioned here is a static image, there are many ways to process the features of the static image, such as using a convolutional neural network (CNN), an auto encoder, a deep Boltzmann machine (DBM), etc. The following uses a convolutional neural network (CNN) as an example for explanation.

[0072] First, the acquired static image is preprocessed, including grayscale processing, that is, converting the color image into a grayscale image to simplify subsequent processing and reduce the amount of calculation. This step is optional and depends on the needs of subsequent processing. Normalization processing adjusts the brightness and contrast of the image to make it have a uniform scale for subsequent processing.

[0073] After preprocessing, feature extraction is performed. The visual encoder extracts useful feature information from the image through a series of operations. These features can be global or local, depending on the design of the encoder and the task requirements.

[0074] In a convolutional neural network (CNN), multiple convolution kernels are used to perform convolution operations with images to extract local features of the image. As the network goes deeper, these features gradually evolve from low-level (such as edges, textures) to high-level (such as shapes, objects). Pooling operations (such as maximum pooling and average pooling) are used to reduce the dimension of the feature map, reduce the amount of calculation, and introduce a certain degree of translation invariance. In addition, activation functions such as ReLU (Rectified Linear Unit) can also be used to increase the nonlinearity of the network and improve the expressiveness of the model. By stacking multiple convolutional layers, pooling layers, and activation function layers, CNN can extract multi-level features of the image. In order to enable the model to learn the positional relationship between elements in the image, position information can be embedded in the feature map. This is usually achieved by using sine and cosine functions to generate position encodings.

[0075] In the deeper layers of CNN, global average pooling (GAP) or global max pooling (GMP) is usually used to extract global features of the entire feature map. These global features represent the overall content of the image.

[0076] At the same time, in some Transformer-based models, the feature map is divided into multiple patches, and each patch is regarded as a visual detail token. These tokens are converted to a uniform dimension through linear transformation and used as input to the Transformer encoder. The encoder interacts and updates these tokens through the self-attention mechanism to capture the dependencies between them. Ultimately, each token contains rich contextual information and detailed features.

[0077] It should be noted that when extracting features from text and multimedia, only global features (including image global features and text global features) or detail features, or visual detail tokens can be extracted. It is also possible to extract global features (including image global features and text global features) and detail features, and visual detail tokens at the same time.

[0078] Through the above scheme, global feature information and detailed feature information can be fully and comprehensively extracted, so that in subsequent processing work, multimedia can be paid more attention to and processed more comprehensively, thereby improving the accuracy of response result output.

[0079] In one or more embodiments of the present application, Figure 2 The following is a flow chart of an enhanced processing method proposed in an embodiment of the present application. Figure 2 As can be seen in the figure, the multimedia features are enhanced by using the semantic information to obtain a text-guided visual identifier, including:

[0080] 201: Performing global enhancement processing on the multimedia feature using the first semantic information extracted based on the global feature of the text to obtain a text-guided global mask.

[0081] 202: Perform detail enhancement processing on the multimedia feature using the second semantic information extracted based on the text detail feature to obtain a text-guided detail-aware token.

[0082] 203: Concatenate the text-guided global mask and the text-guided detail-aware token to obtain a text-guided visual identifier.

[0083] The first semantic information mentioned here is the global mask of the text, and the second semantic information is the text detail guidance token. When performing enhancement processing, only global enhancement processing can be performed, or only detail enhancement processing can be performed. If only global enhancement processing is performed, the global mask guided by the text is used as the text guidance visual identifier. If only detail enhancement processing is performed, the detail perception token guided by the text is used as the text guidance visual identifier. Of course, if global enhancement processing and detail enhancement processing are performed at the same time, the global mask guided by the text and the detail perception token guided by the text need to be concatenated and processed into a vector that meets the input dimension requirements of the large language model.

[0084] The global enhancement process and the detail enhancement process will be described in detail below through specific embodiments, which will not be repeated here. For details, please refer to the following embodiments.

[0085] The enhancement process will be described below through specific embodiments.

[0086] When performing global enhancement processing, the use of learnable latent variables to extract semantic information from the text features to obtain semantic information includes: based on a cross-attention operation, using the first latent variable to extract semantic information from the global features of the text to obtain a global text mask as the first semantic information.

[0087] As described in step 201, the multimedia features are globally enhanced using the first semantic information extracted based on the text global features to obtain a text-guided global mask, including: 2011: using the text global mask to enhance (optimize) the multimedia global features to obtain the text-guided global mask.

[0088] In the present application, a text-guided visual feature optimization mask module (Text-Guided Feature Optimization Mask, TG-FOM) is proposed. The TG-FOM module uses learnable latent variables to parse the global information in the text, and appends the analysis results as a global mask to the multimedia features output by the visual encoder, thereby achieving the purpose of text-based optimization of multimedia features. The optimized features can contain the global information of the text.

[0089] The core of the cross-attention mechanism is to guide the flow and fusion of information by calculating the similarity between different data modalities (such as text and vision). In the TG-FOM module, cross-attention pays special attention to how to map key semantic information in the text to visual features to achieve feature optimization. Cross-attention shows higher flexibility and accuracy when processing multimodal data. Through the carefully designed learnable latent variables as queries, text as keys and values, cross-attention can achieve fine control of the complex interactive relationship between text and visual features, so that the model can learn the precise guidance of text on visual features.

[0090] As an optional solution, in order to further improve the expressiveness of the model, we can consider introducing a multi-head cross attention mechanism. Multi-head attention divides the input features into multiple subspaces, calculates the attention weights independently in each subspace, and then splices the results together to obtain a more comprehensive and rich feature representation. This mechanism helps capture more subtle and complex interactions between text and vision, further improving the performance of the model.

[0091] In the TG-FOM module, learnable latent variables play a vital role as learnable parameters. They are not only responsible for capturing the global semantic information in the text, but also effectively transferring this information to the visual features through the cross-attention mechanism. The introduction of learnable latent variables enables the model to adaptively learn the complex relationship between text and vision, thereby optimizing multimedia features.

[0092] The initialization of learnable latent variables has an important impact on the training effect and convergence speed of the model. In TG-FOM, latent variables are usually initialized by random initialization or based on the parameters of a pre-trained model. Random initialization can ensure that the model has sufficient exploration ability at the beginning of training, while initialization based on a pre-trained model can use existing knowledge to accelerate the convergence process of the model.

[0093] During the training process, the learnable latent variables are updated and optimized through the back-propagation algorithm. By calculating the gradient of the loss function with respect to the latent variables and iteratively updating them using optimizers (such as SGD, Adam, etc.), the learnable latent variables can gradually learn the key semantic information in the text and effectively transfer it to the visual features. In addition, in order to prevent the latent variables from overfitting, regularization techniques (such as L1 regularization, L2 regularization, or Dropout) can be introduced to constrain the update process of the learnable latent variables.

[0094] The overall workflow of the TG-FOM module is described below through a specific embodiment:

[0095] Initialize a set of learnable latent variables for subsequent semantic information extraction and feature optimization.

[0096] The first latent variable is used as a query to participate in the calculation of the cross attention mechanism together with the global features of the text. The attention weights are obtained by calculating the similarity between the text features and the visual features, and these weights are applied to the global visual features to obtain the global text mask as the first semantic information.

[0097] The global text mask is added to the features output by the original visual encoder through a zero-initialized linear layer (as a buffer layer). This process realizes the global enhancement of multimedia features, so that the optimized features can contain the global semantic information of the text.

[0098] The multimedia features processed by the TG-FOM module are output to the subsequent network layers for further processing or classification tasks.

[0099] It initializes a set of learnable latent variables and uses the cross-attention mechanism to extract the semantic information corresponding to the text instructions from the output of the text encoder. Then, in the form of a mask, the zero-initialized linear layer is used as a buffer to add the features output by the original visual encoder to achieve visual feature optimization (that is, the multimedia features mentioned above).

[0100] Through the above scheme, the multimedia features are optimized by using the global information of the text, so that the optimized mask inputted can pay good attention to the global multimedia.

[0101] When performing detail enhancement processing, the learnable latent variables are used to extract semantic information from the text features to obtain semantic information, including: in the first perception layer, based on the cross-attention operation, the second latent variables are used to extract semantic information from the text detail features to obtain a text detail guidance token as the second semantic information.

[0102] As described in step 202, the multimedia feature is enhanced in detail using the second semantic information extracted based on the text detail feature to obtain a text-guided detail-aware token, including: 2021: in the second perception layer, the multimedia feature is enhanced using the text detail guidance token to obtain a text-guided detail-aware token.

[0103] In the present application scheme, a text-guided detail perception module (Text-Guided DetailPerceiver, TG-DP) is also proposed. TG-DP uses another set of latent variables (that is, the second latent variables), first interacts with the detailed information in the text (that is, the text detail guidance token), and then extracts fine-grained details from the high-resolution patches of the static image (that is, the multimedia mentioned above) based on these detailed information. Another set of learnable second latent variables is initialized in the TG-DP module, the difference is that its number is much smaller than the number of first latent variables in the TG-FOM module. These second latent variables first interact with text features, and then interact with the features of the image with detail information at high-frequency resolution. The final result is used as a text-guided detail perception token.

[0104] The cross-attention mechanism is an important component in the Transformer model, which is particularly suitable for processing the interaction between multimodal data (such as text and images). In the text-image fusion task, the cross-attention allows the model to focus on the features of both text and image at the same time, thereby achieving deep semantic alignment between the two.

[0105] In the cross-attention layer, the second latent variable is taken as the query, while the text detail guidance tokens are taken as the key and value. This setup enables the model to retrieve and aggregate relevant information in the image based on the text query. The attention scores are obtained by calculating the similarity between the query and each key (usually using a dot product operation and applying a softmax function for normalization). These scores determine the importance of each image element for the current text query. The value is weighted summed using the attention scores to generate a weighted representation that combines the semantic information of the text and the image.

[0106] The TG-DP module contains two layers, which are described in detail below:

[0107] First-layer criss-cross attention: As mentioned above, the first-layer criss-cross attention layer uses the second latent variable as the query and the text-detail guidance tokens as the key and value. This step aims to extract key information from the text details and map them into the latent variable to form text-guided tokens.

[0108] Second layer cross attention: The second layer takes the text-guided detail-aware tokens as keys and values, and the text-instruction features parsed by the first layer as queries. This layer further strengthens the interaction between text and image details, so that image features are more finely enhanced under text guidance.

[0109] The second latent variable is a variable used in machine learning models to capture the potential relationships or hidden structures between data. In text-image fusion tasks, second latent variables are particularly important because they are able to represent complex semantic information that cannot be directly observed from the original data. The second latent variable converts high-dimensional, complex raw data into a low-dimensional, compact representation through a dimensionality reduction or encoding process while retaining key information. In addition, the jarring latent variable is able to capture the deep semantic structure of the data, such as themes, emotions, or abstract concepts, which are crucial for understanding and interpreting the data. It should be noted that during the training process, the second latent variable is part of the model parameters and can be optimized through the back-propagation algorithm to better capture the complex relationships between the data.

[0110] The TG-DP module initializes a set of learnable latent variables (i.e., second latent variables), the number of which is selected according to task requirements and design. In text-image fusion tasks, the number of latent variables is usually much smaller than the dimension of the input data to achieve effective information compression and representation. The second latent variable interacts with text and image features in the TG-DP module, extracts key information through the cross-attention mechanism, and generates text-guided detail-aware tokens. This process not only enhances the detail information of image features, but also achieves deep semantic alignment between text and image. In summary, cross-attention and latent variables play a key role in the TG-DP module, and through their effective combination, efficient fusion and deep semantic understanding between text and image are achieved.

[0111] In an optional solution, before using the text detail guidance token to enhance the multimedia features, it also includes: converting the multimedia features to obtain multimedia embedding tokens aligned with the large language model; and converting the multimedia embedding tokens using the text detail guidance token to obtain multimedia tokens.

[0112] Before performing detail enhancement processing on multimedia features, it is necessary to perform simple processing on the multimedia features through the projector, as follows:

[0113] The projector contains two linear layers with GELU activation functions. Its task is to convert the multimedia features output by the visual encoder into a form compatible with the language embedding tokens pre-trained by the large language model (LLM) so that the multimedia features can be mapped to the embedding space of the large language model. This step is to achieve the alignment of visual features with the language model and convert multimedia features into multimedia tokens.

[0114] Unify the feature representation of multimedia (such as objects and scenes in pictures and videos) and the feature representation of text information processed by language models (such as large language models LLM) into the same semantic space or representation framework. The purpose of this is to enable visual information and text information to be compared, integrated or interacted at the same level, thereby achieving cross-modal understanding and reasoning. For example, in the image description generation task, the model needs to understand the image content and convert it into a natural language description, which requires the alignment of visual features and language models so that the model can accurately convert visual information into text information.

[0115] From the above scheme, we can see that the original image is enlarged to retain more details and then segmented into blocks that match the size of the original image. This design ensures that all visual features can be extracted by calling the visual encoder only once. After obtaining the visual features of these blocks, position embedding and a learnable multi-layer perceptron (MLP) layer are added to restore the spatial structure information that was disrupted during the segmentation process, thereby obtaining the corrected visual detail tokens.

[0116] In one or more embodiments of the present application, inputting the text-guiding visual identifier into a large language model includes: performing word segmentation processing on the text to obtain a post-word segmentation text identifier; merging the text-guiding visual identifier and the post-word segmentation text identifier and inputting them into the large language model.

[0117] After obtaining the text-guided visual identifier through the above steps, it can be further merged with the post-segmentation text identifier. The post-segmentation text identifier is obtained after the text is segmented, and together with the text-guided visual identifier, it constitutes the input vector input to the large language model. When the large language model processes the vector, it can fully understand the meaning of the vector by combining the global and detailed information of the text guide, and then provide a response result that meets the user's text requirements.

[0118] Based on the same idea, the present application embodiment also proposes a model training method. The model is a multimodal model, such as Figure 3 A flow chart of a model training method provided in an embodiment of the present application. Figure 3 As can be seen in , the method specifically includes the following steps:

[0119] 301: Construct multimodal training samples; wherein the multimodal training samples include multimedia training samples and text training samples related to the multimedia training samples.

[0120] 302: While keeping the weights of the visual encoder and the large language model unchanged, use the multimodal training samples to train the projector and the global mask module to obtain a first training result and a text-guided global mask training result. And,

[0121] 303: Train a detail sensor using the first training result output by the projector training to obtain a text-guided detail-aware token training result.

[0122] 304: Fusing the text-guided global mask training result and the text-guided detail-aware token training result into the large language model for processing to obtain a multimodal output result, so as to optimize the projector, the global mask module and the detail sensor according to the multimodal output result.

[0123] First, prepare the data required for training. Multimodal datasets: Collect and organize multimodal datasets containing images and corresponding text descriptions, instructions, or question-answer pairs. These datasets should cover multiple languages ​​and different subject areas to support the generalization ability of the model. Perform necessary preprocessing on the collected images (such as scaling, cropping, normalization, etc.), tokenize the text, encode it into a format that the model can understand, and may include removing noise or outliers.

[0124] Then, modality alignment is performed. The goal of modality alignment is to align the embedding spaces of the visual encoder and LLM (large language model) so that they can understand and interact with each other. Keep the weights of the visual encoder and LLM unchanged, because these models are usually pre-trained and perform well on their respective unimodal tasks. Focus on optimizing the projector and modules such as TG-FOM and TG-DP. Use contrastive learning, triplet loss or other multimodal alignment techniques to train these modules to ensure that the visual and text embeddings are similar and consistent in the common space.

[0125] Based on modality alignment, the model's ability to follow instructions is further enhanced, that is, the model is able to generate corresponding multimodal outputs based on given instructions. While keeping the weights of the visual encoder and the main part of the LLM stable, continue to train the projector, TG-FOM, and TG-DP modules, paying special attention to how they work together to respond to instructions. Use a dataset containing clear instructions and expected outputs to train the model. These instructions can be in text form to guide the model to generate specific types of multimodal responses.

[0126] It should be noted that an effective fusion mechanism is designed inside the large language model LLM to combine visual embedding and text embedding in a meaningful way so that the model can comprehensively consider the information of the two modalities to generate output. In other words, the results of the enhanced processing by the TG-FOM and TG-DP modules are fused and input into the large language model LLM. Through training, the model learns how to process these fused embeddings in the context of LLM and generate the final output, such as picture description, knowledge questions and answers, etc.

[0127] Based on the same idea, the present application also proposes a multimedia processing system. Figure 4 A schematic diagram of the structure of a multimedia processing system provided in an embodiment of the present application. Figure 4 As can be seen in the figure, the system includes:

[0128] The multimedia encoder 41 is used to encode the multimedia to obtain multimedia features.

[0129] The text encoder 42 is used to encode the text to obtain text features.

[0130] The projector 43 is used to convert the multimedia features to obtain multimedia embedding tokens aligned with the large language model.

[0131] The global processing module 44 is used to perform global enhancement processing on the multimedia features using the text features to obtain a text-guided global mask.

[0132] The detail processing module 45 is used to perform detail enhancement processing on the multimedia features using the text features to obtain text-guided detail-aware tokens.

[0133] The large language model 46 is used to input the global mask of the text guidance and the detail-aware token of the text guidance into the large language model to generate a response result related to the multimedia.

[0134] Below, the technical implementation of several important modules in the system will be explained in detail through specific embodiments.

[0135] In the system described in the present application, the multimedia encoder Ev, usually using a pre-trained visual Transformer such as CLIP, is designed to transform the input image I∈R H×W×C It is divided into several blocks of equal size and further encoded into visual features F i ∈R N×D Where H and W represent the size of the input image, C represents the number of channels, N corresponds to the number of blocks in the output feature, and D represents the feature dimension of each encoding block. When the block size is P, N = HW / P 2 The large language model uses a tokenizer and text embedding module to sequentially convert text data into token IDs and their corresponding embedded tokenized text tokens Tt, effectively transforming language into the feature space of its input. In the VLM architecture, these tokenized text tokens Tt are concatenated with the aligned visual detail tokens Tv processed by the connector to form the input of the large language model for subsequent prediction. For a sequence of length L, the VLM predicts the target answer token Among them, the probability of i=1 can be formalized as:

[0136]

[0137] where θ represents all trainable parameters in the VLM. In this VLM prediction paradigm, visual features are directly encoded from the raw input image via Ev without any interaction with the text modality. This approach is in contrast to the purpose-driven nature of human behavior. Optimizing the encoded features based on text instructions is more conducive to enabling the VLM to generate accurate responses.

[0138] like Figure 5 A schematic diagram of the processing flow of the global processing module provided in an embodiment of the present application. Figure 5 The specific framework of the FOM module is shown. First, a set of learnable latent variables Lm (that is, the first latent variable in the previous text) with the same number of visual detail tokens is initialized. The purpose of these latent variables is to extract language information from text instructions and add it to the original features as a mask. Here, a single-layer Q-former is designed to parse the semantic information in the text instructions and act as a bridge between global text and visual features. In this structure, the cross-attention layer will pass through the pooled text instruction features encoded by the CLIP text encoder Et As Key and Value, it interacts with Query Lm, and the final output is a mask generated based on text information, which is then applied to the visual features. A zero-initialized linear layer is also introduced to ensure that the optimization of the original visual features remains a gradual process. This process can be formalized as:

[0139]

[0140] Where Mt represents the mask obtained by extracting semantic information from the text through the learnable latent variable Lm, Across and Aself represent the cross attention and self attention modules respectively, FFN represents the feedforward neural network, Z represents the zero-initialized linear layer used in the feature addition process, and It represents the visual features optimized by text guidance.

[0141] Text-Guided Detail Perceiver When observing an image, in addition to selecting the focus according to instructions, humans can also automatically adjust their attention to obtain more detailed information. Following this idea, TG-DP is designed, which is responsible for capturing details related to instructions. Figure 6 A schematic diagram of the processing flow of the detail processing module provided in the embodiment of the present application. Figure 6 As can be seen in the figure, the second latent variable Lh is set to the output of the text encoder Et Here, the number of Lh is much smaller than the number of original visual detail tokens, ensuring that the visual detail tokens input to the language model (LLM) will not increase significantly, thereby maintaining reasoning efficiency.

[0142] from Figure 6 As can be seen in Figure 1, the TG-DP module consists of two perception layers: the first perception layer is responsible for parsing fine-grained text to generate text guidance tokens. It receives the second latent variable Lh and the text detail guidance token The second latent variable Lh is used as the query, and the text details guide token As Key and Value, the difference is the text details guide token Connected to the second latent variable Lh. The second layer is responsible for generating detail-aware tokens under the guidance of fine-grained text. In the second layer, the key and value are replaced by the text-guided detail-aware tokens The second interaction is performed using the text instruction features parsed by the first layer as the query. The output of the second layer is the compressed multimedia token. Since multimedia tokens The feature space of is significantly different from the visual features of the original visual language model (VLM), so a dedicated projector C is designed h The whole process can be formalized as:

[0143]

[0144] Where Li represents the i-th layer in the DP module, F L1 is the output of the first layer, and CAT stands for the concatenation operation. Li (i∈(1,2)) represents the binding network, which can be formalized as:

[0145] BN(X)=X+(XW 2 up ·SiLu(XW 1 up ))W 3 dwon . (4)

[0146] So far, under the guidance of the input text, the optimized visual features Fi and the text-guided detail-aware tokens have been obtained. Next, the features obtained from the original VLM connector C are compared with Connect to form the final text-guided visual identity As the input of VLM. The final prediction process of VLM can be expressed as:

[0147]

[0148] In the above system, the proposed TG-LLaVA contains two text-guided modules: the text-guided feature optimization mask (TG-FOM) module and the text-guided detail perceptron (TG-DP) module. In the TG-FOM module, a set of learnable latent embeddings are used to analyze the input text from a global perspective, and then the analyzed language information is added to the image features as guidance through a zero-initialized linear layer. In the TG-DP module, a small number of learnable latent embeddings are used to parse the input text in detail, and then the parsed tokens are used as guidance to fuse information from a focused image perspective. Extensive experiments have demonstrated the effectiveness of the proposed design, showing significant improvements on multiple datasets and different frameworks without any additional data augmentation or complex enhancement means. The main technical effects are summarized as follows:

[0149] We propose TG-LLaVA, a text-guided architecture based on learnable latent embeddings, which is different from or even contrary to most existing VLM optimization methods, opening up a new and valuable research avenue. The proposed TG-FOM module and TG-DP module can be applied to mainstream VLM frameworks as generic plug-ins, consistently bringing improvements. Extensive experiments on various VLM variants in different settings and numerous multimodal tasks show that the proposed TG-LLaVA not only brings significant benefits, but also provides valuable insights and methods to the existing VLM research field.

[0150] Below, examples are given to illustrate the implementation of the technical solution of the present application in different application scenarios.

[0151] Application scenario 1 - e-commerce intelligent customer service:

[0152] For example, a user returns or exchanges a product (for example, if the color of the clothes is wrong after receiving the goods, the user can provide photos of the clothes, as well as a text description of the problem of the wrong color and the need to exchange the goods), and provides photos of the goods and text descriptions. This application solution, as an intelligent customer service, can identify the image information and text information provided by the user (input the text and image into the trained multimodal model to determine whether the text description is true), and if the description is true, the return and exchange process will be carried out. For another example, after receiving the product description text provided by the user, the e-commerce intelligent customer service can also provide product shopping guides for the user and recommend products to the user. For example, if the user says he wants a black and white striped T-shirt, the text provided by the user can be used to search for products that meet the user's needs from the images of the clothes in the server. This application solution can significantly improve the efficiency of the e-commerce platform.

[0153] Application scenario 2 – Intelligent secretary:

[0154] Users summarize documents or tables or extract specific information, and use file screenshots and instructions as input. This solution, as an intelligent secretary, can perform summaries, calculations, logical reasoning, etc. on the file screenshots according to the instructions, output the information the user expects, and improve office efficiency. For example, after the user provides a screenshot, he or she may ask to find the person with the highest score in the screenshot and extract the detailed personal information of the person.

[0155] Application scenario 3 - Intelligent teacher:

[0156] The user (for example, a student) takes a screenshot of a problem encountered in schoolwork and submits a text message requiring an intelligent teacher to explain the problem and the basic concepts. The screenshot of the problem and the text information are used as input. The application scheme, as an intelligent teacher, can provide answers and analysis to the corresponding questions, as well as explanations of basic concepts based on the input.

[0157] It should be noted that the application scheme of this application can be applied in many scenarios, including but not limited to common sense, mathematical reasoning, code writing, etc., to help users solve problems. The above scenarios are only used as examples and do not constitute a limitation on the technical solution of this application. In actual applications, users can apply it in any legal scenario according to their actual needs.

[0158] Based on the same idea, the embodiment of the present application also provides a multimedia processing device. Figure 7 A schematic diagram of a multimedia processing device provided in an embodiment of the present application. Figure 7 It can also be seen that the device includes:

[0159] The acquisition module 71 is used to acquire multimedia and text related to the multimedia.

[0160] The feature processing module 72 is used to perform feature processing on the multimedia and the text to obtain multimedia features and text features.

[0161] The extraction module 73 is used to extract semantic information from the text features using learnable latent variables to obtain semantic information.

[0162] The enhancement processing module 74 is used to enhance the multimedia features using the semantic information to obtain a text-guided visual identifier.

[0163] The generating module 75 is used to input the text-guided visual identifier into a large language model to generate a response result related to the multimedia.

[0164] A feature processing module 72, configured to perform feature processing on the multimedia to obtain the multimedia features including multimedia global features and / or visual detail tokens;

[0165] Feature processing is performed on the text to obtain the text features including text global features and / or text detail features.

[0166] An enhancement processing module 74, configured to perform global enhancement processing on the multimedia feature using the first semantic information extracted based on the global feature of the text, to obtain a global mask guided by the text;

[0167] Performing detail enhancement processing on the multimedia feature using the second semantic information extracted based on the text detail feature to obtain a text-guided detail-aware token;

[0168] The text-guided global mask and the text-guided detail-aware token are concatenated to obtain a text-guided visual identifier.

[0169] An enhancement processing module 74 is used to extract semantic information from the global features of the text using the first latent variable based on a cross attention operation to obtain a global mask of the text as the first semantic information;

[0170] The method of performing global enhancement processing on the multimedia features by using the first semantic information extracted based on the global features of the text to obtain a global mask guided by the text includes:

[0171] The multimedia global features are enhanced using the text global mask to obtain the text-guided global mask.

[0172] An enhancement processing module 74 is used to extract semantic information from text detail features using the second latent variable based on a cross attention operation in the first perception layer to obtain a text detail guidance token as the second semantic information;

[0173] The step of performing detail enhancement processing on the multimedia feature by using the second semantic information extracted based on the text detail feature to obtain a text-guided detail-aware token includes:

[0174] In the second perception layer, the multimedia features are enhanced using text detail guidance tokens to obtain text-guided detail perception tokens.

[0175] An enhancement processing module 74, configured to perform conversion processing on the multimedia features to obtain multimedia embedding tokens aligned with the large language model;

[0176] The multimedia embedded token is converted using the text detail guidance token to obtain a multimedia token.

[0177] A generating module 75 is used to perform word segmentation processing on the text to obtain a text mark after word segmentation;

[0178] The text-guided visual identifier and the post-word segmentation text identifier are combined and then input into the large language model.

[0179] Based on the same idea, the present application embodiment also provides a model training device. Figure 8 A schematic diagram of a model training device provided in an embodiment of the present application. Figure 8 It can also be seen that the device includes:

[0180] The construction module 81 is used to construct multimodal training samples; wherein the multimodal training samples include multimedia training samples and text training samples related to the multimedia training samples.

[0181] The first training module 82 is used to train the projector and the global mask module using the multimodal training samples while keeping the weights of the visual encoder and the large language model unchanged, to obtain a first training result and a text-guided global mask training result. And,

[0182] The second training module 83 is used to train the detail sensor using the first training result output by the projector training to obtain a text-guided detail perception token training result.

[0183] The third training module 84 is used to fuse the text-guided global mask training result and the text-guided detail-aware token training result into the large language model for processing to obtain a multimodal output result, so as to optimize the projector, the global mask module and the detail sensor according to the multimodal output result.

[0184] An embodiment of the present application also provides an electronic device. The electronic device is a master node electronic device in a computing unit. Fig. 9 The electronic device includes a memory 901, a processor 902 and a communication component 903;

[0185] The memory 901 is used to store programs;

[0186] The processor 902 is coupled to the memory and is configured to execute the program stored in the memory to:

[0187] Retrieving multimedia and text associated with the multimedia;

[0188] Performing feature processing on the multimedia and the text to obtain multimedia features and text features;

[0189] Extracting semantic information from the text features using learnable latent variables to obtain semantic information;

[0190] Using the semantic information to enhance the multimedia features, obtaining a text-guided visual identifier;

[0191] The text-guided visual identifier is input into a large language model to generate a response result related to the multimedia.

[0192] The processor 902 is used to perform feature processing on the multimedia to obtain the multimedia features including multimedia global features and / or visual detail tokens;

[0193] Feature processing is performed on the text to obtain the text features including text global features and / or text detail features.

[0194] The processor 902 is used to perform global enhancement processing on the multimedia feature by using the first semantic information extracted based on the global feature of the text to obtain a global mask guided by the text;

[0195] Performing detail enhancement processing on the multimedia feature using the second semantic information extracted based on the text detail feature to obtain a text-guided detail-aware token;

[0196] The text-guided global mask and the text-guided detail-aware token are concatenated to obtain a text-guided visual identifier.

[0197] The processor 902 is used to extract semantic information from the global feature of the text using the first latent variable based on the cross attention operation to obtain a global mask of the text as the first semantic information;

[0198] The method of performing global enhancement processing on the multimedia features by using the first semantic information extracted based on the global features of the text to obtain a global mask guided by the text includes:

[0199] The multimedia global features are enhanced using the text global mask to obtain the text-guided global mask.

[0200] The processor 902 is used to extract semantic information from the text detail feature based on the cross attention operation using the second latent variable in the first perception layer to obtain a text detail guidance token as the second semantic information;

[0201] The step of performing detail enhancement processing on the multimedia feature by using the second semantic information extracted based on the text detail feature to obtain a text-guided detail-aware token includes:

[0202] In the second perception layer, the multimedia features are enhanced using text detail guidance tokens to obtain text-guided detail perception tokens.

[0203] The processor 902 is used to convert the multimedia features to obtain multimedia embedding tokens aligned with the large language model;

[0204] The multimedia embedded token is converted using the text detail guidance token to obtain a multimedia token.

[0205] The processor 902 is used to perform word segmentation processing on the text to obtain a text identifier after word segmentation;

[0206] The text-guided visual identifier and the post-word segmentation text identifier are combined and then input into the large language model.

[0207] The processor 902 is used to construct a multimodal training sample; wherein the multimodal training sample includes a multimedia training sample and a text training sample related to the multimedia training sample;

[0208] While keeping the weights of the visual encoder and the large language model unchanged, the projector and the global mask module are trained using the multimodal training samples to obtain a first training result and a text-guided global mask training result; and

[0209] Using the first training result output by the projector training to train the detail sensor, and obtaining a text-guided detail perception token training result;

[0210] The text-guided global mask training result and the text-guided detail-aware token training result are fused into the large language model for processing to obtain a multimodal output result, so as to optimize the projector, the global mask module and the detail sensor according to the multimodal output result.

[0211] The memory 601 can be configured to store various other data to support operations on the electronic device. Examples of these data include instructions for any application or method for operating on the electronic device. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0212] Furthermore, the processor 602 in this embodiment may specifically be: a programmable switching processing chip, in which a data replication engine is configured to replicate received data.

[0213] When the processor 602 executes the program in the memory, in addition to the above functions, it can also realize other functions, which can be specifically referred to the description of the above embodiments. Figure 6 As shown, the electronic device also includes: a power supply component 604 and other components.

[0214] The present application also provides a non-transitory machine-readable storage medium, wherein the non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes Figure 2 The method described in the corresponding embodiment.

[0215] The present application also provides a computer program product, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, Figure 2 The method described in the corresponding embodiment.

[0216] Based on the above embodiment, the scheme is applied to a multimodal model, specifically a multimodal model capable of image and text processing. Assuming that the multimodal model for image and text processing has been pre-trained, the multimedia provided by the user or actively received, as well as the text related to the multimedia, can be obtained. Feature extraction is performed on the multimedia and text respectively to obtain multimedia features and text features. Further, the multimedia features are enhanced using the text features. Specifically as follows: the multimedia features are globally enhanced using the text features to obtain a global mask guided by the text, thereby realizing global attention to the multimedia with the help of the text; and the multimedia features are enhanced using the text features to obtain a detail perception token guided by the text, thereby realizing detail perception of the multimedia with the help of the text. Here, by establishing an interactive relationship between the text features and the multimedia features, the multimodal model can pay attention to the details and the global aspects of the multimedia after analyzing the text like a human being, thereby being able to obtain a more accurate model output result. After the obtained global mask guided by the text and the detail perception token guided by the text are input into the large language model, an accurate response result related to the multimedia is obtained. Through the above method, the semantic information in the text is refined and used as guiding information to enhance the original multimedia features. This overcomes the dilemma of isolated encoding processes between different modes in traditional multimodal models, and is more in line with the thinking logic of humans in reality, observing objects with questions and purposes, thereby making the multimodal model output a response result that is more in line with the text requirements put forward by users.

[0217] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0218] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0219] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0220] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multimedia processing method, characterized in that: Applied to a multimodal model, the method comprises: Retrieving multimedia and text associated with the multimedia; Performing feature processing on the multimedia and the text to obtain multimedia features and text features; Extracting semantic information from the text features using learnable latent variables to obtain semantic information; Using the semantic information to enhance the multimedia features, obtaining a text-guided visual identifier; The text-guided visual identifier is input into a large language model to generate a response result related to the multimedia.

2. The method according to claim 1, characterized in that The performing feature processing on the multimedia and the text to obtain multimedia features and text features includes: Performing feature processing on the multimedia to obtain the multimedia features including multimedia global features and / or visual detail tokens; Feature processing is performed on the text to obtain the text features including text global features and / or text detail features.

3. The method according to claim 1 or 2, characterized in that: The method of enhancing the multimedia features by using the semantic information to obtain a text-guided visual identifier includes: Performing global enhancement processing on the multimedia features using the first semantic information extracted based on the global features of the text to obtain a global mask guided by the text; Performing detail enhancement processing on the multimedia feature using the second semantic information extracted based on the text detail feature to obtain a text-guided detail-aware token; The text-guided global mask and the text-guided detail-aware token are concatenated to obtain a text-guided visual identifier.

4. The method according to claim 3, characterized in that The step of extracting semantic information from the text features using learnable latent variables to obtain semantic information includes: Based on the cross attention operation, the first latent variable is used to extract semantic information from the global features of the text, and a global mask of the text as the first semantic information is obtained; The method of performing global enhancement processing on the multimedia features by using the first semantic information extracted based on the global features of the text to obtain a global mask guided by the text includes: The multimedia global features are enhanced using the text global mask to obtain the text-guided global mask.

5. The method according to claim 3, characterized in that: The step of extracting semantic information from the text features using learnable latent variables to obtain semantic information includes: In the first perception layer, based on the cross attention operation, the second latent variable is used to extract semantic information from the text detail features, and a text detail guidance token as the second semantic information is obtained; The step of performing detail enhancement processing on the multimedia feature by using the second semantic information extracted based on the text detail feature to obtain a text-guided detail-aware token includes: In the second perception layer, the multimedia features are enhanced using text detail guidance tokens to obtain text-guided detail perception tokens.

6. The method according to claim 5, characterized in that Before the multimedia features are enhanced using the text detail guidance token, the method further includes: Performing conversion processing on the multimedia features to obtain multimedia embedding tokens aligned with the large language model; The multimedia embedded token is converted using the text detail guidance token to obtain a multimedia token.

7. The method according to claim 1, characterized in that The step of inputting the text-guided visual identifier into a large language model comprises: Performing word segmentation processing on the text to obtain a text identifier after word segmentation; The text-guided visual identifier and the post-word segmentation text identifier are combined and then input into the large language model.

8. A model training method, characterized in that: The method comprises: Constructing multimodal training samples; wherein the multimodal training samples include multimedia training samples and text training samples related to the multimedia training samples; While keeping the weights of the visual encoder and the large language model unchanged, the projector and the global mask module are trained using the multimodal training samples to obtain a first training result and a text-guided global mask training result; and Using the first training result output by the projector training to train the detail sensor, and obtaining a text-guided detail perception token training result; The text-guided global mask training result and the text-guided detail-aware token training result are fused into the large language model for processing to obtain a multimodal output result, so as to optimize the projector, the global mask module and the detail sensor according to the multimodal output result.

9. A multimedia processing system, characterized in that: The system comprises: A multimedia encoder is used to encode multimedia to obtain multimedia features; A text encoder is used to encode the text to obtain text features; A projector, configured to perform conversion processing on the multimedia features to obtain multimedia embedding tokens aligned with a large language model; A global processing module, used for performing global enhancement processing on the multimedia features using the text features to obtain a text-guided global mask; A detail processing module, used to perform detail enhancement processing on the multimedia features using the text features to obtain text-guided detail-aware tokens; The large language model is used to input the text-guided global mask and the text-guided detail-aware token into the large language model to generate a response result related to the multimedia.

10. A multimedia processing device, characterized in that: Applied to a server, the device comprises: An acquisition module, used for acquiring multimedia and text related to the multimedia; A feature processing module, used for performing feature processing on the multimedia and the text to obtain multimedia features and text features; An extraction module, used for extracting semantic information from the text features using learnable latent variables to obtain semantic information; An enhancement processing module, used to enhance the multimedia features using the semantic information to obtain a text-guided visual identifier; A generating module is used to input the text-guided visual identifier into a large language model to generate a response result related to the multimedia.

11. An electronic device comprising a memory and a processor; wherein: The memory is used to store programs; The processor is coupled to the memory, and is configured to execute the program stored in the memory, so as to implement the method according to any one of claims 1 to 8.

12. A non-transitory machine-readable storage medium having executable code stored thereon, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as claimed in any one of claims 1 to 8.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images