Training method and device of multi-modal reasoning system for large language model, equipment and medium

Abstract

The application discloses a training method and device of a large language model-oriented multi-modal reasoning system, equipment and a medium, and relates to the field of artificial intelligence. The method comprises the following steps: processing sample visual features and first sample text features by a large language model to obtain first embedding representation and second embedding representation; obtaining a first loss value according to a sample target title and the first embedding representation; obtaining conditional distribution embedding representation and deterministic embedding representation by a to-be-trained deep neural network, and obtaining a second loss value based on the conditional distribution embedding representation and the deterministic embedding representation; obtaining a third loss value based on fusion embedding representation and second sample text features; training the to-be-trained deep neural network based on the three loss values to obtain a trained deep neural network; and performing multi-modal reasoning by a multi-modal reasoning system comprising the large language model and the trained deep neural network, so as to improve the generalization capability of complex multi-modal reasoning tasks.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a training method, apparatus, device, and medium for a multimodal reasoning system for large language models. Background Technology

[0002] Multimodal semantic reasoning methods for large language models have become a research hotspot in the field of artificial intelligence in recent years. These methods aim to achieve joint understanding and reasoning of vision and language through multimodal data pre-training and information interaction. To enable large models to possess multimodal information processing and semantic reasoning capabilities, it is typically necessary to construct a pre-trained visual encoder, connection layers, a large language model architecture, and utilize cross-modal semantic alignment and autoregressive language modeling techniques.

[0003] In the field of multimodal reasoning, the standard model architecture is a "visual encoder-connector layer-large language model" architecture. Its core idea is to utilize a lightweight query Transformer module to connect the visual encoder and the language model, and then propose a two-stage model training mechanism responsible for semantic alignment and semantic reasoning respectively. This architecture demonstrates good performance in tasks such as image-text retrieval, image caption generation, and visual question answering. Furthermore, related technologies attempt to achieve multimodal combined reasoning tasks by freezing the large language model and only adjusting the input and output projection layers. All of these employ deterministic feature embedding methods to map visual and textual information to a fixed semantic space.

[0004] In other words, in relevant multimodal reasoning methods, visual and linguistic representations generally employ a single deterministic embedding approach. This involves projecting data from different modalities into a shared semantic space for similarity measurement using a fixed feature mapping function. While this representation method is structurally simple and easy to implement, it suffers from significant technical limitations in complex multimodal tasks. Within this deterministic representation framework, the model's semantic mapping can only capture the average relationships between modalities, resulting in insufficient generalization ability and weak semantic consistency in complex reasoning tasks. Summary of the Invention

[0005] In view of the above-mentioned technical problems, the present invention provides a training method, apparatus, device and medium for a multimodal reasoning system for large language models, aiming to overcome the above problems or at least partially solve the above problems.

[0006] The first aspect of this invention provides a training method for a multimodal reasoning system oriented towards large language models, the method comprising:

[0007] The sample visual features of the sample reference image and the first sample text features of the sample condition title are processed by a pre-trained large language model to obtain a first embedding representation and a second embedding representation; the sample condition title is used to characterize the modification content of the sample reference image.

[0008] A first loss value is obtained based on the first predicted sample title corresponding to the sample target title and the first embedded representation; the sample target title is obtained based on the sample reference image and the sample condition title.

[0009] The second embedding representation is processed by the deep neural network to be trained to obtain a conditional distribution embedding representation and a deterministic embedding representation; based on the conditional distribution embedding representation and the deterministic embedding representation, a second loss value is obtained;

[0010] The conditional distribution embedding representation and the deterministic embedding representation are fused to obtain a fused embedding representation; a third loss value is obtained by comparing and learning the fused embedding representation with the second sample text features corresponding to the sample target title.

[0011] Based on the first loss value, the second loss value, and the third loss value, the parameters of the deep neural network to be trained are updated to obtain the trained deep neural network.

[0012] A multimodal reasoning system oriented towards a large language model, comprising the pre-trained large language model and the trained deep neural network, processes the reference image and conditional title to obtain the target title and / or target image.

[0013] A second aspect of the present invention provides a training apparatus for a multimodal reasoning system for large language models, the apparatus comprising:

[0014] The first processing module is used to process the sample visual features of the sample reference image and the first sample text features of the sample condition title using a pre-trained large language model to obtain a first embedding representation and a second embedding representation; the sample condition title is used to characterize the modification content of the sample reference image.

[0015] The first loss determination module is used to obtain a first loss value based on the first predicted sample title corresponding to the sample target title and the first embedded representation; the sample target title is obtained based on the sample reference image and the sample condition title;

[0016] The second loss determination module is used to process the second embedding representation through the deep neural network to be trained to obtain a conditional distribution embedding representation and a deterministic embedding representation; and to obtain a second loss value based on the conditional distribution embedding representation and the deterministic embedding representation.

[0017] The third loss determination module is used to fuse the conditional distribution embedding representation and the deterministic embedding representation to obtain a fused embedding representation; and to perform comparative learning based on the fused embedding representation and the second sample text features corresponding to the sample target title to obtain a third loss value.

[0018] The network training module is used to update the parameters of the deep neural network to be trained based on the first loss value, the second loss value and the third loss value, so as to obtain the trained deep neural network.

[0019] The multimodal reasoning module is used to process the reference image and conditional title through a multimodal reasoning system oriented towards a large language model, including the pre-trained large language model and the trained deep neural network, to obtain the target title and / or target image.

[0020] A third aspect of the present invention provides an electronic device comprising a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the training method for a multimodal reasoning system for a large language model as described in the first aspect of the present invention.

[0021] A fourth aspect of the present invention provides a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the training method for a multimodal inference system for a large language model as described in the first aspect of the present invention.

[0022] In the training method for a multimodal reasoning system oriented towards a large language model proposed in this invention, the second embedding representation output by the large language model is processed by the deep neural network to be trained, and distribution alignment is performed in the probability space to achieve probabilistic modeling of semantic distribution, resulting in conditional distribution embedding representation and deterministic embedding representation. The conditional distribution embedding representation can compensate for the deficiency of the deterministic embedding representation, which can only capture the average relationship between modalities, and characterize the potential multimodal multimodal distribution and semantic hierarchical structure, reflecting the multimodality and uncertainty in the visual-language mapping. By fusing the deterministic embedding representation and the conditional distribution embedding representation, deterministic-probabilistic fusion is achieved, which can introduce uncertainty information while maintaining the accuracy of deterministic semantics, thereby enhancing the expressive power and generalization ability of the multimodal reasoning system. In the training phase, this invention not only obtains a first loss value based on the first predicted sample title corresponding to the first embedded representation of the sample target title, but also obtains a third loss value by performing distribution alignment in the probability space and combining it with contrastive learning, so as to simultaneously constrain semantic consistency and semantic diversity. In addition, this invention also introduces probability distribution constraints to obtain a second loss value, so that the multimodal reasoning system can achieve cross-modal alignment convergence in fewer iterations. Thus, the multimodal reasoning system trained by this invention through these three loss values ​​can provide a distribution-level representation when faced with multiple linguistic descriptions of the same visual object or the polysemy of language. This overcomes the problems of strong determinism in visual semantic representation embedding and limited reasoning ability of multimodal large language models in related technologies, reduces reasoning errors caused by semantic ambiguity in multimodal reasoning, and improves the reasoning performance of multimodal reasoning. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the steps of a training method for a multimodal reasoning system for large language models, as shown in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the training process of a multimodal reasoning system according to an embodiment of the present invention;

[0026] Figure 3 This is a structural block diagram of a training device for a multimodal reasoning system for large language models, provided in an embodiment of the present invention.

[0027] Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] This invention carefully examines the current multimodal reasoning methods that employ a single deterministic embedding approach for visual and linguistic representations, and identifies at least the following technical shortcomings and their causes:

[0030] First, deterministic embeddings struggle to reflect the semantic uncertainty between vision and language: In real-world multimodal reasoning scenarios, the same visual object may correspond to multiple natural language descriptions, and different linguistic expressions may refer to different visual regions. For example, "a running dog" and "an animal on the grass" have semantic overlap, while large language models with deterministic embeddings can only generate single-point representations, failing to reflect the ambiguity and polysemy of semantic distribution, leading to unstable model performance in polysemous, ambiguous, or complex semantic reasoning.

[0031] Second, deterministic representations lack the ability to model visual hierarchical structures: traditional visual-language mapping methods often encode the entire image as a single global feature vector, ignoring the hierarchical semantic relationships and region-level semantic combinations present in the image. For example, although the BLIP-2 model achieves cross-modal feature interaction by querying the Transformer module, it still performs semantic alignment within a fixed embedding space, failing to explicitly distinguish the probabilistic associations between different semantic granularities, thus limiting the model's generalization performance in complex visual semantic synthesis and structured reasoning tasks.

[0032] Third, deterministic embeddings lack robustness and stability: due to the lack of probability distribution constraints, the model's embedding results are prone to significant fluctuations when the input visual features are noisy, occluded, or subject to multi-view variations, leading to inference bias or misjudgment. Existing methods have attempted to enhance training stability through regularization or feature perturbation, but have not systematically addressed the semantic drift problem caused by deterministic embeddings from a probabilistic modeling perspective. For example, the FROMAGe model improves combinatorial reasoning ability by freezing the language model and optimizing the visual projection layer, but it is still based on single-point embedding representations, making it difficult to characterize or estimate input uncertainty.

[0033] This invention argues that the fundamental reason for the aforementioned technical deficiencies lies in the lack of a probabilistic modeling mechanism for semantic distribution in current multimodal reasoning methods. Therefore, to at least partially address one or more of the aforementioned problems and other potential issues, this invention proposes a training method for multimodal reasoning systems targeting large language models. This method proposes a probabilistic representation learning approach for multimodal reasoning in large language models, achieving multimodal semantic fusion and probabilistic representation learning. This enables explicit modeling and representation optimization of multimodal semantic uncertainty, thereby significantly improving the multimodal semantic reasoning capability of large language models and overcoming the problems of strong deterministic embedding of visual semantic representations and limited model reasoning capabilities in related technologies. Specifically, this invention achieves probabilistic modeling of semantic distribution by aligning the second embedding representation output by the large language model within the probability space, resulting in conditional distribution embedding representations and deterministic embedding representations. The conditional distribution embedding representation compensates for the limitation of the deterministic embedding representation, which can only capture the average relationship between modalities, characterizing the potential multimodal distribution and semantic hierarchy of multimodalities, and reflecting the multimodality and uncertainty in the visual-language mapping. By fusing the deterministic and conditional distribution embedding representations, deterministic-probabilistic fusion is achieved, which can introduce uncertainty information while maintaining the accuracy of deterministic semantics, thereby enhancing the expressive and generalization capabilities of the multimodal reasoning system. Furthermore, during the training phase, this invention aligns the sample target title with the first embedding representation... Based on the first loss value obtained from the corresponding first predicted sample title, a third loss value is obtained by performing distribution alignment in the probability space and combining it with contrastive learning, so as to simultaneously constrain semantic consistency and semantic diversity. In addition, the present invention also introduces probability distribution constraints to obtain a second loss value, so that the multimodal reasoning system can achieve cross-modal alignment convergence in fewer iterations. Thus, the multimodal reasoning system trained by the present invention through these three loss values ​​can provide a distribution-level representation when faced with multiple linguistic descriptions of the same visual object or the polysemy of language. This overcomes the problems of strong determinism in the embedding of visual semantic representation and limited reasoning ability of multimodal large language models in related technologies, reduces reasoning errors caused by semantic ambiguity in multimodal reasoning, and improves the reasoning performance of multimodal reasoning.

[0034] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a training method for a multimodal reasoning system for large language models, as shown in an embodiment of the present invention. Figure 1 As shown, the training method for a multimodal reasoning system for large language models provided in this embodiment includes at least the following steps:

[0035] Step S11: Process the sample visual features of the sample reference image and the first sample text features of the sample condition title using a pre-trained large language model to obtain the first embedding representation and the second embedding representation.

[0036] In this embodiment, the multimodal reasoning system for large language models includes at least a pre-trained large language model. This system can perform multimodal reasoning tasks such as image-text retrieval, image description generation, and visual question answering. The sample reference image is a reference image used during training and is one of the training samples. In this embodiment, the reference image can be understood as the image to be modified. The sample condition title is a condition title used during training and is one of the training samples. This condition title is used to characterize the modifications made to the sample reference image. It can be understood that the condition title in this embodiment characterizes the modifications made to the reference image. This condition title can serve as an editing command for modifying attributes and objects of the reference image, or it can describe the differences between the reference image and the target image.

[0037] In this embodiment, the visual features of the sample reference image and the first sample text features of the sample condition title are input into a pre-trained large language model. The pre-trained large language model processes the visual features and the first sample text features to obtain a first embedding representation and a second embedding representation. The visual features are the visual features corresponding to the sample reference image (e.g., global visual features), which can be obtained by extracting features from the sample reference image using a pre-trained visual encoder. The first sample text features are the text features corresponding to the sample condition title (e.g., global text features), which can be obtained by extracting features from the sample condition title using a pre-trained text encoder; alternatively, they can be obtained by extracting features from the sample condition title using a text segmenter. In this embodiment, the first embedding representation is the embedding representation output by the large language model that corresponds to the sample condition title; the second embedding representation is the embedding representation output by the large language model that is not the first embedding representation.

[0038] Step S12: Obtain the first loss value based on the first predicted sample title corresponding to the first embedded representation of the sample target title.

[0039] In this embodiment, the sample target title is obtained based on the sample reference image and the sample conditional title. It can be understood that the sample target title is the target title during the training process, one of the training samples, and can be understood as a training label. This target title is an image description obtained by modifying the reference image based on the conditional title. The first embedding representation characterizes the prediction result of the pre-trained large language model for the target title, and its corresponding first predicted sample title can be determined based on the first embedding representation. In this embodiment, a first loss value can be calculated based on the sample target title corresponding to the sample reference image and the sample conditional title, and the first predicted sample title corresponding to the first embedding representation. In an optional embodiment, the first loss value is an autoregressive loss, which can be calculated based on the sample target title and the first predicted sample title.

[0040] Step S13: Process the second embedding representation using the deep neural network to be trained to obtain a conditional distribution embedding representation and a deterministic embedding representation; based on the conditional distribution embedding representation and the deterministic embedding representation, obtain a second loss value.

[0041] In this embodiment, the second embedding representation can be processed by a deep neural network to be trained, and distribution alignment can be performed in the probability space (i.e., the conditional distribution can be modeled by the deep neural network to be trained), thereby achieving probabilistic modeling of the semantic distribution and obtaining conditional distribution embedding representations and deterministic embedding representations. This achieves the goal of semantic consistency and capturing cross-modal uncertainty. In this embodiment, the conditional distribution embedding representation is a probabilistic representation (i.e., an uncertainty representation), which can characterize the potential multimodal multimodal distribution and semantic hierarchy, reflecting the multimodality and uncertainty in the visual-language mapping. During the training process, a second loss value can be calculated based on the obtained conditional distribution embedding representation and deterministic embedding representation. This second loss value represents the probability distribution constraints.

[0042] Step S14: Fuse the conditional distribution embedding representation and the deterministic embedding representation to obtain a fused embedding representation; perform comparative learning based on the fused embedding representation and the second sample text features corresponding to the sample target title to obtain a third loss value.

[0043] In this embodiment, conditional distribution embeddings and deterministic embeddings are fused to obtain a fused embedding representation, which introduces uncertainty information while maintaining the accuracy of deterministic semantics. After obtaining the fused embedding representation, it is compared with the second sample text features corresponding to the target title to calculate a third loss value. Specifically, the comparison learning aims to minimize the difference between the fused embedding representation and the second sample text features, and maximize the difference between the fused embedding representation and other sample text features of the target title, to obtain the third loss value. The second sample text features are the text features corresponding to the target title (such as global text features), which can be obtained by extracting features from the target title using a pre-trained text encoder.

[0044] Step S15: Based on the first loss value, the second loss value, and the third loss value, update the parameters of the deep neural network to be trained to obtain the trained deep neural network.

[0045] In this embodiment, the parameters of the deep neural network to be trained can be updated based on the first loss value, the second loss value, and the third loss value until the loss converges, resulting in a trained deep neural network. In an optional example, the sum of the first loss value, the second loss value, and the third loss value (or the weighted sum of the first loss value, the second loss value, and the third loss value) can be determined as the total loss value. The network parameters of the deep neural network to be trained can be updated based on the total loss until the total loss converges. The network parameters are then fixed, resulting in a trained deep neural network.

[0046] Step S16: Using a multimodal reasoning system oriented towards a large language model, including the pre-trained large language model and the trained deep neural network, the reference image and conditional title are processed to obtain the target title and / or target image.

[0047] In this embodiment, after obtaining the trained deep neural network, a trained multimodal reasoning system for a large language model can be obtained. This trained multimodal reasoning system for a large language model includes at least: a pre-trained large language model and a trained deep neural network. In another optional embodiment, the trained multimodal reasoning system for a large language model includes at least: a pre-trained large language model, a pre-trained visual encoder, a pre-trained text encoder, and a trained deep neural network.

[0048] In this embodiment, during actual multimodal inference, a reference image and conditional title can be input into a multimodal inference system oriented towards large language models. The system processes the reference image and conditional title to obtain the target title and / or target image output by the multimodal inference system. In this embodiment, the target image is a modified image based on the conditional title, derived from the reference image.

[0049] In this embodiment, the second embedding representation output by the large language model is processed by the deep neural network to be trained, and the distribution is aligned in the probability space to achieve probabilistic modeling of the semantic distribution, resulting in conditional distribution embedding representation and deterministic embedding representation. The conditional distribution embedding representation can compensate for the deficiency of the deterministic embedding representation, which can only capture the average relationship between modalities, and characterize the potential multimodal distribution and semantic hierarchy, reflecting the multimodality and uncertainty in the visual-language mapping. By fusing the deterministic embedding representation and the conditional distribution embedding representation, deterministic-probabilistic fusion is achieved, which can introduce uncertainty information while maintaining the accuracy of deterministic semantics, thereby enhancing the expressive power and generalization ability of the multimodal reasoning system. In the training phase, this invention not only obtains a first loss value based on the first predicted sample title corresponding to the first embedded representation of the sample target title, but also obtains a third loss value by performing distribution alignment in the probability space and combining it with contrastive learning, so as to simultaneously constrain semantic consistency and semantic diversity. In addition, this invention also introduces probability distribution constraints to obtain a second loss value, so that the multimodal reasoning system can achieve cross-modal alignment convergence in fewer iterations. Thus, the multimodal reasoning system trained by this invention through these three loss values ​​can provide a distribution-level representation when faced with multiple linguistic descriptions of the same visual object or the polysemy of language. This overcomes the problems of strong determinism in visual semantic representation embedding and limited reasoning ability of multimodal large language models in related technologies, reduces reasoning errors caused by semantic ambiguity in multimodal reasoning, and improves the reasoning performance of multimodal reasoning.

[0050] The technical effects of the present invention will be explained below using the aforementioned specific examples:

[0051] From the perspective of the shortcomings of relevant deterministic embedding methods, in traditional multimodal models, regardless of whether the input text is "a dog running" or "an animal on the grass," the model typically maps the image or text to a single deterministic vector representation. Only when there is indeed a dog running on the grass in the input image will these two text descriptions show significant overlap in semantic space, but their semantic granularity differs: the former has a clear object category (dog) and action attribute (running), while the latter is a more abstract and vague description, only specifying the scene and biological attributes. Therefore, under the relevant deterministic embedding framework, this "semantic concrete—semantic abstract" relationship cannot be explicitly distinguished, and the multimodal model can only provide an average representation, which is prone to producing unstable or biased results during retrieval or inference.

[0052] In this embodiment, multimodal semantics is no longer modeled as a single-point vector, but rather as a parameterized probability distribution. Continuing with the example: 1. For "a running dog," the semantic distribution learned in this embodiment (e.g., conditional distribution embedding representation) will exhibit a concentrated and sharp distribution pattern, with its probability quality mainly concentrated in the semantic subspace related to "dog," "running," and "animal behavior"; 2. For "animals on the grass," due to the stronger uncertainty in its semantics (uncertainty about animal species and behavior), the semantic distribution learned in this embodiment (e.g., conditional distribution embedding representation) will exhibit a more dispersed and wider distribution pattern, covering "animals such as dogs, cats, and deer" as well as multiple potential semantic patterns such as "standing, walking, and running." This difference, reflected through conditional distribution embedding representation (e.g., distribution width, shape, and scale parameters), is the direct modeling method for semantic ambiguity and polysemy in this embodiment, characterizing the potential multimodal distribution and semantic hierarchy, reflecting the multimodality and uncertainty in the visual-language mapping, rather than relying on posterior rules or artificial thresholds. Within the probabilistic representation framework of this embodiment, the description "animals on the grass" naturally corresponds to a multi-peaked distribution in the semantic space. For example: 1. One probability peak corresponds to "a dog on the grass" activity; 2. Another probability peak corresponds to "a deer standing on the grass"; 3. There may also be secondary semantic peaks corresponding to "cattle, sheep, and other animals". "A dog running" is more likely to correspond to a unimodal or weakly multimodal distribution, with its main probability mass concentrated around the semantic pattern of "dog + running". Thus, through this multi-peaked structure, this invention can explicitly characterize the fact that "a description may correspond to multiple reasonable visual interpretations," which is difficult to express using traditional deterministic embedding methods.

[0053] Furthermore, since this embodiment employs a distribution-level representation method, at least the following technical effects can be achieved in subsequent multimodal reasoning or retrieval processes: 1. The multimodal reasoning system no longer compares only the point-to-point distance between two semantic vectors, but rather compares the matching relationship between semantic distributions; 2. When the input query is "animals on the grass," the multimodal reasoning system can tolerate a larger range of semantic deviations, thereby matching images containing "running dogs"; 3. When the query is "a running dog," the multimodal reasoning system tends to select visual content with high distribution overlap and more precise semantics. This mechanism enables the multimodal reasoning system to have adaptive reasoning capabilities between semantic abstract descriptions and semantically precise descriptions, significantly improving the stability and rationality of multimodal retrieval and reasoning tasks.

[0054] In conjunction with the above embodiments, in one implementation, the present invention also provides a training method for a multimodal reasoning system for large language models. In this method, step S11 specifically includes step S21; step S13, "processing the second embedding representation through the deep neural network to be trained to obtain a conditional distribution embedding representation and a deterministic embedding representation," specifically includes steps S22 and S23; and step S14, "fusing the conditional distribution embedding representation and the deterministic embedding representation to obtain a fused embedding representation," specifically includes step S24.

[0055] Step S21: Input multiple tags to be learned, the visual features of the sample, and the text features of the first sample into the pre-trained large language model to obtain a first embedding representation and multiple second embedding representations corresponding to the multiple tags to be learned.

[0056] In this embodiment, multiple learnable tokens, sample visual features, and first sample text features can be input into a pre-trained large language model. The pre-trained large language model processes these multiple learnable tokens, sample visual features, and first sample text features to obtain a first embedding representation and multiple second embedding representations corresponding to the multiple learnable tokens. Here, the learnable token is an initialized token, a learnable [RET] token, and one learnable token corresponds to one second embedding representation. During training, the learnable tokens are updated based on the text modal content corresponding to the sample condition title and the image modal content corresponding to the sample reference image. The learnable tokens are used to represent multimodal semantics. The second embedding representation corresponding to the learnable token is an embedding representation combining the text modal content corresponding to the sample condition title and the image modal content corresponding to the sample reference image.

[0057] In an optional example, the i-th second embedding representation The calculation method is as follows: Where L is a pre-trained large language model, For the i-th tag to be learned, For sample visual features or sample visual features after dimensional transformation, The first sample text features. In an optional example, this could involve inserting multiple tokens to be learned into the word segmenter of a large language model.

[0058] It is understood that the second embedding representation in this embodiment is the embedding representation output by the pre-trained large language model corresponding to the tag to be learned. For example, multiple tags to be learned, sample visual features, and first sample text features are input into the pre-trained large language model to obtain 100 embedding representations output by the large language model. Among the 100 embedding representations, the first 50 are the first embedding representations corresponding to the first sample text features, the middle 40 are the embedding representations corresponding to the sample visual features, and the last 10 are the second embedding representations corresponding to the tags to be learned. In this embodiment, the first 50 first embedding representations and the last 10 second embedding representations will be used subsequently.

[0059] Step S22: Input the plurality of second embedding representations into the fusion module in the deep neural network to be trained, and process the plurality of second embedding representations through the self-attention mechanism in the fusion module to obtain the deterministic embedding representation.

[0060] In this embodiment, multiple second embedding representations are first input into the fusion module of the deep neural network to be trained. The self-attention mechanism within this fusion module fuses these second embedding representations to obtain a deterministic embedding representation. Within the fusion module, intrinsic connections can be established between the multiple second embedding representations, enabling them to collaborate. In an optional embodiment, the fusion module includes two Transformer layers, each containing a standard multi-head self-attention module and a feedforward network. Finally, the fusion module processes the output of the Transformer layers using global average pooling to fuse the second embedding representations, obtaining the final embedding representation and improving the correlation and expressive power of multimodal features. This final embedding representation is the deterministic embedding representation, representing a deterministic global representation.

[0061] Step S23: Input the deterministic embedding representation into the distribution parameter estimation module in the deep neural network to be trained to obtain the first distribution parameter embedding representation, the second distribution parameter embedding representation, and the third distribution parameter embedding representation.

[0062] In this embodiment, the conditional distribution can be modeled based on the deterministic embedding representation using the distribution parameter estimation module in the deep neural network to be trained, resulting in a first distribution parameter embedding representation, a second distribution parameter embedding representation, and a third distribution parameter embedding representation. The conditional distribution embedding representation in this embodiment includes the first, second, and third distribution parameter embedding representations. Furthermore, the first distribution parameter embedding representation satisfies the following conditions: semantic alignment with the deterministic embedding representation; the second and third distribution parameter embedding representations are used to capture the semantic uncertainty of the deterministic embedding representation.

[0063] Semantic uncertainty includes at least one of the following: object-level semantic uncertainty, semantic granularity uncertainty; the more potential objects a deterministic embedding represents, the greater the object-level semantic uncertainty; the coarser the semantic granularity of a deterministic embedding, the greater the semantic granularity uncertainty. In an example, continuing with the above example, let's illustrate semantic uncertainty: Object-level semantic uncertainty: "Animals on the grass" does not explicitly define the animal category, and it can correspond to multiple potential visual entities. Semantic granularity uncertainty: Compared to "a running dog," "animals on the grass" lacks action and behavioral constraints, and its semantic granularity is coarser.

[0064] In other words, in this embodiment, based on deterministic embedding representation, the conditional distribution P(v|{v) is trained using a deep neural network. p , α p , β p When modeling, the deterministic embedding representation v is used as the mean estimate, with the aim of achieving the following two objectives: (1) preserving the first distribution parameter embedding representation v p Alignment with the deterministic embedding representation v to maintain semantic consistency; (2) by estimating the second distribution parameter embedding representation α p and the third distribution parameter embedding representation β p To capture cross-modal uncertainty.

[0065] Step S24: Fuse the first distribution parameter embedding representation with the deterministic embedding representation to obtain the fused embedding representation.

[0066] In this embodiment, during deterministic-probabilistic fusion, the first distribution parameter embedding representation and the deterministic embedding representation are fused to obtain a fused embedding representation. For example, residual connections are used to fuse the first distribution parameter embedding representation and the deterministic embedding representation to balance semantic accuracy and uncertainty modeling requirements. This fused embedding representation is the feature of a sample target image output by the multimodal inference system for large language models to be trained in this embodiment, which fuses the sample condition title and the sample reference image. The sample target image is the target image during the training process.

[0067] In this embodiment, by learning multiple distribution parameter embedding representations in the conditional distribution (such as the second distribution parameter embedding representation and the third distribution parameter embedding representation), the modal reasoning system can automatically determine the "degree of uncertainty" (i.e. semantic uncertainty) corresponding to the semantic expression and reflect it in the probability space, rather than forcibly compressing it into a single-point representation equivalent to fine semantics.

[0068] In conjunction with any of the above embodiments, the present invention also provides a training method for a multimodal reasoning system for large language models, wherein step S23 may specifically include steps S31 to S34:

[0069] Step S31: Input the deterministic embedding representation into the first multilayer perceptron in the distribution parameter estimation module to obtain the intermediate embedding representation.

[0070] In this embodiment, the distribution parameter estimation module can construct four multilayer perceptrons to estimate the conditional distribution embedding representation. Specifically, the deterministic embedding representation can be first input into the first multilayer perceptron in the distribution parameter estimation module to obtain an intermediate embedding representation. For example, it can be expressed as follows: =MLP(v), where, is the intermediate embedding representation, MLP is the first multilayer perceptron, and v is the deterministic embedding representation.

[0071] Step S32: Input the intermediate embedded representation into the second multilayer perceptron in the distribution parameter estimation module to obtain the first distribution parameter embedded representation.

[0072] In this embodiment, after obtaining the intermediate embedding representation, it can be input into the second multilayer perceptron in the distribution parameter estimation module to obtain the first distribution parameter embedding representation, which characterizes the mean of the deterministic embedding representation. For example, it can be expressed by the following formula: v p =MLP v ( ),in, MLP is an intermediate embedding representation. v For the second multilayer perceptron, v pThe first distribution parameter embedding representation.

[0073] Step S33: Input the intermediate embedded representation into the third multilayer perceptron in the distribution parameter estimation module to obtain the second distribution parameter embedded representation.

[0074] In this embodiment, after obtaining the intermediate embedding representation, it can be input into the third multilayer perceptron in the distribution parameter estimation module to obtain the second distribution parameter embedding representation, which characterizes the proportion of the deterministic embedding representation, i.e., the scale parameter of the deterministic embedding representation. For example, it can be expressed by the following formula: α p =MLP α ( ),in, MLP is an intermediate embedding representation. α For the third multilayer perceptron, α p The second distribution parameter embedding representation.

[0075] Step S34: Input the intermediate embedded representation into the fourth multilayer perceptron in the distribution parameter estimation module to obtain the third distribution parameter embedded representation.

[0076] In this embodiment, after obtaining the intermediate embedding representation, it can be input into the fourth multilayer perceptron in the distribution parameter estimation module to obtain the third distribution parameter embedding representation, which characterizes the shape parameters of the deterministic embedding representation. For example, it can be expressed by the following formula: β p =MLP β ( ),in, MLP is an intermediate embedding representation. β For the fourth multilayer perceptron, β p This is the embedding representation of the third distribution parameter. In this embodiment, each of the four multilayer perceptrons uses ReLU as the activation function.

[0077] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a training method for a multimodal reasoning system for large language models. In this method, and specifically, the step S13 above, "obtaining a second loss value based on the conditional distribution embedding representation and the deterministic embedding representation," may include the following steps S41 to S44:

[0078] Step S41: Based on the difference between the deterministic embedding representation and the first distribution parameter embedding representation, the second distribution parameter embedding representation, and the third distribution parameter embedding representation, obtain the first part of the loss value.

[0079] In this embodiment, the difference between the deterministic embedding representation and the first distribution parameter embedding representation can be obtained based on the deterministic embedding representation and the first distribution parameter embedding representation. Then, based on the difference between the deterministic embedding representation and the first distribution parameter embedding representation, the second distribution parameter embedding representation, and the third distribution parameter embedding representation, the first part of the loss value can be calculated.

[0080] Step S42: Based on the second distribution parameter embedding representation and the third distribution parameter embedding representation, obtain the second part of the loss value.

[0081] In this embodiment, the second part of the loss value can be calculated based on the second distribution parameter embedding representation and the third distribution parameter embedding representation.

[0082] Step S43: Based on the embedding representation of the third distribution parameter, obtain the third part of the loss value.

[0083] In this embodiment, the third part of the loss value can be calculated based on the embedding representation of the third distribution parameter.

[0084] Step S44: Based on the first part of the loss value, the second part of the loss value, and the third part of the loss value, obtain the second loss value.

[0085] In this embodiment, after obtaining the first part of the loss value, the second part of the loss value, and the third part of the loss value, the second loss value can be determined based on the first part of the loss value, the second part of the loss value, and the third part of the loss value.

[0086] In an optional specific example, the second loss value can be determined by the following formula. :

[0087] ;

[0088] in, For the network parameters to be optimized, For deterministic embedding representation, The first distribution parameter embedding representation, The second distribution parameter embedding representation, The third distribution parameter embedding representation, To differentiate between the deterministic embedding representation and the first distribution parameter embedding representation, ( ) represents the gamma function.

[0089] In an alternative implementation, for probability distribution reconstruction optimization, the probabilistic representation learning problem can be defined as a probability reconstruction problem, given an image. I refFirst, a deterministic embedding representation, i.e., a global visual representation v, is obtained through a large language model and a fusion module. Then, the reconstruction result is learned by maximizing the likelihood function. The parameter update is performed with the first distribution parameter embedding representation as the objective. The parameter optimization method is as follows:

[0090] ;

[0091] Where N represents the size of the samples in the dataset, Indicates the generalized Gaussian distribution. ( ) represents the gamma function. For the updated network parameters, ( ) represents maximizing the likelihood function. For the network parameters to be optimized, For the i-th deterministic embedding representation, For the embedding representation of the i-th third distribution parameter, For the i-th first distribution parameter embedding representation, Let be the embedding representation of the i-th second distribution parameter, and , , This represents the mean, scale, and shape parameters for a given v.

[0092] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a training method for a multimodal reasoning system oriented towards a large language model. In this method, in addition to the steps described above, it may further include step S51, and the step of "processing the sample visual features of the sample reference image and the first sample text features of the sample condition title using a pre-trained large language model" in S11 may specifically include step S52, and step S15 may specifically include step S53.

[0093] Step S51: Perform dimensional transformation on the visual features of the sample through the connection layer to be trained to obtain the dimensional transformation visual features of the sample.

[0094] In this embodiment, after obtaining the visual features of the samples, the visual features can be dimensionally transformed using the connection layer to be trained, resulting in the dimensionally transformed visual features. In an optional example, the multimodal inference system for large language models to be trained includes at least: a pre-trained visual encoder V(·), a pre-trained text encoder T(·), a pre-trained large language model L(·), and a randomly initialized connection layer M(·) (i.e., the connection layer to be trained). The visual encoder V(·) is used to extract the sample reference image. I ref The visual features of the sample v0=V( I refThe text encoder T(·) is used to extract sample conditional titles. T con The first sample text features t=T( T con The connection layer to be trained is used to perform dimensional transformation on the visual features v0 of the sample to obtain the dimensional transformation visual features v1=M(v0), which can be concatenated with the text features t of the first sample. Finally, the concatenated features are input into the large language model L(·) for prediction.

[0095] Step S52: Process the visual features of the dimension-transformed sample and the text features of the first sample using the pre-trained large language model.

[0096] In this embodiment, the dimensionality-transformed visual features of the sample and the first sample text features are input into a pre-trained large language model. The pre-trained large language model processes the dimensionality-transformed visual features and the first sample text features to perform relevant predictions. In an optional example, the dimensionality-transformed visual features and the first sample text features can be concatenated to obtain concatenated features, which are then input into the pre-trained large language model for processing.

[0097] Step S53: Based on the first loss value, the second loss value, and the third loss value, update the parameters of the deep neural network to be trained and the connection layer to be trained to obtain the trained deep neural network and the trained connection layer.

[0098] In this embodiment, the network parameters of the deep neural network to be trained and the connection layer to be trained can be updated based on the first loss value, the second loss value, and the third loss value until the loss converges, resulting in a trained deep neural network and a trained connection layer. In an optional example, the sum of the first loss value, the second loss value, and the third loss value (or the weighted sum of the first loss value, the second loss value, and the third loss value) can be determined as the total loss value. Based on the total loss, the network parameters of the deep neural network to be trained and the connection layer to be trained can be updated until the total loss converges. The network parameters are then fixed, resulting in a trained deep neural network and a trained connection layer.

[0099] In this embodiment, the multimodal inference system for large language models includes at least: a pre-trained large language model, a trained deep neural network, and a trained connection layer. In another optional embodiment, the trained multimodal inference system for large language models includes at least: a pre-trained large language model, a pre-trained visual encoder, a pre-trained text encoder, a trained deep neural network, and a trained connection layer.

[0100] In this embodiment, considering the large number of parameters of the visual encoder and the large language model, a pre-trained visual encoder and large language model are directly used. However, since the features output by the visual encoder cannot be directly applied to the large language model, this embodiment fine-tunes the network parameters of the connection layer to improve the multimodal inference performance of the entire multimodal inference system while minimizing the amount of training computation.

[0101] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a training method for a multimodal reasoning system for large language models. In addition to the steps described above, this method may further include steps S61 to S64:

[0102] Step S61: Obtain the image-text pair dataset, take the image in an image-text pair as the sample reference image, and take the text in the image-text pair as the sample reference title of the sample reference image.

[0103] In this embodiment, a dataset of image-text pairs can be obtained. This dataset includes multiple image-text pairs, each pair consisting of an image and its corresponding text. In this embodiment, for the image-text pair dataset, the image in an image-text pair can be used as a sample reference image, and the text in the image-text pair can be used as a sample reference title for that sample reference image. The sample reference title is the text describing the sample reference image.

[0104] In an optional example, an open-source dataset containing image-text pairs can be selected, denoted as the sample reference image. I ref and sample reference title T ref .

[0105] Step S62: Input the sample reference image, the sample reference title, and the first prompt text into the first large language model to obtain the sample condition title.

[0106] In this embodiment, the sample reference image, sample reference title, and first prompt text can be input into the first language model for processing to obtain the sample condition title output by the first language model. T con. Among them, the sample condition title T con. The modifications to the sample reference image can serve as editing commands to modify attributes and objects, or as descriptions of the differences between the sample reference image and the sample target image. The first prompt text represents the prompt for generating modifications to the image and description. The first large language model is a pre-trained large language model, which may be the same as or different from the aforementioned pre-trained large language models.

[0107] Step S63: Input the sample reference title, the sample condition title, and the second prompt text into the first large language model to obtain the sample target title.

[0108] In this embodiment, the sample reference title, sample condition title, and second prompt text can be input into the first language model for processing to obtain the sample target title output by the first language model. T tar The second prompt text represents a prompt for generating a modified title based on the reference title and the conditional title. It can be understood that the sample target title is a description of the sample target image.

[0109] Step S64: Use the sample reference image, the sample condition title, and the sample target title as a training sample to obtain a training sample set.

[0110] In this embodiment, the sample reference image, sample condition title, and sample target title obtained in steps S61-S63 can be used as a training sample, and multiple training samples can be obtained through the same method. In this way, a training sample set including multiple training samples can be constructed, thereby enabling the training of a multimodal reasoning system for large language models based on the training sample set.

[0111] In an alternative embodiment, a quadruple can be constructed. I ref ，T ref ，T con ，T tar >As network training data. Among them, I ref Representative sample reference image; T ref Representative sample reference title; T con This represents the title of the sample condition, which specifies the modifications made to the sample reference image. T tar This represents the target title of the sample, i.e., the target title of the modified image. Below is an example of its construction using a large language model:

[0112] User: I need to build a multimodal dataset. It uses a four-tuple format: sample reference image, sample reference title, sample condition title, and sample target title. I would greatly appreciate your assistance in creating the four-tuple data based on the provided sample reference image title. Here are the specific steps:

[0113] Sample reference title: This is the description of this image that I will provide to you.

[0114] Sample Condition Title: This text, generated from the original title content, explains the specific modification requirements or regulations that the sample target title (and the corresponding sample target image) must follow.

[0115] Sample target title: This should be accomplished by combining the sample reference image title and the sample condition title. It will generate a new image that conforms to the requirements set by the text conditions, based on the sample reference image.

[0116] Large Language Model: Understood! Please provide the original sample reference title. I will generate sample condition titles related to the image and the corresponding sample target titles.

[0117] User: Sample title: A cute kitten looking up.

[0118] Large Language Model: Excellent! Below are the sample conditional titles and sample target titles:

[0119] "Sample condition title: With a toy mouse. Sample target title: A cute cat is playing with a toy mouse."

[0120] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a training method for a multimodal reasoning system oriented towards large language models. In this method, step S16 may specifically include steps S71 and S72:

[0121] Step S71: The reference image and the conditional title are processed by the multimodal reasoning system to obtain a first target embedding representation and a target fusion embedding representation.

[0122] In this embodiment, during the actual multimodal inference process, reference images and conditional titles can be input into a multimodal inference system for large language models. The multimodal inference system processes the reference images and conditional titles to obtain a first target embedding representation and a target fusion embedding representation output by the multimodal inference system. The first target embedding representation can be understood as the first embedding representation during model application (i.e., the actual multimodal inference process), and the target fusion embedding representation can be understood as the fusion embedding representation during model application. Specifically, the method for obtaining the first target embedding representation and the target fusion embedding representation during the actual multimodal inference process is the same as or similar to the method for obtaining the first embedding representation and the fusion embedding representation during model training in any of the aforementioned embodiments, and can be referred to the aforementioned embodiments.

[0123] Step S72: Obtain the target title based on the first target embedding representation, the target title being used to describe the modified image obtained after modifying the reference image according to the condition title; and / or, retrieve the target image from the image library based on the target fusion embedding representation.

[0124] In this embodiment, a target title corresponding to a reference image and a conditional title can be obtained based on the first target embedding representation. This target title describes the modified image obtained after modifying the reference image according to the conditional title. And / or

[0125] In this embodiment, a target image can be retrieved from an image library based on the target fusion embedding representation, thereby achieving image retrieval. In this embodiment, the target fusion embedding representation is the feature of the modified image obtained by modifying the reference image according to the conditional title, predicted by the multimodal inference system. In another optional embodiment, after obtaining the target fusion embedding representation, a target image can be generated based on the target fusion embedding representation, thereby enabling modifications to the reference image, such as image retouching or image editing.

[0126] In this embodiment, the trained multimodal reasoning system for large language models can perform tasks such as semantic combination, semantic modification, or conditional reasoning (e.g., combined image retrieval and semantic editing reasoning), thereby enhancing the expressive power and generalization performance of large language models for tasks such as semantic combination, semantic modification, or conditional reasoning.

[0127] In one embodiment, such as Figure 2 As shown, Figure 2 This is a schematic diagram illustrating the training process of a multimodal inference system according to an embodiment of the present invention. Figure 2 In this process, the first sample text feature is obtained by passing the sample condition title (adding a bird in the background) through a word segmenter (or a text encoder); the sample reference image is used to extract features through a pre-trained visual encoder to obtain sample visual features, which are then input into the connection layer to be trained to obtain dimensionality-transformed sample visual features; then, multiple learnable tags, dimensionality-transformed sample visual features, and the first sample text feature are input into a pre-trained large language model to obtain a first embedding representation and multiple second embedding representations corresponding to the multiple learnable tags; autoregression is performed on the first predicted sample title corresponding to the first embedding representation based on the sample target title (a cat in the kitchen window and a bird in the background) and the first embedding representation to obtain a first loss value; the multiple second embedding representations are input into the deep neural network to be trained, and the multiple second embedding representations are fused to obtain a deterministic embedding representation v; then, conditional distribution modeling is performed based on the deterministic embedding representation v to map the deterministic embedding representation v to a parameterized probability distribution (such as mean and variance or more generalized distribution parameters) to obtain a first distribution parameter embedding representation. Second distribution parameter embedding representation and the third distribution parameter embedding representation Based on v, , and The second loss value is calculated; then, the first distribution parameters are embedded in the representation. The deterministic embedding representation v is fused with a residual connection to obtain a fused embedding representation, thereby reconstructing the probability distribution. Then, based on the fused embedding representation, it is compared with the second sample text features extracted by the pre-trained text encoder (including large-scale negative sample comparison) to obtain a third loss value. Finally, based on the first, second, and third loss values, the parameters of the deep neural network and the connection layer to be trained are updated to obtain the trained deep neural network and the trained connection layer, thus obtaining the trained multimodal inference system for large language models. In this embodiment, the title mode refers to predicting the sample target title using sample condition titles and sample reference images; that is, predicting the sample target title for a sample target image that matches the sample condition titles and sample reference images using the multimodal inference system. The retrieval mode refers to retrieving sample target images using sample condition titles and sample reference images; that is, predicting the vector of a sample target image that matches the sample condition titles and sample reference images using the multimodal inference system. In the early stages of model training, two training modes can be initialized: retrieval mode and title mode. Then, based on the initialized retrieval mode and title mode, the reconstruction results are learned by maximizing the likelihood function based on the generalized Gaussian distribution. Finally, the network is trained and optimized by combining contrastive learning loss and autoregressive loss.

[0128] In summary, combining any of the above embodiments, the training method for a multimodal reasoning system for large language models provided by the present invention includes at least the following technical effects:

[0129] 1. Explicitly modeling semantic uncertainty: This invention maps deterministic features to parameterized probability distributions (such as mean and variance or more generalized distribution parameters), which can reflect the multimodality and uncertainty in the visual-language mapping. This enables the model to provide a distribution-level representation when faced with multiple linguistic descriptions of the same visual object or the polysemy of language, thereby reducing retrieval or inference errors caused by semantic ambiguity.

[0130] 2. Improved Reasoning Stability for Polysemous and Ambiguous Semantics: By performing distribution alignment within the probability space and combining it with contrastive learning (including large-scale negative sample comparison), this invention can simultaneously constrain semantic consistency and semantic diversity during the training phase. Residual deterministic-probabilistic fusion introduces uncertainty information while maintaining the accuracy of deterministic semantics, thereby enhancing the model's expressive ability and generalization performance for tasks involving semantic combination, semantic modification, or conditional reasoning (such as combined image retrieval and semantic editing reasoning).

[0131] 3. Significantly improves the training efficiency of multimodal representation learning: This invention simplifies the process of extensive parameter adjustments and repeated optimizations in traditional multimodal pre-training by constructing training data and keeping the parameters of the pre-trained base model frozen during training. Furthermore, by introducing probability distribution constraints during the training phase, the model can achieve cross-modal aligned convergence in fewer iterations.

[0132] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0133] Based on the same inventive concept, one embodiment of the present invention provides a training device for a multimodal reasoning system oriented towards large language models. (Reference) Figure 3 , Figure 3 This is a structural block diagram of a training device for a multimodal reasoning system for large language models, provided in an embodiment of the present invention. Figure 3 As shown, the device includes:

[0134] The first processing module is used to process the sample visual features of the sample reference image and the first sample text features of the sample condition title using a pre-trained large language model to obtain a first embedding representation and a second embedding representation; the sample condition title is used to characterize the modification content of the sample reference image.

[0135] The first loss determination module is used to obtain a first loss value based on the first predicted sample title corresponding to the sample target title and the first embedded representation; the sample target title is obtained based on the sample reference image and the sample condition title;

[0136] The second loss determination module is used to process the second embedding representation through the deep neural network to be trained to obtain a conditional distribution embedding representation and a deterministic embedding representation; and to obtain a second loss value based on the conditional distribution embedding representation and the deterministic embedding representation.

[0137] The third loss determination module is used to fuse the conditional distribution embedding representation and the deterministic embedding representation to obtain a fused embedding representation; and to perform comparative learning based on the fused embedding representation and the second sample text features corresponding to the sample target title to obtain a third loss value.

[0138] The network training module is used to update the parameters of the deep neural network to be trained based on the first loss value, the second loss value and the third loss value, so as to obtain the trained deep neural network.

[0139] The multimodal reasoning module is used to process the reference image and conditional title through a multimodal reasoning system oriented towards a large language model, including the pre-trained large language model and the trained deep neural network, to obtain the target title and / or target image.

[0140] Optionally, the first processing module includes:

[0141] The first input module is used to input multiple tags to be learned, the visual features of the sample, and the text features of the first sample into the pre-trained large language model to obtain a first embedding representation and multiple second embedding representations corresponding to the multiple tags to be learned.

[0142] The second loss determination module includes:

[0143] The second processing module is used to input the plurality of second embedding representations into the fusion module in the deep neural network to be trained, and process the plurality of second embedding representations through the self-attention mechanism in the fusion module to obtain the deterministic embedding representation;

[0144] A distribution parameter determination module is used to input the deterministic embedding representation into the distribution parameter estimation module in the deep neural network to be trained, to obtain a first distribution parameter embedding representation, a second distribution parameter embedding representation, and a third distribution parameter embedding representation; the first distribution parameter embedding representation satisfies the following: semantic alignment with the deterministic embedding representation; the second and third distribution parameter embedding representations are used to capture the semantic uncertainty of the deterministic embedding representation; the semantic uncertainty includes at least one of the following: object-level semantic uncertainty, semantic granularity uncertainty; the more potential objects corresponding to the deterministic embedding representation, the greater the object-level semantic uncertainty; the coarser the semantic expression granularity of the deterministic embedding representation, the greater the semantic granularity uncertainty;

[0145] The third loss determination module includes:

[0146] The first fusion module is used to fuse the first distribution parameter embedding representation with the deterministic embedding representation to obtain the fused embedding representation.

[0147] Optionally, the distributed parameter determination module includes:

[0148] The second input module is used to input the deterministic embedded representation into the first multilayer perceptron in the distribution parameter estimation module to obtain the intermediate embedded representation;

[0149] The third input module is used to input the intermediate embedded representation into the second multilayer perceptron in the distribution parameter estimation module to obtain the first distribution parameter embedded representation.

[0150] The fourth input module is used to input the intermediate embedded representation into the third multilayer perceptron in the distribution parameter estimation module to obtain the second distribution parameter embedded representation;

[0151] The fifth input module is used to input the intermediate embedded representation into the fourth multilayer perceptron in the distribution parameter estimation module to obtain the third distribution parameter embedded representation.

[0152] Optionally, the second loss determination module includes:

[0153] The first part of the determination module is used to obtain a first part of the loss value based on the difference between the deterministic embedding representation and the first distribution parameter embedding representation, the second distribution parameter embedding representation, and the third distribution parameter embedding representation;

[0154] The second part of the determination module is used to obtain the second part of the loss value based on the second distribution parameter embedding representation and the third distribution parameter embedding representation;

[0155] The third part determination module is used to obtain the third part loss value based on the embedding representation of the third distribution parameter;

[0156] The second loss value determination module is used to obtain the second loss value based on the first part of the loss value, the second part of the loss value, and the third part of the loss value.

[0157] Optionally, the device further includes:

[0158] The dimension transformation module is used to perform dimension transformation on the visual features of the sample through the connection layer to be trained, so as to obtain the visual features of the sample after dimension transformation.

[0159] The first processing module includes:

[0160] The third processing module is used to process the visual features of the dimension-transformed sample and the text features of the first sample using the pre-trained large language model;

[0161] The network training module includes:

[0162] The parameter update module is used to update the parameters of the deep neural network to be trained and the connection layer to be trained based on the first loss value, the second loss value and the third loss value, so as to obtain the trained deep neural network and the trained connection layer.

[0163] The multimodal reasoning system includes the pre-trained large language model, the trained deep neural network, and the trained connection layer.

[0164] Optionally, the device further includes:

[0165] The data acquisition module is used to obtain a dataset of image-text pairs, using the image in an image-text pair as the sample reference image and the text in the image-text pair as the sample reference title of the sample reference image.

[0166] The first title determination module is used to input the sample reference image, the sample reference title and the first prompt text into the first large language model to obtain the sample condition title;

[0167] The second title determination module is used to input the sample reference title, the sample condition title, and the second prompt text into the first large language model to obtain the sample target title;

[0168] The sample set construction module is used to take the sample reference image, the sample condition title, and the sample target title as a training sample to obtain a training sample set.

[0169] Optionally, the multimodal reasoning module includes:

[0170] The fourth processing module is used to process the reference image and the conditional title through the multimodal reasoning system to obtain a first target embedding representation and a target fusion embedding representation;

[0171] A multimodal prediction module is configured to obtain the target title based on the first target embedding representation, wherein the target title describes the modified image obtained after modifying the reference image according to the conditional title; and / or, retrieve the target image from an image library based on the target fusion embedding representation.

[0172] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0173] The training device for the multimodal inference system for large language models in this embodiment of the invention can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This embodiment of the invention does not impose specific limitations.

[0174] The training device for the multimodal reasoning system for large language models in this embodiment of the invention can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment of the invention does not impose specific limitations.

[0175] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, such as... Figure 4 As shown, Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a memory, a processor, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps in the training method of the multimodal reasoning system for large language models described in any of the above embodiments of the present invention.

[0176] It should be noted that the electronic devices in the embodiments of the present invention include the mobile electronic devices and non-mobile electronic devices described above.

[0177] Based on the same inventive concept, another embodiment of the present invention provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps in the training method for a multimodal inference system for large language models as described in any of the above embodiments of the present invention. The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0178] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0179] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0180] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0181] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A method for training a large language model oriented multi-modal reasoning system, characterized in that, The method includes: The sample visual features of the sample reference image and the first sample text features of the sample condition title are processed by a pre-trained large language model to obtain a first embedding representation and a second embedding representation; the sample condition title is used to characterize the modification content of the sample reference image. A first loss value is obtained based on the first predicted sample title corresponding to the sample target title and the first embedded representation; the sample target title is obtained based on the sample reference image and the sample condition title. The second embedding representation is processed by the deep neural network to be trained to obtain a conditional distribution embedding representation and a deterministic embedding representation; based on the conditional distribution embedding representation and the deterministic embedding representation, a second loss value is obtained; The conditional distribution embedding representation and the deterministic embedding representation are fused to obtain a fused embedding representation; a third loss value is obtained by comparing and learning the fused embedding representation with the second sample text features corresponding to the sample target title. Based on the first loss value, the second loss value, and the third loss value, the parameters of the deep neural network to be trained are updated to obtain the trained deep neural network. A multimodal reasoning system oriented towards a large language model, including the pre-trained large language model and the trained deep neural network, processes the reference image and conditional title to obtain the target title and / or target image. The visual features of the sample reference image and the first sample text features of the sample condition title are processed by a pre-trained large language model to obtain a first embedding representation and a second embedding representation, including: Multiple tags to be learned, the visual features of the sample, and the text features of the first sample are input into the pre-trained large language model to obtain a first embedding representation and multiple second embedding representations corresponding to the multiple tags to be learned. The second embedding representation is processed by the deep neural network to be trained to obtain a conditional distribution embedding representation and a deterministic embedding representation, including: The plurality of second embedding representations are input into the fusion module of the deep neural network to be trained, and the plurality of second embedding representations are processed by the self-attention mechanism in the fusion module to obtain the deterministic embedding representation; The deterministic embedding representation is input into the distribution parameter estimation module of the deep neural network to be trained to obtain a first distribution parameter embedding representation, a second distribution parameter embedding representation, and a third distribution parameter embedding representation. The first distribution parameter embedding representation satisfies the following: it is semantically aligned with the deterministic embedding representation. The second and third distribution parameter embedding representations are used to capture the semantic uncertainty of the deterministic embedding representation. The semantic uncertainty includes at least one of the following: object-level semantic uncertainty, semantic granularity uncertainty. The more potential objects corresponding to the deterministic embedding representation, the greater the object-level semantic uncertainty. The coarser the semantic expression granularity of the deterministic embedding representation, the greater the semantic granularity uncertainty. By fusing the conditional distribution embedding representation and the deterministic embedding representation, a fused embedding representation is obtained, including: The first distribution parameter embedding representation is fused with the deterministic embedding representation to obtain the fused embedding representation.

2. The method of claim 1, wherein the method is performed by a processor. The deterministic embedding representation is input into the distribution parameter estimation module of the deep neural network to be trained to obtain a first distribution parameter embedding representation, a second distribution parameter embedding representation, and a third distribution parameter embedding representation, including: The deterministic embedded representation is input into the first multilayer perceptron in the distribution parameter estimation module to obtain the intermediate embedded representation; The intermediate embedded representation is input into the second multilayer perceptron in the distribution parameter estimation module to obtain the first distribution parameter embedded representation; The intermediate embedded representation is input into the third multilayer perceptron in the distribution parameter estimation module to obtain the second distribution parameter embedded representation; The intermediate embedded representation is input into the fourth multilayer perceptron in the distribution parameter estimation module to obtain the third distribution parameter embedded representation. 3.The method of claim 1, wherein, Based on the conditional distribution embedding representation and the deterministic embedding representation, a second loss value is obtained, including: Based on the difference between the deterministic embedding representation and the first distribution parameter embedding representation, the second distribution parameter embedding representation, and the third distribution parameter embedding representation, a first part of the loss value is obtained; Based on the second distribution parameter embedding representation and the third distribution parameter embedding representation, the second part of the loss value is obtained; Based on the embedding representation of the third distribution parameter, the third part of the loss value is obtained; The second loss value is obtained based on the first part of the loss value, the second part of the loss value, and the third part of the loss value. 4.The method of claim 1, wherein, The method further includes: The visual features of the sample are transformed by the connection layer to be trained, and the visual features of the sample after dimensional transformation are obtained. The sample visual features of the sample reference image and the first sample text features of the sample conditional title are processed using a pre-trained large language model, including: The pre-trained large language model is used to process the visual features of the dimension-transformed sample and the text features of the first sample. Based on the first loss value, the second loss value, and the third loss value, the parameters of the deep neural network to be trained are updated to obtain a trained deep neural network, including: Based on the first loss value, the second loss value, and the third loss value, the parameters of the deep neural network to be trained and the connection layer to be trained are updated to obtain the trained deep neural network and the trained connection layer. The multimodal reasoning system includes the pre-trained large language model, the trained deep neural network, and the trained connection layer.

5. The method of claim 1 to 4, wherein, The method further includes: Obtain the image-text pair dataset, use the image in an image-text pair as the sample reference image, and use the text in the image-text pair as the sample reference title of the sample reference image; Input the sample reference image, the sample reference title, and the first prompt text into the first large language model to obtain the sample condition title; Input the sample reference title, the sample condition title, and the second prompt text into the first large language model to obtain the sample target title; The sample reference image, the sample condition title, and the sample target title are used as a training sample to obtain a training sample set.

6. The method of claim 1 to 4, wherein the method is performed by a processor. A multimodal reasoning system oriented towards a large language model, comprising the pre-trained large language model and the trained deep neural network, processes the reference image and conditional title to obtain the target title and / or target image, including: The multimodal reasoning system processes the reference image and the conditional title to obtain a first target embedding representation and a target fusion embedding representation. The target title is obtained based on the first target embedding representation, and the target title is used to describe the modified image obtained after modifying the reference image according to the condition title; and / or, the target image is retrieved from the image library based on the target fusion embedding representation.

7. A training device for a large language model-oriented multi-modal reasoning system, characterized in that, The device includes: The first processing module is used to process the sample visual features of the sample reference image and the first sample text features of the sample condition title using a pre-trained large language model to obtain a first embedding representation and a second embedding representation; the sample condition title is used to characterize the modification content of the sample reference image. The first loss determination module is used to obtain a first loss value based on the first predicted sample title corresponding to the sample target title and the first embedded representation; the sample target title is obtained based on the sample reference image and the sample condition title; The second loss determination module is used to process the second embedding representation through the deep neural network to be trained to obtain a conditional distribution embedding representation and a deterministic embedding representation; and to obtain a second loss value based on the conditional distribution embedding representation and the deterministic embedding representation. The third loss determination module is used to fuse the conditional distribution embedding representation and the deterministic embedding representation to obtain a fused embedding representation; and to perform comparative learning based on the fused embedding representation and the second sample text features corresponding to the sample target title to obtain a third loss value. The network training module is used to update the parameters of the deep neural network to be trained based on the first loss value, the second loss value and the third loss value, so as to obtain the trained deep neural network. The multimodal reasoning module is used to process the reference image and conditional title through a multimodal reasoning system oriented towards a large language model, including the pre-trained large language model and the trained deep neural network, to obtain the target title and / or target image. The first processing module includes: The first input module is used to input multiple tags to be learned, the visual features of the sample, and the text features of the first sample into the pre-trained large language model to obtain a first embedding representation and multiple second embedding representations corresponding to the multiple tags to be learned. The second loss determination module includes: The second processing module is used to input the plurality of second embedding representations into the fusion module in the deep neural network to be trained, and process the plurality of second embedding representations through the self-attention mechanism in the fusion module to obtain the deterministic embedding representation; A distribution parameter determination module is used to input the deterministic embedding representation into the distribution parameter estimation module in the deep neural network to be trained, to obtain a first distribution parameter embedding representation, a second distribution parameter embedding representation, and a third distribution parameter embedding representation; the first distribution parameter embedding representation satisfies the following: semantic alignment with the deterministic embedding representation; the second and third distribution parameter embedding representations are used to capture the semantic uncertainty of the deterministic embedding representation; the semantic uncertainty includes at least one of the following: object-level semantic uncertainty, semantic granularity uncertainty; the more potential objects corresponding to the deterministic embedding representation, the greater the object-level semantic uncertainty; the coarser the semantic expression granularity of the deterministic embedding representation, the greater the semantic granularity uncertainty; The third loss determination module includes: The first fusion module is used to fuse the first distribution parameter embedding representation with the deterministic embedding representation to obtain the fused embedding representation.

8. An electronic device, characterized in that, The system includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the training method for a multimodal reasoning system for a large language model as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that, The program or instructions are stored on the readable storage medium, and when the program or instructions are executed by the processor, they implement the steps of the training method for a multimodal reasoning system for a large language model as described in any one of claims 1 to 6.

Images