A method and device for question and answer reasoning on image-text content based on a target model

Abstract

Embodiments of the present specification provide a method for question and answer reasoning of image-text content based on a target model, the target model comprising a plurality of decoding layers, and the image-text content comprising a plurality of images and a plurality of text segments; the method comprising: for a first image, performing first representation compression to obtain L visual compressed representations corresponding to the L decoding layers; for a first text segment, performing second representation compression to obtain L text compressed representations corresponding to the L decoding layers; replacing the images and the text segments in the image-text content with placeholders respectively, and inputting the plurality of decoding layers for decoding operation, comprising: in each decoding layer of the L decoding layers, superimposing the visual compressed representation and the text compressed representation corresponding to the decoding layer on the current hidden representation of the corresponding placeholder, and then performing fusion processing to obtain a decoding representation; and generating an answer text corresponding to the image-text content based on the decoding representation output by the last decoding layer.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method and apparatus for question-and-answer reasoning of text and image content based on a target model. Background Technology

[0002] With the rapid development of artificial intelligence technology, visual language models (VLMs) have demonstrated powerful processing capabilities in single-image understanding and short-text interaction tasks. However, their capabilities are currently mainly limited to processing snapshot-style single images or short texts. For ultra-long, interwoven image and text content with narrative coherence that is widespread in the real world, visual language models still cannot grasp the overall narrative context and perform deep reasoning.

[0003] Ultra-long image-text interleaved content refers to data composed of alternating sequences of images and text. Examples include illustrated instruction manuals, medical records containing patient histories, and narrative descriptions in lengthy videos. This type of data is not a simple stacking of images and text, but rather a complex content where multimodal data is tightly interwoven and mutually corroborating across time series and logical chains. In applications that use ultra-long image-text interleaved content for reasoning, the model not only needs to be able to recognize individual images or understand isolated text, but also needs to possess holistic, narrative-level understanding capabilities to globally integrate and logically connect clues scattered across multiple frames of images and long texts, and then perform reasoning.

[0004] Currently, mainstream visual language models generally adopt the Transformer as their core architecture. Their reasoning ability stems from the self-attention mechanism, which captures long-distance semantic and structural dependencies by calculating the relevance scores between tokens in the input sequence. However, this capability comes at the cost of high computational complexity. The computational and memory usage of the self-attention mechanism increases quadratically with the length n of the input sequence, meaning its computational complexity is O(n^2). This quadratic scaling property causes a sharp increase in the computational resources and time costs required when the model processes extremely long sequences, leading to a decline in model performance and severely impacting its performance on complex tasks.

[0005] For reasoning tasks involving interleaved text and images, especially extremely long interleaved text and images, the aforementioned problems become even more pronounced. On the one hand, an image, after encoding, typically generates a large number of visual units; on the other hand, interleaved text and images also contain a large number of descriptive text segments, which, after encoding, also generate a large number of text units. The combination of these two factors allows the total number of units in interleaved text and images to easily exceed the context window limitations of existing models. Current models often experience memory overflow or performance degradation when processing extremely long contexts of interleaved text and images.

[0006] Several solutions have been proposed in related technologies, but all have significant drawbacks. One approach is input pruning, which shortens the sequence length by discarding some input words. While this reduces computation, it leads to irreversible information loss, especially when the pruned words contain crucial information, severely degrading the model's understanding and reasoning quality. Another approach is to change the model architecture, replacing Transformer-based models with state-space models. State-space models are efficient in handling long sequences, but their reasoning capabilities are weaker than Transformers. Therefore, while this solution improves efficiency, it sacrifices the model's ability to reason accurately, failing to guarantee reasoning quality. Furthermore, related technologies often process data from different modalities in isolation, lacking synergy between images and text, making it difficult to handle reasoning tasks involving interwoven text and images.

[0007] Meanwhile, the field also suffers from deficiencies in its model evaluation system. Existing benchmarks for measuring model progress are mostly designed for single-image or short-context scenarios, lacking effective assessment of holistic, narrative-level reasoning capabilities for extremely long text-image interleaved content. Existing evaluation sample sets for reasoning capabilities in extremely long text-image interleaved content are often composed of short-context datasets, or rely on artificially constructed retrieval tasks (rather than reasoning tasks), or even deliberately lengthen sequences by padding them with irrelevant noisy documents. These evaluation sample sets can only test the model's recall ability for basic facts, failing to truly reflect the model's level of holistic narrative reasoning and comprehensive understanding in extremely long text-image interleaved scenarios, causing model research to deviate from actual needs.

[0008] Therefore, in the field of reasoning with ultra-long interwoven text and image content, we hope to propose a solution that, through technical means, enables the model to achieve multimodal collaborative processing of images and text while retaining the reasoning capabilities of the standard Transformer architecture. This would improve the model's overall understanding of ultra-long contexts, thereby significantly reducing memory overhead and computational load while endowing the model with the ability to perform narrative question-answering reasoning for ultra-long interwoven text and image content, and improving the quality of question-answering reasoning for such content. Summary of the Invention

[0009] This specification describes one or more embodiments of a method and apparatus for question-and-answer reasoning based on a target model for text and image content, which can solve the above-mentioned technical problems and achieve the above-mentioned technical effects.

[0010] According to the first aspect, a method for question-answering reasoning based on a target model for text and image content is provided, wherein the target model includes multiple decoding layers, and the text and image content includes several images and several text segments; the method includes:

[0011] For the first image among the plurality of images, a first representation compression is performed to obtain L visual compressed representations of the first image corresponding to L decoding layers among the plurality of decoding layers.

[0012] For the first text segment among the plurality of text segments, a second representation compression is performed to obtain L text compression representations of the first text segment corresponding to the L decoding layers.

[0013] The images and text segments in the graphic content are replaced with corresponding placeholders, and then input into the multiple decoding layers for decoding operations. The decoding operations include: in each of the L decoding layers, obtaining the current latent representation of the placeholder input to the current decoding layer, superimposing the visual compression representation and text compression representation corresponding to the current decoding layer onto the current latent representation, and then performing a fusion process to obtain the decoded representation.

[0014] Based on the decoded representation output by the last decoding layer, the answer text corresponding to the image and text content is generated.

[0015] According to the second aspect, this specification provides a method for training a target model, the target model including multiple decoding layers, an image compressor, and a text compressor, the method comprising:

[0016] Using an image compressor, for any first image in the image and text training samples, a first representation compression is performed to obtain L visual compressed representations of the first image corresponding to L decoding layers in the plurality of decoding layers.

[0017] Using a text compressor, a second representation compression is performed on any first text segment in the image and text training samples to obtain L compressed text representations of the first text segment corresponding to the L decoding layers.

[0018] The images and text segments in the image-text training samples are replaced with corresponding placeholders and input into the multiple decoding layers for decoding operations. The decoding operations include: in each of the L decoding layers, obtaining the current latent representation of the placeholder input to the current decoding layer, superimposing the visual compressed representation and the text compressed representation corresponding to the current decoding layer onto the current latent representation, and then performing a fusion process to obtain the decoding representation; and generating the first predicted answer based on the decoding representation output by the last decoding layer.

[0019] The multiple decoding layers, image compressor, and text compressor are updated based on the first predicted answer and the label answer corresponding to the image and text training sample.

[0020] According to a third aspect, this specification provides an apparatus for question-answering reasoning based on a target model for text and image content. The target model includes multiple decoding layers, and the text and image content includes several images and several text segments. The apparatus includes:

[0021] The first compression module is configured to perform a first representation compression on the first image among the plurality of images to obtain L visual compressed representations of the first image corresponding to L decoding layers among the plurality of decoding layers.

[0022] The second compression module is configured to perform second representation compression on the first text segment among the plurality of text segments to obtain L text compression representations of the first text segment corresponding to the L decoding layers.

[0023] The decoding module is configured to replace the images and text segments in the graphic content with corresponding placeholders, and input them into the multiple decoding layers for decoding operations. The decoding operations include: in each of the L decoding layers, obtaining the current latent representation of the placeholder input to the current decoding layer, superimposing the visual compression representation and text compression representation corresponding to the current decoding layer onto the current latent representation, and then performing a fusion process to obtain the decoded representation.

[0024] The generation module is configured to generate the answer text corresponding to the image and text content based on the decoding representation output by the last decoding layer.

[0025] According to the fourth aspect, this specification provides an apparatus for training a target model, the target model including multiple decoding layers, an image compressor, and a text compressor, the apparatus comprising:

[0026] An image compressor is configured to perform a first representation compression on any first image in the image-text training samples to obtain L visual compressed representations of the first image corresponding to L decoding layers in the plurality of decoding layers.

[0027] The text compressor is configured to perform a second representation compression on any first text segment in the image and text training samples to obtain L compressed text representations of the first text segment corresponding to the L decoding layers.

[0028] The decoding module is configured to replace the images and text segments in the image-text training samples with corresponding placeholders, and input them into the multiple decoding layers for decoding operations. The decoding operations include: in each of the L decoding layers, obtaining the current latent representation of the placeholder input to the current decoding layer, superimposing the visual compressed representation and the text compressed representation corresponding to the current decoding layer onto the current latent representation, and then performing a fusion process to obtain a decoded representation; and generating a first predicted answer based on the decoded representation output by the last decoding layer.

[0029] The update module is configured to update the multiple decoding layers, image compressor, and text compressor based on the first predicted answer and the label answer corresponding to the image and text training sample.

[0030] According to a fifth aspect, this specification provides a computer program product including a computer program / instructions that, when executed by a processor, implement the steps of the method described in either the first or second aspect.

[0031] According to a sixth aspect, a computing device is provided, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the method described in the first aspect or the second aspect.

[0032] In summary, the methods and apparatus provided in the embodiments of this specification can inject the globally contextual information of images and text segments in the graphic content, after representation compression, into the decoding layer of the target model. This enables the target model to perform reasoning with a complete understanding of the graphic content, while the attention mechanism only performs computation on representation sequences of controllable length. This significantly reduces memory overhead and computational load, while endowing the target model with the ability to perform narrative question-and-answer reasoning on extremely long graphic content, improving the quality of question-and-answer reasoning for graphic content, and thereby generating high-quality answer text that conforms to the semantics of the graphic content. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0034] Figure 1 This specification provides an implementation framework for a method for question-answering reasoning of text and image content based on a target model.

[0035] Figure 2 This is a flowchart illustrating a method for question-and-answer reasoning based on a target model for text and image content, according to an embodiment of this specification.

[0036] Figure 3 This is a flowchart illustrating a method for training a target model according to an embodiment of this specification.

[0037] Figure 4 This is a flowchart of a method for generating training sets and evaluation sets according to embodiments of this specification;

[0038] Figure 5 This is a schematic diagram of a device for question-and-answer reasoning based on a target model for text and image content, according to an embodiment of this specification.

[0039] Figure 6 This is a schematic diagram of an apparatus for training a target model according to an embodiment of this specification. Detailed Implementation

[0040] The solutions provided in the embodiments of this specification will now be described with reference to the accompanying drawings.

[0041] As mentioned earlier, when performing reasoning and question answering on extremely long text-image interleaved content, the model needs to continuously capture the multimodal features and global semantic relationships of images and text across an extremely long context. Simultaneously, it needs to overcome the quadratic computational complexity and memory bottlenecks introduced by the self-attention mechanism to achieve understanding of the complete narrative and logical reasoning. However, among existing technologies, no method can achieve efficient collaborative processing of image and text modal information while preserving the reasoning capabilities of the standard Transformer architecture. This makes it difficult for the model to perform high-quality reasoning and question answering with high computational efficiency and low memory consumption when faced with extremely long text-image interleaved content.

[0042] In view of this, the inventors propose a method for question-answering reasoning based on a target model for text and image content in this specification. This method can effectively shorten the length of the word sequence and reduce the computational complexity of self-attention, while providing global contextual features for the decoding layer of the target model. This overcomes the bottleneck of the model in question-answering reasoning for ultra-long text and image content. Figure 1 A schematic implementation architecture of the method is shown.

[0043] See Figure 1 The graphic content shown in this embodiment includes several images and several text segments arranged in a mixed manner. In practical applications, the graphic content may also include question text for initiating reasoning tasks. This question text can include guiding text content such as system prompts, task instructions, role settings, or format constraints, as needed, to clarify the reasoning direction and output specifications of the target model. It should be noted that... Figure 1 The alternating arrangement of text segments and images in the illustrated text content is merely illustrative and does not constitute a limitation on the organization format of the text and image content. In some embodiments, the combination of text segments and images is highly flexible; for example, multiple consecutive images may be followed by a text segment, or a single image may be interspersed among multiple text segments, and so on. Regardless of the combination method used, as long as the text and image content contains both images and text segments, the question-and-answer reasoning method illustrated in the embodiments of this specification can be applied.

[0044] Continue reading Figure 1 The target model comprises multiple decoding layers. The attached diagram shows L decoding layers, labeled #1, #2, ..., #L. When performing reasoning on the text and image content, several images and text segments within the content can be compressed to obtain compressed representations corresponding to the L decoding layers.

[0045] Specifically, for any image among several images (hereinafter referred to as the first image), a representation compression operation is performed to compress the visual representation generated by encoding the first image into L compressed visual representations. These L compressed visual representations correspond one-to-one with L decoding layers. A compressed visual representation is a compact representation of the visual representation, with a smaller data volume than the original visual representation generated after encoding the image. In other words, through representation compression, the massive number of visual tokens contained in the visual representation of the first image can be compressed into fewer, more information-dense feature representations. Moreover, these L compressed visual representations do not represent unified visual information, but rather are adapted to their respective decoding layers, carrying the visual context features that embody the semantic abstraction capabilities of that decoding layer.

[0046] Similarly, for any text segment among several text segments (hereinafter referred to as the first text segment), by performing a representation compression operation, the text representation generated by encoding the first text segment is compressed into L compressed text representations. These L compressed text representations correspond one-to-one with L decoding layers. A compressed text representation is a compact representation of the text representation, with a smaller data volume than the original text representation generated after encoding the text segment. In other words, through representation compression, the massive number of text tokens contained in the text representation of the first text segment can be compressed into fewer, more information-dense feature representations. Moreover, these L compressed text representations do not represent unified text information, but rather are adapted to their respective decoding layers, carrying the text context features that embody the semantic abstraction capabilities of that decoding layer.

[0047] The above representation compression of images and text segments achieves semantic information compression of visual and text representations. No visual / text lexical units are deleted during the representation compression process. Instead, the representations of the original latent space are extracted into representations with fewer lexical units. While reducing the length of the lexical sequence, the global feature information under multimodal conditions is fully preserved.

[0048] For the input image and text content, the images and text segments can be replaced with corresponding placeholders. For example, the first image can be replaced with the first placeholder, and the first text segment with the second placeholder. These placeholders represent the positions of the original images or text segments in the sequence and can be used to carry the feature information of the corresponding images or text segments. However, the placeholders themselves do not express any features. Thus, the length of the sequence input to the multiple decoding layers of the target model can be shortened by using placeholders. It can be understood that after the replacement, the sequence input to the multiple decoding layers of the target model consists of the question text and a series of placeholders, and its total length is compressed, much smaller than the total number of words in the original image and text content. The image and text content after the placeholder replacement can then be fed into the encoding layer of the target model for feature extraction, providing feature information for the decoding inference of the subsequent multiple decoding layers.

[0049] Continue reading Figure 1 The target model contains multiple decoding layers cascaded in sequence. During the decoding process, the graphic content containing placeholders, which has been processed by the encoding layer, is passed layer by layer and the decoding operation is performed by each decoding layer in sequence to obtain the decoded representation.

[0050] Specifically, when any one of the L decoding layers performs processing, it first obtains the current latent representation of each placeholder input to the current layer. Then, it superimposes the visual compressed representation and text compressed representation corresponding to the current latent representation of the corresponding placeholder. Superposition can be understood as a representation fusion operation, such as injecting the compressed representation obtained after the aforementioned steps into the corresponding placeholder through vector addition, vector concatenation, or other vector fusion methods. After the superposition operation is completed, the current decoding layer obtains the input representation containing global context information. Based on this input representation, the decoding operation is performed to obtain the decoded representation output by the current decoding layer. The decoding operation of the decoding layer can be performed through the self-attention module of the current layer. Since the sequence length of the compressed representation has been controlled within a reasonable range, the sequence length of the input representation input to the decoding layer can also be kept within a controllable range after the compressed representation is injected, thereby avoiding the problem of memory overflow or quadratic increase in computational load during self-attention calculation due to excessively long input sequences. Meanwhile, since the aforementioned representation compression process is not achieved by simply deleting words, but by extracting and compressing features from the original complete visual and text representations, the resulting compressed representation fully preserves the core semantics and global feature information of the original multimodal data. By injecting this fully information-expressed compressed representation into the latent representation of the corresponding placeholder, the decoding layer can obtain untruncated global contextual features that match its level.

[0051] The above process is executed accordingly on each of the L decoding layers, so that each decoding layer can perform decoding operations based on its matching global context feature information, thereby shortening the length of the input sequence, reducing computational overhead, and avoiding the loss of feature information in the input representation.

[0052] Finally, based on the decoded representation output by the last decoding layer, the answer text corresponding to the image and text content is generated. Because the decoding layer obtains complete contextual feature support during the decoding process, the target model is able to perform accurate semantic understanding and logical reasoning based on compressed but complete features, thereby generating high-quality answer text that conforms to the semantics of the image and text content.

[0053] Following the above technical concept, in Figure 2 The diagram illustrates a method for question-answering reasoning based on a target model for text and image content, according to embodiments of this specification. It is understood that the method disclosed in these embodiments can be executed by any device, apparatus, platform, or cluster of devices with computing and processing capabilities.

[0054] See Figure 2In one embodiment, the target model includes multiple decoding layers, and the graphic content includes several images and several text segments. The method includes at least the following steps: Step S201, for a first image among the several images, perform a first representation compression to obtain L visual compressed representations corresponding to L decoding layers among the multiple decoding layers. Step S203, for a first text segment among the several text segments, perform a second representation compression to obtain L text compressed representations corresponding to the L decoding layers. Step S205, replace the images and text segments in the graphic content with corresponding placeholders, and input them into the multiple decoding layers for decoding operations. The decoding operations include: in each of the L decoding layers, obtaining the current latent representation of the placeholder input to the current decoding layer, superimposing the visual compressed representation and text compressed representation corresponding to the current decoding layer onto the current latent representation, and then performing a fusion process to obtain a decoded representation. Step S207, based on the decoded representation output by the last decoding layer, generate the answer text corresponding to the graphic content.

[0055] The following section will combine the implementation architecture and accompanying diagrams of the methods described above to... Figure 2 The specific steps of each method are explained in detail.

[0056] As mentioned earlier, the text and image content refers to question-and-answer prompts that include several images and several text segments, which may include system prompts. ,question and multimodal contexts containing images and text segments. It can be further defined as: ,in, Indicates a text segment, Represents an image; it can be understood that the formula defines a multimodal context. The arrangement and number of images and text segments in the text are examples and not limitations. Therefore, the text and image content described herein... It can be shown as .

[0057] The target model includes multiple decoding layers, and each decoding layer may include a self-attention module to perform decoding operations based on a self-attention mechanism.

[0058] In steps S201 and S203, for the first image among the plurality of images, a first representation compression is performed to obtain L visual compressed representations of the first image corresponding to L decoding layers among the plurality of decoding layers. For the first text segment among the plurality of text segments, a second representation compression is performed to obtain L text compressed representations of the first text segment corresponding to the L decoding layers.

[0059] The first and second representation compression methods can respectively compress the original visual / text representation of an image / text segment into a compressed visual / text representation containing complete feature information but with fewer lexical units. In some implementations, the first and second representation compression can be performed by image / text compressors built based on the Q-Former module. The image / text compressor can receive the input visual / text representation and a set of learnable queries, and output the encoded compressed visual / text representation. The first and second representation compression will be described in detail below.

[0060] The first representation compression performed on any first image among the plurality of images may include: encoding the first image to obtain a first visual representation. In practice, the first image can be input into a visual encoder for feature extraction to obtain the first visual representation. Since the output feature space of the visual encoder may differ from the feature space of the decoding layer of the target model, a projection adapter can be used to map the first visual representation to the feature space of the target model. Specifically, the first visual representation can be dimensionally mapped through the projection adapter to a feature space consistent with the dimension of the decoding layer of the target model. The first visual representation is essentially a sequence of a large number of fine-grained visual words, each visual word corresponding to a high-dimensional representation vector, which fully preserves the local texture details and global spatial structure information of the first image. That is, the first visual representation includes the representation vectors corresponding to a first number of visual words. After obtaining the first visual representation, attention can be calculated on the first visual representation to obtain the L compressed visual representations. Any compressed visual representation includes the representation vectors corresponding to a second number of visual words, where the second number is less than the first number. In practice, the attention computation can be an attention distillation based on a learnable query vector performed on the first visual representation, enabling the target model to extract a compact visual compressed representation that can represent complete feature information from the first visual representation. Since the target model contains L decoding layers, each decoding layer differs in feature abstraction depth and feature focus range. Therefore, in practice, a corresponding visual compressed representation can be generated independently for each decoding layer.

[0061] To improve the efficiency of the first representation compression, a first query matrix corresponding to L decoding layers can be set, allowing the target model to generate all L compressed representations in a single forward propagation. Based on this concept, in a specific practice, the first visual representation can be divided into several visual representation blocks based on a first unit length. The first unit length can be set according to the number of visual words. For example, if the first visual representation contains representation vectors corresponding to a total of 1000 visual words, dividing the first visual representation into units of 25 visual words (first unit length) yields 40 visual representation blocks, each containing representation vectors corresponding to 25 visual words. Next, based on the first query matrix, attention is calculated on each of the visual representation blocks to obtain the L compressed visual representations; the first query matrix includes L first sub-query matrices corresponding one-to-one with the L decoding layers. The above first representation compression process can be represented as follows:

[0062] Set the first image First visual representation is generated through encoding. ,in, It is the feature space dimension of the target model. This refers to the number of visual lexical units contained in the first visual representation. The purpose of first representation compression is to reduce the number of visual lexical units contained in the first visual representation. The first visual representation of the (first quantity mentioned above) visual lexical units is compressed into a form containing The visual compression representation of the first visual representation (the second number mentioned above). Specifically, the first visual representation can be... by Dividing the data into units yields several visual representation blocks. It can be understood that the number of visual representation blocks is... For any visual representation block Based on the first query matrix Perform attention-based representation compression to obtain the compressed representation. .in, This includes L first sub-query matrices, each corresponding one-to-one with one of the L decoding layers. Each first sub-query matrix contains... A query vector, .in, Attention mechanisms will integrate visual representation blocks Key information from each visual lexical unit is aggregated into In the output vector corresponding to each visual word. For a given visual representation block. (image The (3 visual representation blocks), the first representation compression process is as follows:

[0063]

[0064]

[0065] in, Representation layer normalization. This indicates positional encoding. This represents the compressed representation of the output of multi-head attention. It can be further reorganized into , of which piece This corresponds to a visual representation block. of A visual word element, Corresponding to the Lth decoding layer One decoding layer.

[0066] Therefore, it can be known that the number of visual terms contained in the visual compressed representation obtained after the first image is compressed by the first representation (the second number mentioned above) is known. for: .

[0067] The above is a detailed introduction to the first characterization compression. The second characterization compression will be introduced below.

[0068] The second representation compression performed on any first text segment among the plurality of text segments may include: encoding the first text segment to obtain a first text representation. In practice, the first text segment can be input into a text encoder for feature extraction to obtain the first text representation, which includes representation vectors corresponding to a third number of text words. After obtaining the first text representation, attention calculation can be performed on the first text representation to obtain the L compressed text representations, where each compressed text representation includes representation vectors corresponding to a fourth number of text words, where the fourth number is less than the third number. In practice, the attention calculation may be attention distillation based on learnable query vectors performed on the first text representation, enabling the target model to extract a compact compressed text representation that can represent complete semantic information from the first text representation. Since the target model contains L decoding layers, different decoding layers differ in semantic abstraction depth and semantic focus range; therefore, in practice, a corresponding compressed text representation can be generated independently for each decoding layer.

[0069] To improve the efficiency of the second representation compression, a second query matrix corresponding to L decoding layers can be set, allowing the target model to generate all L compressed representations in a single forward propagation. Based on this concept, in a specific practice, the first text representation can be divided into several text representation blocks based on a second unit length. The second unit length can be set according to the number of text words. For example, if the first text representation contains representation vectors corresponding to a total of 1000 text words, dividing the first text representation into units of 25 text words (the second unit length) yields 40 text representation blocks, each containing representation vectors corresponding to 25 text words. Next, based on the second query matrix, attention is calculated on each of the text representation blocks to obtain the L compressed text representations; the second query matrix includes L second sub-query matrices corresponding one-to-one with the L decoding layers. The above-described second representation compression process can be represented as follows:

[0070] Set the first text segment The first text representation is generated through encoding. ,in, It is the dimension of the feature space output by the text encoder. This refers to the number of text terms contained in the first text representation. The purpose of second representation compression is to reduce the number of text terms contained in the first text representation. The first text representation of the (third quantity mentioned above) text units is compressed into a form containing The text compression representation of the first text representation (the fourth number mentioned above). Specifically, the first text representation can be... by By dividing the text into units, we obtain several text representation blocks. It can be understood that the number of text representation blocks is... For any text representation block When necessary (output space dimension of the text encoder) Not equal to the feature space dimension of the target model ), can be used for Perform a mapping to the feature space of the target model, and then based on the second query matrix. Perform attention-based representation compression to obtain the compressed representation. .in, This includes L second sub-query matrices, each corresponding one-to-one with one of the L decoding layers. Each second sub-query matrix contains... A query vector, .in, Attention mechanisms will focus on the text representation blocks Key information from each text word unit is aggregated into In the output vector corresponding to each text word. For a given text representation block. (Text paragraph) The (each text representation block), the second representation compression process is as follows:

[0071]

[0072]

[0073]

[0074] in, It is a way to change the output spatial dimension of a text encoder. Mapped to the feature space dimension of the target model The projection matrix, Representation layer normalization, Indicates position code, Representing multi-head attention, the compressed representation of the output It can be further reorganized into , of which piece This indicates a text representation block. of Each text word element Corresponding to the Lth decoding layer One decoding layer.

[0075] Therefore, we can know the number of text terms contained in the compressed text representation obtained after the first text segment is compressed by the second representation (the fourth number mentioned above). for: .

[0076] The above is a detailed introduction to the second characterization compression.

[0077] After performing first representation compression and second representation compression, generating L visual / text compressed representations corresponding to L decoding layers for each of the images and text segments contained in the image and text content, the target model can use these representations to perform reasoning and question answering on the image and text content.

[0078] Next, returning to the main process, in step S205, the images and text segments in the graphic content are replaced with corresponding placeholders, and the multiple decoding layers are input for decoding operations. The decoding operations include: in each of the L decoding layers, obtaining the current latent representation of the placeholder input to the current decoding layer, superimposing the visual compression representation and text compression representation corresponding to the current decoding layer onto the current latent representation, and then performing fusion processing to obtain the decoded representation.

[0079] As mentioned above, the text and images... It will not be directly processed by the decoding layer of the target model, but will instead be... Replace the images and text segments in the document with their corresponding placeholders. For example, the first image can be replaced with a first placeholder, and the first text segment can be replaced with a second placeholder. For each visual representation block... , Include A visual placeholder for each text representation block. , Include A text placeholder. In this way, the target model can avoid processing lengthy text and image content, and instead process content composed of... , and several placeholders The sequence formed.

[0080] When each of the L decoding layers performs a decoding operation, the hidden representation of the placeholder can be updated by adding the corresponding visual / text compression representation to the placeholder, so that the decoding layer can perform the decoding operation based on the global feature information corresponding to the image and text content.

[0081] Specifically, setting and The first image is shown below. The A visual representation block and the first text segment The The current implicit representation of the placeholders corresponding to each text representation block. The process of updating the implicit representation of the placeholders described above can be represented as:

[0082]

[0083] In other words, the visual compressed representation of the first image corresponding to this decoding layer is vector-added with the current latent representation of the first placeholder; the text compressed representation of the first text segment corresponding to this decoding layer is vector-added with the current latent representation of the second placeholder.

[0084] It should be noted that the above method updates the placeholder's latent representation by adding the latent representation bit by bit, thereby completing the injection of the visual / text compressed representation into the target model's decoding layer. However, the calculation method for updating the latent representation is not limited to this. In practical applications, different calculation methods can be used to complete the injection of the visual / text compressed representation according to specific needs. For example, it can be the vector concatenation of the visual / text compressed representation and the current latent representation of the placeholder, etc.

[0085] The above steps inject the compressed global context information into the decoding layer of the target model, allowing the updated placeholder latent representations to be fed into the self-attention module of the decoding layer. This enables the target model to perform reasoning with a complete understanding of the text and image content, while the attention mechanism only performs computation on representation sequences of controllable length. This significantly reduces memory overhead and computational cost, while endowing the target model with the ability to perform narrative question-answering reasoning on extremely long text and image content, thus improving the quality of question-answering reasoning for text and image content.

[0086] In some applications, the target model contains L of the multiple decoding layers, meaning that the above representation injection process can be applied to each decoding layer of the target model.

[0087] After the above steps are completed and the decoding operation is executed by the multiple decoding layers, in step S207, the answer text corresponding to the graphic content can be generated based on the decoding representation output by the last decoding layer.

[0088] The foregoing describes a method for question-answering reasoning based on a target model for text and image content, according to one or more embodiments. In some embodiments, the target model may include an image compressor for performing a first representation compression, a text compressor for performing a second representation compression, and multiple decoding layers. To effectively train the various components in the target model, this specification also provides a method for training the target model. Figure 3 A flowchart of the method provided according to an embodiment of this specification is shown. It will be understood that the method disclosed in the embodiments of this specification can be executed by any apparatus, device, platform, or cluster of devices with computing and processing capabilities.

[0089] See Figure 3In one embodiment, the target model includes multiple decoding layers, an image compressor, and a text compressor; the method includes at least the following steps: Step S301, using the image compressor, performing a first representation compression on any first image in the image-text training samples to obtain L visual compressed representations of the first image corresponding to L decoding layers in the multiple decoding layers. Step S303, using the text compressor, performing a second representation compression on any first text segment in the image-text training samples to obtain L text compressed representations of the first text segment corresponding to the L decoding layers. Step S305, replacing the images and text segments in the image-text training samples with corresponding placeholders, and inputting them into the multiple decoding layers for decoding operations; the decoding operations include: in each of the L decoding layers, obtaining the current latent representation of the placeholder input to the current decoding layer, superimposing the visual compressed representation and text compressed representation corresponding to the current decoding layer onto the current latent representation, and then performing fusion processing to obtain a decoded representation; generating a first predicted answer based on the decoded representation output by the last decoding layer. Step S307: Update the multiple decoding layers, image compressor, and text compressor according to the first predicted answer and the label answer corresponding to the image and text training sample.

[0090] The following will refer to the accompanying drawings to discuss... Figure 3 The specific steps of each method will be explained in detail. It should be noted that the method for reasoning about text and image content during the training of the target model has been described in detail above, and will not be repeated in the introduction of the various embodiments of the training method.

[0091] Step S301: Using an image compressor, perform first representation compression on any first image in the image and text training samples to obtain L visual compressed representations of the first image corresponding to L decoding layers in multiple decoding layers.

[0092] The image compressor is used to perform a first representation compression to compress the original visual representation obtained after encoding the image into a compact visual compressed representation that retains the complete feature information of the image. In some embodiments, the image compressor can be pre-trained in the following manner:

[0093] A single-image question-answering training sample is input into a first model to generate a third predicted answer. The first model includes the image compressor. The image compressor is updated based on the third predicted answer and the label answer corresponding to the single-image question-answering training sample.

[0094] The pre-training of the image compressor aims to enable it to efficiently distill and compress high-dimensional visual representations into hierarchical, compact representations. During training, a large-scale single-image training dataset can be used as the training corpus. Regarding the model's parameter update strategy, the parameters of the image compressor (e.g., learnable parameters of the Q-Former module) and the parameters of the projection adapter can be updated. The training process can be, for example, as follows: the model receives single-image question-answering training samples, processes them through the image compressor to generate a visually compressed representation, and infers and generates a predicted answer (i.e., the third predicted answer) based on this visually compressed representation. The parameters of the image compressor are then optimized through backpropagation based on the loss (e.g., cross-entropy loss) between the third predicted answer and the true labeled answer (the label corresponding to the question-answering training sample). By performing this training on a large amount of training data, the image compressor can learn how to effectively distill and compress raw, lengthy visual word sequences into compact representations.

[0095] Step S303: Using a text compressor, perform second representation compression on any first text segment in the image and text training samples to obtain L compressed text representations of the first text segment corresponding to the L decoding layers.

[0096] The text compressor is used to perform a second representation compression to compress the original text representation obtained after encoding the text segment into a compact text compressed representation that preserves the semantic integrity of the text segment. In some embodiments, the text compressor can be pre-trained in the following manner:

[0097] The text training sample containing the second text segment is input into the second model to generate the first reconstructed text; the second model contains the text compressor; the text compressor is updated according to the first reconstructed text and the second text segment.

[0098] The pre-training of the text compressor can employ a self-supervised sequence reconstruction training paradigm, aiming to ensure that the text compressor retains the complete semantics of the original text while compressing the length of the text word sequence. During training, a training dataset containing text segments can be selected as the training corpus. Regarding the model's parameter update strategy, the parameters of the text compressor (e.g., encoder parameters, learnable parameters of the Q-Former module) and the projection matrix can be updated. The training process can be, for example, as follows: the model receives a text segment (i.e., the second text segment) from the training samples, processes it through the text compressor to generate a compressed text representation, and reconstructs a text segment (i.e., the first reconstructed text) based on this compressed text representation. By calculating the loss between the first reconstructed text and the second text segment (e.g., cross-entropy loss), backpropagation optimizes the parameters of the text compressor. By performing this training on a large amount of training data, the text compressor can learn how to effectively distill and compress lengthy original text word sequences into compact representations, ensuring that the compressed text representation still fully carries the semantic information of the original text segment.

[0099] Next, returning to the main training process of the target model, in step S305, the images and text segments in the image and text training samples are replaced with corresponding placeholders and input into the multiple decoding layers for decoding operations. The decoding operations include: in each of the L decoding layers, obtaining the current latent representation of the placeholder input to the current decoding layer, superimposing the visual compressed representation and text compressed representation corresponding to the current decoding layer onto the current latent representation, and then performing fusion processing to obtain the decoding representation; generating the first predicted answer based on the decoding representation output by the last decoding layer.

[0100] Step S307: Update the multiple decoding layers, image compressor, and text compressor according to the first predicted answer and the label answer corresponding to the image and text training sample.

[0101] By performing the training process described in steps S301-S307 above, the target model can learn how to simultaneously fuse and infer compressed representations from multiple modalities (images and text segments).

[0102] In some embodiments, to improve the long-context reasoning capability of the target model, in addition to the above-described training, the target model can be further trained using a long text training set. This training process may involve: inputting long text training samples into the target model for target reasoning to generate a second predicted answer; the long text training samples include text segments with more than a preset threshold number of characters. Based on the second predicted answer and the label answer corresponding to the long text training sample, the plurality of decoding layers, image compressor, and text compressor are updated.

[0103] By training the target model on a large training set of long texts, the inference strategy of the target model can be gradually adjusted to an inference paradigm that is adapted to long contexts, ensuring that the target model exhibits robust inference performance in question-answering inference tasks with ultra-long text and image content.

[0104] Reviewing the shortcomings of the related technologies described above, it is clear that there is currently no evaluation set or training set that can effectively evaluate or train the target model. In other embodiments of this specification, methods for generating training sets and evaluation sets for training and evaluating the target model are also provided. Figure 4 A flowchart of the method provided according to an embodiment of this specification is shown. It will be understood that the method disclosed in the embodiments of this specification can be executed by any device, apparatus, platform, or cluster of devices with computing and processing capabilities. The method will be described below in conjunction with embodiments.

[0105] See Figure 4 In step S401, the video to be processed can be frame-stripped to obtain a key frame sequence that satisfies the first constraint; the first constraint includes that the similarity of visual representations between adjacent key frames is lower than a preset threshold.

[0106] Video data (the video to be processed) typically contains a large number of temporally consecutive frames, and adjacent frames often exhibit extremely high visual similarity. Directly using all frames would lead to data redundancy and increased computational burden. Therefore, a frame extraction operation can be used to filter out representative keyframes. The first constraint aims to ensure that the selected keyframes have significant visual differences. During the operation, a visual feature extraction model can be used to encode the frames in the video data to obtain the visual representation of each frame. Subsequently, the similarity between the visual representations of adjacent frames is calculated. When the similarity between two adjacent frames is lower than a preset threshold, it indicates that the scene has changed significantly, and the current frame can be marked as a keyframe and added to the keyframe sequence; otherwise, it is filtered out. This dynamic frame extraction mechanism based on a similarity threshold can effectively remove redundant still images and minor jitter in the video data. The constructed keyframe sequence can also maintain a strict sequential order and logical coherence between frames on the timeline, thereby capturing the core narrative of the video data.

[0107] Step S403: Using the first generation model, generate description text corresponding to each of the key frames in the key frame sequence; or use the first generation model to generate summary text corresponding to the key frame sequence.

[0108] The first generative model can be a large language model with visual understanding capabilities (e.g., ShareGPT4Video). For each keyframe in the keyframe sequence, it is input into the first generative model. By recognizing visual elements such as objects, scenes, and actions within the keyframe, corresponding natural language descriptive text is generated, thus achieving initial alignment from the visual modality to the text modality. To capture the overall semantic context of the video, the entire keyframe sequence can also be input into the first generative model to generate a summary text. This summary text can summarize the main content, core events, or story outline of the video. In this way, semantically rich and temporally rigorous text segment data can be constructed, mimicking the narrative structure of text-image interweaving content in the real world.

[0109] According to one implementation, the descriptive text of a keyframe can be constructed by combining the keyframe title with the description of the transition content between keyframes. For example, for any first keyframe in the keyframe sequence, a first title can be generated using a first model; based on the preceding keyframe of the first keyframe, a first text can be generated using the first model, the first text containing content describing the transition from the preceding keyframe to the first keyframe; combining the first title and the first text yields the descriptive text of the first keyframe.

[0110] Step S405: Generate question-answer text pairs based on the reasoning relationships between the keyframes, including the question text and its corresponding tagged answer.

[0111] Specifically, the reasoning relationship can be the temporal logic, causal relationship, or event evolution relationship between different keyframes in a keyframe sequence. When generating question-and-answer text pairs, it does not rely on explicit information from a single keyframe or isolated text segment, but is generated based on the implicit temporal evolution relationship, interaction logic, causal chain, or narrative change between several keyframes, thereby constructing question-and-answer text pairs that require multiple frames and multiple text segments to be answered.

[0112] Step S407: Arrange the keyframes and their corresponding description texts in chronological order, and combine them with the question text to determine the input data of the image-text training sample; determine the labeled answers as the label data of the image-text training sample.

[0113] According to one implementation, a first generative model can also be used to generate summary text corresponding to the keyframe sequence; and the summary text can be added to the input data of the image-text training samples to enrich the data content in the image-text training samples.

[0114] Through the above steps, a complete image-text training sample can be constructed. The input data simulates multimodal image-text content provided by users in real-world applications, including image-text data interwoven with several images (keyframes) arranged in chronological order and corresponding text segments. The question text serves as prompts for the image-text content reasoning task, requiring the target model to reason and answer based on the multimodal image-text data. The labeled data provides labeled answer text, used to calculate the loss during training to update the target model's parameters, or to measure the target model's reasoning ability during evaluation.

[0115] In this specification, the terms "first" in phrases such as "first characterization compression" and "first text segment," as well as the corresponding terms "second" and "third" (if they exist) in the text, are merely for the convenience of distinction and description and do not have any limiting meaning.

[0116] The foregoing description describes specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than those shown in the embodiments, and the desired result may still be achieved. Furthermore, the processes depicted in the drawings do not necessarily need to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0117] Figure 5 This is a schematic diagram of a device for question-answering reasoning based on a target model for text and image content, according to an embodiment of this specification. The device 500 is deployed in a computing device, which can be implemented using any device, equipment, platform, device cluster, etc., with computing and processing capabilities. This device embodiment is similar to... Figure 2 Corresponding to the method embodiment shown, the target model includes multiple decoding layers, and the graphic content includes several images and several text segments; the device 500 includes:

[0118] The first compression module 501 is configured to perform a first representation compression on the first image among the plurality of images to obtain L visual compressed representations of the first image corresponding to L decoding layers among the plurality of decoding layers.

[0119] The second compression module 502 is configured to perform second representation compression on the first text segment among the plurality of text segments to obtain L text compression representations of the first text segment corresponding to the L decoding layers.

[0120] The decoding module 503 is configured to replace the images and text segments in the graphic content with corresponding placeholders, and input them into the multiple decoding layers for decoding operations. The decoding operations include: in each of the L decoding layers, obtaining the current latent representation of the placeholder input to the current decoding layer, superimposing the visual compression representation and text compression representation corresponding to the current decoding layer onto the current latent representation, and then performing a fusion process to obtain the decoded representation.

[0121] The generation module 504 is configured to generate the answer text corresponding to the graphic content based on the decoding representation output by the last decoding layer.

[0122] Figure 6 This is a schematic diagram of an apparatus for training a target model according to an embodiment of this specification. The apparatus 600 is deployed in a computing device, which can be implemented using any device, equipment, platform, device cluster, etc., with computing and processing capabilities. This apparatus embodiment is similar to... Figure 3 Corresponding to the method embodiment shown, the target model includes multiple decoding layers, an image compressor, and a text compressor; the apparatus 600 includes:

[0123] Image compressor 601 is configured to perform first representation compression on any first image in the image and text training samples to obtain L visual compressed representations of the first image corresponding to L decoding layers in the plurality of decoding layers;

[0124] The text compressor 602 is configured to perform a second representation compression on any first text segment in the image and text training samples to obtain L text compression representations of the first text segment corresponding to the L decoding layers.

[0125] The decoding module 603 is configured to replace the images and text segments in the image-text training samples with corresponding placeholders, and input them into the multiple decoding layers for decoding operations. The decoding operations include: in each of the L decoding layers, obtaining the current latent representation of the placeholder input to the current decoding layer, superimposing the visual compressed representation and the text compressed representation corresponding to the current decoding layer onto the current latent representation, and then performing a fusion process to obtain a decoded representation; and generating a first predicted answer based on the decoded representation output by the last decoding layer.

[0126] The update module 604 is configured to update the multiple decoding layers, image compressor, and text compressor based on the first predicted answer and the label answer corresponding to the image and text training sample.

[0127] This specification also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the methods described in the above embodiments.

[0128] According to another embodiment, this specification also provides a computing device, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the foregoing combination. Figure 2 or Figure 3 The steps of the method are described.

[0129] Those skilled in the art will recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0130] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made based on the technical solutions of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for question-answering reasoning based on a target model for text and image content, wherein the target model includes multiple decoding layers, and the text and image content includes several images and several text segments; the method includes: For the first image among the plurality of images, perform a first representation compression to obtain L visual compressed representations of the first image corresponding to L decoding layers among the plurality of decoding layers; For the first text segment among the plurality of text segments, perform second representation compression to obtain L text compression representations of the first text segment corresponding to the L decoding layers; The images and text segments in the graphic content are replaced with corresponding placeholders, and the data is input into the multiple decoding layers for decoding. The decoding operation includes: in each of the L decoding layers, obtaining the current latent representation of the placeholder input to the current decoding layer, superimposing the visual compression representation and text compression representation corresponding to the current decoding layer onto the current latent representation, and then performing a fusion process to obtain the decoded representation. Based on the decoded representation output by the last decoding layer, the answer text corresponding to the image and text content is generated.

2. The method according to claim 1, wherein, Performing the first characterization compression includes: Encoding the first image yields a first visual representation, which includes the representation vectors corresponding to each of a first number of visual words; Attention is calculated on the first visual representation to obtain the L compressed visual representations; each compressed visual representation includes the representation vector corresponding to a second number of visual lexical units; the second number is less than the first number.

3. The method according to claim 2, wherein, The attention calculation on the first visual representation to obtain the L compressed visual representations includes: The first visual representation is divided into several visual representation blocks based on the first unit length; Based on the first query matrix, attention calculation is performed on the plurality of visual representation blocks respectively to obtain the L visual compressed representations; the first query matrix includes L first sub-query matrices that correspond one-to-one with the L decoding layers.

4. The method according to claim 1, wherein, Performing the second characterization compression includes: Encoding the first text segment yields a first text representation, which includes the representation vectors corresponding to each of the third number of text words; Attention is calculated on the first text representation to obtain the L compressed text representations; each compressed text representation includes the representation vector corresponding to a fourth number of text words; the fourth number is less than the third number.

5. The method according to claim 4, wherein, The attention calculation performed on the first text representation to obtain the L compressed text representations includes: The first text representation is divided into several text representation blocks based on the second unit length; Based on the second query matrix, attention is calculated on each of the several text representation blocks to obtain the L text compression representations; the second query matrix includes L second sub-query matrices that correspond one-to-one with the L decoding layers.

6. The method according to claim 1, wherein, Replace the images and text segments in the aforementioned graphic content with their corresponding placeholders, including: Replace the first image with the first placeholder, and replace the first text segment with the second placeholder; And, the superposition of the visual compressed representation and the text compressed representation corresponding to this decoding layer onto the current latent representation includes: The visual compressed representation of the first image corresponding to this decoding layer is vector-added with the current latent representation of the first placeholder; The first text segment, corresponding to the text compression representation of this decoding layer, is vector-added with the current hidden representation of the second placeholder.

7. The method according to claim 1, wherein, The number of the plurality of decoding layers is L.

8. The method according to claim 1, wherein, The decoding layer includes a self-attention module.

9. The method according to claim 1, wherein, The first / second characterization compression is performed by an image / text compressor built on the Q-Former module, respectively.

10. A method for training a target model, the target model comprising multiple decoding layers, an image compressor, and a text compressor, the method comprising: Using an image compressor, for any first image in the image and text training samples, perform first representation compression to obtain L visual compressed representations of the first image corresponding to L decoding layers in the plurality of decoding layers; Using a text compressor, a second representation compression is performed on any first text segment in the image and text training samples to obtain L compressed text representations of the first text segment corresponding to the L decoding layers; The images and text segments in the image and text training samples are replaced with corresponding placeholders and then input into the multiple decoding layers for decoding. The decoding operation includes: in each of the L decoding layers, obtaining the current latent representation of the placeholder input to the current decoding layer, superimposing the visual compressed representation and the text compressed representation corresponding to the current decoding layer onto the current latent representation, and then performing a fusion process to obtain the decoding representation; generating a first predicted answer based on the decoding representation output by the last decoding layer. The multiple decoding layers, image compressor, and text compressor are updated based on the first predicted answer and the label answer corresponding to the image and text training sample.

11. The method according to claim 10, wherein, After updating the plurality of decoding layers, image compressors, and text compressors, the method further includes: The long text training samples are input into the target model for target inference to generate a second predicted answer; the long text training samples include text segments with more than a preset threshold number of words; The multiple decoding layers, image compressor, and text compressor are updated based on the second predicted answer and the label answer corresponding to the long text training sample.

12. The method according to claim 10, wherein, The image compressor is pre-trained in the following manner: A single-image question-answering training sample is input into a first model to generate a third predicted answer. The first model includes the image compressor. The image compressor is updated based on the third predicted answer and the label answer corresponding to the single-image question answering training sample.

13. The method according to claim 10, wherein, The text compressor is pre-trained in the following manner: The text training sample containing the second text segment is input into the second model to generate the first reconstructed text; the second model includes the text compressor. The text compressor is updated based on the first reconstructed text and the second text segment.

14. The method of claim 10, wherein, The image and text training samples were obtained in the following way: The video to be processed is frame-stripped to obtain a keyframe sequence that satisfies a first constraint; the first constraint includes that the similarity of visual representations between adjacent keyframes is lower than a preset threshold. Using the first generation model, generate descriptive text corresponding to each of the key frames in the key frame sequence; Using the first generation model, a summary text corresponding to the keyframe sequence is generated; Based on the reasoning relationships between the keyframes, question-answer text pairs are generated, including the question text and its corresponding tagged answer. The keyframes and their corresponding descriptive texts are arranged in chronological order and, together with the question text, are determined as the input data for the image-text training sample; the labeled answers are determined as the label data for the image-text training sample.

15. The method of claim 14, further comprising: Using the first generation model, a summary text corresponding to the keyframe sequence is generated; The summary text is added to the input data of the image and text training samples.

16. The method of claim 14, wherein, Using the first generation model, descriptive text corresponding to each of the keyframes in the keyframe sequence is generated, including: For any first keyframe in the keyframe sequence, a first title is generated using the first model; Based on the preceding keyframe of the first keyframe, a first text is generated using a first model, the first text containing content describing the transition from the preceding keyframe to the first keyframe. Combine the first title with the first text to obtain the description text of the first keyframe.

17. An apparatus for question-answering reasoning based on a target model for text and image content, wherein the target model includes multiple decoding layers, and the text and image content includes several images and several text segments; the apparatus includes: The first compression module is configured to perform a first representation compression on the first image among the plurality of images to obtain L visual compressed representations of the first image corresponding to L decoding layers among the plurality of decoding layers; The second compression module is configured to perform second representation compression on the first text segment among the plurality of text segments to obtain L text compression representations of the first text segment corresponding to the L decoding layers; The decoding module is configured to replace the images and text segments in the graphic content with corresponding placeholders, and input them into the multiple decoding layers for decoding operations. The decoding operations include: in each of the L decoding layers, obtaining the current latent representation of the placeholder input to the current decoding layer, superimposing the visual compression representation and text compression representation corresponding to the current decoding layer onto the current latent representation, and then performing a fusion process to obtain the decoded representation. The generation module is configured to generate the answer text corresponding to the image and text content based on the decoding representation output by the last decoding layer.

18. An apparatus for training a target model, the target model including multiple decoding layers, an image compressor, and a text compressor, the apparatus comprising: An image compressor is configured to perform a first representation compression on any first image in the image and text training samples to obtain L visual compressed representations of the first image corresponding to L decoding layers in the plurality of decoding layers; The text compressor is configured to perform a second representation compression on any first text segment in the image and text training samples to obtain L compressed text representations of the first text segment corresponding to the L decoding layers. The decoding module is configured to replace the images and text segments in the image-text training samples with corresponding placeholders and input them into the multiple decoding layers for decoding operations. The decoding operation includes: in each of the L decoding layers, obtaining the current latent representation of the placeholder input to the current decoding layer, superimposing the visual compressed representation and the text compressed representation corresponding to the current decoding layer onto the current latent representation, and then performing a fusion process to obtain the decoding representation; generating a first predicted answer based on the decoding representation output by the last decoding layer. The update module is configured to update the multiple decoding layers, image compressor, and text compressor based on the first predicted answer and the label answer corresponding to the image and text training sample.

19. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-16.

20. A computing device, comprising a memory and a processor, characterized in that, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-16.

Images