Multi-modal information processing method and device, computer device, readable storage medium and program product

Abstract

The application relates to a multi-modal information processing method and device, computer equipment, a readable storage medium and a program product, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring multi-modal information and prompt information; the multi-modal information comprises text content and visual content, and the total context length of the multi-modal information exceeds a preset context length threshold; a multi-modal information processing model is called; the multi-modal information processing model is obtained by pre-training based on the strategy of preferentially retaining text content samples contained in multi-modal training samples, and then optimizing the sampling quantity of time sequence data in visual content samples contained in the multi-modal training samples, and then optimizing the block quantity of image data in the visual content samples; the multi-modal information and the prompt information are input into the multi-modal information processing model, so that the multi-modal information processing model performs context analysis on the multi-modal information based on the prompt information. The method can improve the accuracy of the multi-modal information processing result.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a multimodal information processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of artificial intelligence technology, multimodal information processing has been widely applied in scenarios such as video understanding, image question answering, and document analysis. Multimodal information typically includes textual and visual content, and existing models encode and process input information by setting a pre-defined upper limit for context length. However, in practical applications, the total context length of the multimodal information to be processed often exceeds the model's context length threshold. Examples include long videos, high-resolution images, or combinations of multi-page documents and text inputs. These long-context scenarios place higher demands on the model's encoding and information filtering capabilities.

[0003] Traditional techniques for processing multimodal information exceeding a context length threshold typically employ uniform compression or truncation to reduce the input information to a manageable range for the model. For example, this might involve downsampling visual content at a fixed ratio or randomly truncating or prioritizing text content to enforce context length limitations. However, truncation or compression can lead to the loss of crucial information, preventing the model from accurately understanding user intent or acquiring necessary contextual information, thus reducing the accuracy of multimodal information processing results. Summary of the Invention

[0004] Therefore, it is necessary to provide a multimodal information processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of multimodal information processing results, in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a multimodal information processing method, including:

[0006] Acquire multimodal information and prompts for the multimodal information; wherein the multimodal information includes text content and visual content, and the total context length of the multimodal information exceeds a preset context length threshold;

[0007] Call the pre-trained multimodal information processing model; wherein, the multimodal information processing model is pre-trained based on the strategy of prioritizing the retention of text content samples contained in the multimodal training samples, and first optimizing the number of temporal data samples in the visual content samples contained in the multimodal training samples, and then optimizing the number of image data blocks in the visual content samples.

[0008] Multimodal information and prompts are input into the multimodal information processing model, so that the multimodal information processing model can perform contextual analysis on the multimodal information based on the prompts and output the contextual analysis results corresponding to the multimodal information.

[0009] In one embodiment, the multimodal information processing method further includes:

[0010] Obtain multimodal training samples; wherein, multimodal training samples include text content samples and visual content samples;

[0011] Determine the first context length corresponding to the text content sample, and determine the difference between the upper limit of the context length and the first context length as the second context length corresponding to the visual content sample;

[0012] With the image data in a fixed visual content sample using a baseline number of blocks, the number of samples of temporal data in the visual content sample is determined by using the second context length as a constraint.

[0013] The number of target blocks corresponding to the image data is determined based on the number of samples and the second context length.

[0014] Obtain the temporal feature sequence obtained by sampling the temporal data according to the sampling quantity, and the image sequence feature obtained by dividing the image data into blocks according to the target number of blocks;

[0015] Based on temporal feature sequences, image sequence features, and text feature sequences corresponding to text content samples, the initial multimodal model is trained to obtain a multimodal information processing model.

[0016] In one embodiment, determining the number of target blocks corresponding to the image data based on the number of samples and the second context length includes:

[0017] The third context length corresponding to the time series data is determined based on the number of samples, and the difference between the second context length and the third context length is obtained.

[0018] Obtain the number of multiple candidate blocks, and divide the image data into blocks according to the order of the number of each candidate block, to obtain the fourth context length corresponding to the image data.

[0019] The fourth context length is compared with the difference. If the fourth context length does not exceed the difference, the number of candidate blocks currently selected is determined as the number of target blocks corresponding to the image data.

[0020] In one embodiment, the image data is segmented to obtain the fourth context length corresponding to the image data, including:

[0021] For each frame of the image data, obtain the original area and original aspect ratio of the image;

[0022] Given the current number of candidate blocks, iterate through various combinations of the number of rows and columns in each block.

[0023] Determine the ratio of the area of ​​the image after segmentation to the original area, and the matching degree between the aspect ratio of the image after segmentation and the original aspect ratio, under each combination of the number of rows and columns in the segmentation;

[0024] Based on the ratio and matching degree, the target segmentation scheme of the image is determined;

[0025] The image is divided into blocks according to the target block segmentation scheme to obtain the sub-context length of the image;

[0026] The lengths of each sub-context are summed to obtain the fourth context length corresponding to the image data.

[0027] In one embodiment, obtaining multimodal training samples includes:

[0028] Obtain initial training samples; these include text content samples and visual content samples.

[0029] When the visual content sample contains video data, perform a global annotation operation on the video data to generate a first set of prompt information and a corresponding first set of answers.

[0030] Perform local annotation on the video data to generate a second set of hint information and a corresponding second set of answers;

[0031] The first set of prompt information, the first set of answers, the second set of prompt information, the second set of answers, visual content samples, and text content samples are combined to obtain multimodal training samples.

[0032] In one embodiment, the upper limit of the context length includes multiple candidate upper limits, each corresponding to a model training phase; the multimodal information processing method further includes:

[0033] Sort the multiple candidate upper limit values ​​in ascending order;

[0034] The second context length is determined and the model is trained sequentially based on each candidate upper limit value, following the ascending order. The model parameters obtained in the previous training stage will be used as the initial parameters for the next training stage.

[0035] Secondly, this application also provides a multimodal information processing apparatus, comprising:

[0036] The information acquisition module is used to acquire multimodal information and prompts for the multimodal information; wherein, the multimodal information includes text content and visual content, and the total context length of the multimodal information exceeds a preset context length threshold;

[0037] The model invocation module is used to invoke a pre-trained multimodal information processing model. The multimodal information processing model is pre-trained based on a strategy that prioritizes retaining text content samples contained in the multimodal training samples, and first optimizes the number of temporal data samples in the visual content samples contained in the multimodal training samples, and then optimizes the number of image data blocks in the visual content samples.

[0038] The multimodal information analysis module is used to input multimodal information and prompt information into the multimodal information processing model, so that the multimodal information processing model can perform contextual analysis on the multimodal information based on the prompt information and output the contextual analysis results corresponding to the multimodal information.

[0039] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described multimodal information processing method.

[0040] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described multimodal information processing method.

[0041] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described multimodal information processing method.

[0042] The aforementioned multimodal information processing method, apparatus, computer device, computer-readable storage medium, and computer program product first acquire multimodal information and prompts related to the multimodal information. This multimodal information includes text content and visual content, and the total context length of the multimodal information exceeds a preset context length threshold. Next, a pre-trained multimodal information processing model is invoked. During the training phase, this model prioritizes retaining text content samples from the multimodal training samples and optimizes the number of temporal data samples first, followed by the number of image data blocks in the visual content samples. This training strategy directly determines the model's processing capability during the inference phase. Specifically, when the total context length of the multimodal information to be processed exceeds the preset threshold, the model first retains the complete text content, avoiding information loss caused by truncation or cropping of text in traditional methods. Then, within the limited remaining context length, it prioritizes ensuring the number of temporal data samples to maintain the ability to capture the development of events and temporal relationships. With temporal coverage guaranteed, the spatial details of a single frame image are optimized by adjusting the number of image blocks. This hierarchical optimization mechanism enables the model to retain important information to the maximum extent possible under conditions of limited context length. Finally, based on the prompt information, this multimodal information processing model performs contextual analysis on the multimodal information, thereby obtaining the contextual analysis results corresponding to the multimodal information. In summary, compared with the uniform compression or truncation methods in traditional technologies, this technical solution can significantly improve the output accuracy of the model in scenarios with long contextual multimodal information. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is an application environment diagram of a multimodal information processing method in one embodiment;

[0045] Figure 2 This is a flowchart illustrating a multimodal information processing method in one embodiment;

[0046] Figure 3 This is a schematic diagram of the model training process in one embodiment;

[0047] Figure 4 This is a flowchart illustrating the process of determining the number of image blocks in one embodiment;

[0048] Figure 5This is a schematic diagram of the image segmentation process in one embodiment;

[0049] Figure 6 This is a flowchart illustrating the process of double annotation of a sample in one embodiment;

[0050] Figure 7 This is a schematic diagram of the progressive training process in one embodiment;

[0051] Figure 8 This is a schematic diagram of dual annotations in a specific embodiment;

[0052] Figure 9 This is a structural block diagram of a multimodal information processing device in one embodiment;

[0053] Figure 10 This is an internal structural diagram of a computer device in one embodiment;

[0054] Figure 11 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0057] The multimodal information processing method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, and IoT devices. Server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0058] The multimodal information processing method provided in this application embodiment can be derived from... Figure 1 The process can be executed independently by terminal 102 or server 104, or interactively by terminal 102 and server 104. The following description uses the interactive execution of terminal 102 and server 104. Specifically, the user inputs multimodal information and prompts related to the multimodal information through terminal 102. The multimodal information includes text content and visual content, and the total context length of the multimodal information exceeds a preset context length threshold. After receiving the multimodal information and prompts, server 104 calls a pre-trained multimodal information processing model. This model is pre-trained based on a strategy that prioritizes retaining text content samples included in the multimodal training samples, first optimizing the number of temporal data samples in the visual content samples included in the multimodal training samples, and then optimizing the number of image data blocks in the visual content samples. Server 104 inputs the multimodal information and prompts into the multimodal information processing model, enabling the model to perform context analysis on the multimodal information based on the prompts and output the context analysis results corresponding to the multimodal information. Server 104 can return the context analysis results to terminal 102.

[0059] In one exemplary embodiment, such as Figure 2 As shown, a multimodal information processing method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0060] Step S202: Obtain multimodal information and prompts for the multimodal information; wherein, the multimodal information includes text content and visual content, and the total context length of the multimodal information exceeds a preset context length threshold.

[0061] Multimodal information refers to input information containing at least two different modalities of data. In this embodiment, it specifically includes text content and visual content. Text content refers to text information existing in character encoding form, such as readable text in a document, subtitle text in a video, or at least one of these. Visual content can be images, videos, page views in a multi-page document, or any combination thereof. The amount of information in visual content is usually much greater than that in text content, and its total context length increases dramatically with the increase in image resolution, video duration, or number of document pages. Prompt information refers to specific task instructions or questions posed by the user to the multimodal information, used to clarify the analysis task that the model needs to perform, such as "describe the actions of the characters in this video" or "calculate the values ​​in the table on the third page of the document," or at least one of these. In some embodiments, prompt information can be obtained independently of the text content, or it can be included in the multimodal information as part of the text content. This embodiment does not specifically limit this.

[0062] The total context length refers to the total number of tokens generated after the text and visual content are encoded by the model. A token is the smallest unit of information processed by the model; a piece of text or an image is decomposed into multiple tokens. The total context length is the sum of the number of text content tokens and the number of visual content tokens. A preset context length threshold is a predefined value used to define the boundary between long and short contexts. When the total context length of multimodal information exceeds this threshold, the current input is considered to belong to a long context scenario, and the model needs to adopt an information-first sampling strategy, i.e., prioritizing the preservation of text content, optimizing the number of temporal samples first, and then optimizing the number of image blocks. When the total context length of multimodal information does not exceed this threshold, it can be processed directly in the conventional way. This threshold can be flexibly set according to the actual application scenario. In some embodiments, the context length threshold does not exceed the upper limit of the context length that the model can process in a single inference process.

[0063] For example, the server first receives a prompt from the user through the terminal interface, such as "Please describe the main actions in this video," and simultaneously obtains the multimodal information specified by the user. This multimodal information may include a video lasting several minutes, several high-resolution images, a multi-page document, or any combination thereof. Subsequently, the server calls the tokenization module to convert the prompt and text content into a sequence of text tokens, and the visual content into a sequence of visual tokens, accumulating these to obtain the total context length. The server compares this total context length with a preset context length threshold, which is a pre-defined boundary value used to distinguish between short and long contexts, for example, set to 32K (kilo) tokens. If the total context length does not exceed the threshold, the current input is determined to be a short context scenario. In this case, the model does not need to execute the information-first sampling strategy and can directly perform conventional inference on the multimodal information based on the original prompt. If the total context length exceeds the threshold, the current input is determined to be a long context scenario. In this case, the model triggers the information-first sampling strategy, that is, it prioritizes retaining the text content and first optimizes the sampling quantity of temporal data in the visual content, then optimizes the number of image data blocks in the visual content. It should be noted that when the prompt message is retrieved separately, it should be retained in its entirety along with the text content.

[0064] Step S204: Invoke the pre-trained multimodal information processing model; wherein, the multimodal information processing model is pre-trained based on a strategy of prioritizing the retention of text content samples contained in the multimodal training samples, and first optimizing the number of temporal data samples in the visual content samples contained in the multimodal training samples, and then optimizing the number of image data blocks in the visual content samples.

[0065] The pre-trained multimodal information processing model refers to a visual language model that has been trained and is ready for the inference stage. This model is an improvement on the LLaVA (Large Language and Vision Assistant) architecture, consisting of a visual encoder, a projection layer, and a language model connected sequentially. The visual encoder can employ a SigLIP (Sigmoid Loss for Language Image Pre-Training) model to extract accurate visual features from images. The projection layer can use an MLP (Multilayer Perceptron), which aligns the features extracted by the SigLIP visual encoder to the representation space of the language model, thus achieving effective fusion of multimodal information. The language model can use the Qwen2.5 series, responsible for natural language understanding and generation. This model does not introduce any dedicated modules for compressing visual tokens to maintain its general adaptability to different multimodal tasks. The model has a total of approximately 8 billion parameters and can efficiently process text, high-resolution images, long video sequences, multi-page documents or combinations thereof, and supports video sequences of more than 512 frames and ultra-high-definition image input.

[0066] For example, the server invokes a pre-trained multimodal information processing model. During the model training phase, the server first constructs a massive number of multimodal training samples. For each training sample, a preset upper limit for the context length is read, and the context length of the text content samples is fully preserved. Then, the remaining context length available for visual content is calculated. When allocating the context length budget for visual content, a two-stage degradation strategy is adopted. The first stage fixes the number of blocks for all image data to a baseline value. Under this premise, the number of samples for time-series data is maximized, constrained by the remaining context length. This stage ensures that the model can observe a sufficient number of time points to learn the causal order and dynamic changes of events. The second stage, with the number of samples already determined, subtracts the context length occupied by the time-series data, and then uses the remaining context length budget to determine the number of blocks for the image data, selecting the maximum number of blocks that can be used without exceeding the remaining budget. After training, when faced with multimodal information exceeding long contexts during inference, the model prioritizes retaining all text content, then automatically samples time-series data such as videos or documents, and finally performs block-based downgrading of image data, thereby maximizing information density within a limited context and significantly improving processing accuracy in long context scenarios.

[0067] Step S206: Input the multimodal information and the prompt information into the multimodal information processing model so that the multimodal information processing model can perform contextual analysis on the multimodal information based on the prompt information and output the contextual analysis results corresponding to the multimodal information.

[0068] The context analysis results refer to the natural language text generated by the model, which can be the direct answer to a question, a description of video content, a summary of key points in a document, or structured information (such as tables or lists).

[0069] For example, during the inference phase, the server inputs the obtained prompts and multimodal information into the model. After receiving the input, the model performs contextual analysis on the multimodal information based on the prompts and outputs a complete natural language sequence as the contextual analysis result, such as "between the 12th and 13th seconds of the video, the person raised their right hand." The server then returns this result to the user.

[0070] In this embodiment, multimodal information and prompts related to the multimodal information are first acquired. This multimodal information includes text and visual content, and the total context length of the multimodal information exceeds a preset context length threshold. Next, a pre-trained multimodal information processing model is invoked. During training, this model prioritizes retaining text content samples from the multimodal training samples and optimizes the number of temporal data samples first, followed by the number of image data blocks in the visual content samples. This training strategy directly determines the model's processing capability during the inference phase. Specifically, when the total context length of the multimodal information to be processed exceeds the preset threshold, the model first retains the complete text content, avoiding information loss caused by truncation or cropping of text in traditional methods. Then, within the limited remaining context length, it prioritizes ensuring the number of temporal data samples to maintain the ability to capture the development of events and temporal relationships. With temporal coverage guaranteed, the spatial details of a single frame image are optimized by adjusting the number of image blocks. This hierarchical optimization mechanism allows the model to retain important information to the maximum extent possible under context length constraints. Finally, based on the prompt information, the multimodal information processing model performs contextual analysis on the multimodal information to obtain the contextual analysis results corresponding to the multimodal information. In summary, compared with the uniform compression or truncation methods in traditional technologies, this embodiment can significantly improve the output accuracy of the model in scenarios with long contextual multimodal information.

[0071] In one exemplary embodiment, such as Figure 3 As shown, the model training process includes:

[0072] Step S302: Obtain multimodal training samples; wherein, the multimodal training samples include text content samples and visual content samples.

[0073] In this context, multimodal training samples refer to the raw data units used to train the model. Each sample contains at least one text content sample and one visual content sample. Text content samples can be at least one of the following: text from a document, video subtitles, etc. Visual content samples can be at least one of the following: images, videos, multi-page documents, etc. Training samples typically also include prompt information samples (questions) and corresponding answer samples.

[0074] For example, during the training phase, the server first acquires multimodal training samples, which can come from open-source datasets (such as video question answering datasets, image description datasets, document understanding datasets, etc.) or self-developed datasets. For instance, a training sample may contain a two-minute video (visual content) and the corresponding subtitle text (text content).

[0075] Step S304: Determine the first context length corresponding to the text content sample, and determine the difference between the upper limit of the context length and the first context length as the second context length corresponding to the visual content sample.

[0076] The first context length refers to the number of tokens corresponding to a text content sample. The upper limit of the context length is the maximum total number of tokens the model can process in one training iteration. This upper limit is determined by the model's architecture, memory capacity, and training efficiency, and is usually dynamically set according to the training stage, for example, a smaller value in the initial stage and gradually increased in subsequent stages. The difference is the arithmetic difference between the upper limit of the context length and the first context length. This difference represents the token budget available for visual content samples while preserving the complete text content. The second context length, which is the aforementioned difference, represents the maximum number of tokens that a visual content sample can occupy.

[0077] For example, the server calls the tokenization module to encode the text content sample and calculates the first context length, for example, the text content sample generates two hundred tokens. Simultaneously, it reads the preset upper limit of the context length, for example, the upper limit for the current training phase is 32K tokens. The difference between the upper limit and the first context length is calculated, which is the second context length, representing the maximum token budget that the visual content sample can occupy.

[0078] For example, suppose a training sample is S={ , }, Represents visual content samples. This represents a sample of text content. (Get) The corresponding token length (i.e., the first context length) And the maximum number of tokens the model can process in one training iteration (i.e., the upper limit of the context length). Therefore, calculation The corresponding token length (i.e., the second context length) , .

[0079] Step S306: With the image data in the fixed visual content sample using the baseline number of blocks, the sampling number of temporal data in the visual content sample is determined by using the second context length as a constraint.

[0080] In this context, "fixed image data with a baseline number of blocks" means that when determining the sampling quantity of time-series data, the number of blocks in the image data is fixed at a baseline value. In this embodiment, the baseline number of blocks is set to 1, meaning that each image generates only one overall visual token, i.e., the image is not segmented, thereby minimizing the token budget occupied by the image and allowing as much budget as possible to be allocated to time-series sampling. Time-series data refers to visual content arranged continuously on the timeline, such as at least one of video frame sequences or multi-page document pages. The sampling quantity refers to the actual number of frames or pages selected from the original time-series data. For example, for a ten-second video, the sampling quantity can be set to 20 frames, i.e., two frames per second.

[0081] For example, in determining the number of temporal samples, the number of blocks for all image data is first fixed at a baseline value, meaning each image generates only one overall token. Under this premise, an attempt is made to determine the largest possible number of samples from the original temporal data, such as long videos or multi-page documents, such that the total number of tokens used by the temporal data does not exceed the second context length minus the total number of tokens used by all images under the baseline block. If the number of samples is lower than a preset minimum sampling rate, the sample is discarded. In a specific implementation, the number of samples can be initially calculated based on the preset sampling rate, and then checked whether it exceeds the budget. If it exceeds the budget, the number of samples is reduced proportionally. If the budget is sufficient, the number of samples is maintained or increased. The goal of this process is to maximize the sampling density of the temporal data while using only the minimum token budget for each image, thereby ensuring that the model can observe enough time points to learn the development path and temporal relationships of events.

[0082] In one example, the expression for calculating the number of samples is as follows:

[0083]

[0084] Where 256 is the fixed number of tokens occupied by each temporal unit (one frame or one page), and also the number of tokens occupied by each image under the baseline number of blocks. M represents the number of images. The meaning of this formula is: first, divide the token length corresponding to the visual content sample by the number of tokens per temporal unit to get the theoretical maximum total number of temporal units that can be accommodated; then subtract the number of images, because each image has already occupied the equivalent of one temporal unit's worth of tokens under the baseline number of blocks; finally, round down to obtain the actual number of temporal units that can be sampled.

[0085] Step S308: Determine the number of target blocks corresponding to the image data based on the number of samples and the second context length.

[0086] The target number of blocks refers to the actual number of blocks each image is divided into when processing image data. The target number of blocks determines the degree of spatial detail preservation in the image; the more blocks, the richer the detail, but the more tokens are consumed. Image data refers to static images in visual content samples, including at least one type such as an independent single image, a keyframe in a video, or each page of a multi-page document.

[0087] For example, after the number of samples of time-series data is determined, the total number of tokens actually occupied by the time-series data can be further calculated, i.e. Then use Subtracting this occupancy value yields the remaining token budget available for image data. Select a number of chunks such that the total number of tokens does not exceed the remaining budget as the target number of chunks.

[0088] Step S310: Obtain the temporal feature sequence obtained by sampling the temporal data according to the sampling quantity, and the image sequence feature obtained by dividing the image data into blocks according to the target block quantity.

[0089] Temporal feature sequences refer to the sequence of feature vectors extracted from the original temporal data by selecting specific frames or pages based on the number of samples and using a visual encoder. Each temporal unit generates a fixed-length feature vector. Arranging all the sampled feature vectors in chronological order constitutes the temporal feature sequence. Image sequence features refer to the feature sequence formed by segmenting each image according to the number of target blocks, extracting features from each block using a visual encoder, and arranging the feature vectors of all blocks according to their spatial positions. If there are multiple images, the block features of all images are concatenated sequentially.

[0090] For example, the original temporal data is sampled according to a determined sampling number. Specifically, for video, a specified number of frames can be selected evenly over time; for multi-page documents, a specified number of pages can be selected in page number order. Each selected frame or page is fed into a visual encoder, and each temporal unit generates a fixed-length feature vector. All feature vectors are concatenated in temporal order to form a temporal feature sequence. Simultaneously, each image is divided into blocks according to a determined target number of blocks. Each small block obtains a feature vector through the visual encoder, and the feature vectors of all small blocks are concatenated in spatial order to form a block feature sequence for that image. These two sequences represent information in the temporal dimension and detailed information in the spatial dimension, respectively, and will subsequently be input into the model along with the text feature sequence.

[0091] Step S312: Based on the temporal feature sequence, image sequence features, and text feature sequence corresponding to the text content sample, train the initial multimodal model to obtain the multimodal information processing model.

[0092] The text feature sequence refers to the text vector sequence obtained by transforming text content samples through the word embedding layer of a language model. The text feature sequence retains the semantic and sequential information of the original text content. In some embodiments, the text feature sequence may also include the vector sequences corresponding to the prompt information samples and the answer samples. The initial multimodal model refers to a multimodal model that has not yet been trained. This model has the same architecture (visual encoder, projection layer, language model) as the final model, but its parameters have not yet been optimized.

[0093] For example, the obtained temporal feature sequence, image sequence features, and text feature sequences corresponding to text content samples (usually also including feature sequences of prompt information samples and answer samples) are concatenated into a complete input feature sequence in a predetermined order. The model first aligns the visual features to the language model space through a projection layer, and then performs self-attention calculation through the language model to learn the associations between different modalities. The model generates a predicted output based on the prompt information samples and calculates the loss with the answer samples. The model parameters are updated through the backpropagation algorithm. After iterative optimization with a large number of training samples, the model can automatically execute the optimal sampling and block segmentation strategy without additional rules when facing multimodal information input with long contexts during the inference stage, thereby significantly improving the processing accuracy in long context scenarios.

[0094] In this embodiment, by prioritizing the retention of all tokens in the text content samples and allocating the visual token budget according to the priority of optimizing temporal sampling first and then optimizing image blocks, the trained model can automatically maximize the information density within a limited context length during inference, thereby significantly improving the accuracy of multimodal information processing in long context scenarios.

[0095] In one exemplary embodiment, such as Figure 4 As shown, the number of target blocks corresponding to the image data is determined based on the number of samples and the second context length, including:

[0096] Step S402: Determine the third context length corresponding to the time series data based on the number of samples, and obtain the difference between the second context length and the third context length.

[0097] The third context length refers to the total number of tokens occupied by the time-series data calculated based on the number of samples, which is equal to the number of samples multiplied by the number of tokens fixedly occupied by each time-series unit.

[0098] For example, based on the determined number of samples, the total number of tokens used by the time-series data, i.e., the third context length, is calculated. Then, the second context length is subtracted from the third context length to obtain the difference, which is the remaining token budget available for all image data blocks after retaining the time-series data.

[0099] Step S404: Obtain the number of multiple candidate blocks, and according to the order of the number of each candidate block, perform block processing on the image data in turn based on the number of each candidate block to obtain the fourth context length corresponding to the image data.

[0100] The candidate block count refers to a predefined set of block counts, arranged in descending order (i.e., from high resolution to low resolution). For example, the candidate block count T = {12, 8, 6, 4, 2, 1}. The fourth context length refers to the total number of tokens generated after processing all images using the currently selected candidate block count. The token count for each image equals the number of blocks in that image multiplied by the fixed number of tokens allocated to each block. The fourth context length is obtained by summing the token counts of all images.

[0101] For example, a preset set of candidate block counts is obtained, and each candidate block count is iterated sequentially in descending order. For the currently iterated candidate block count, the block processing of all images in the batch is simulated; that is, each image is uniformly divided into a specified number of blocks, and each block generates a fixed number of tokens through the visual encoder. Therefore, the number of tokens occupied by each image is the number of candidate blocks multiplied by the fixed number of tokens. The token counts of all images are accumulated to obtain the fourth context length.

[0102] Step S406: Compare the fourth context length with the difference. If the fourth context length does not exceed the difference, determine the number of candidate blocks currently selected as the number of target blocks corresponding to the image data.

[0103] For example, the calculated fourth context length is compared with the obtained difference. If the fourth context length does not exceed the difference, it means that the image token consumption under the current candidate block number is within the remaining budget. In this case, the current candidate block number is determined as the target block number, and attempts to use fewer blocks are stopped. If, after traversing all candidate block numbers, even using the minimum block number still exceeds the difference, then the process falls back to adjusting the temporal sampling number or discarding the sample.

[0104] In one example, the expression for calculating the target number of blocks is as follows:

[0105]

[0106] Where T represents the set of candidate blocks, and M represents the number of images. For the i-th image The token function for the number of tokens under the block size t.

[0107] This embodiment is essentially a constrained optimization problem, and its core objective is to... Optimize the number of blocks per image under the constraints. and the number of samples of time series data The objective function expression for this optimization problem is as follows:

[0108]

[0109] The objective function indicates that a set of... and This ensures that the sum of the total number of tokens generated by image segmentation and the total number of tokens generated by temporal sampling is as large as possible.

[0110] The constraints include:

[0111]

[0112]

[0113]

[0114]

[0115] The sum of the number of tokens corresponding to the image data and the number of tokens corresponding to the time-series data cannot exceed the number of tokens corresponding to the visual content sample (i.e., the second context length). The value range of is [1, 12]. The range of values ​​for is [1, ...]. ], This represents the maximum number of samples in the time-series data. When the visual content samples include video, Equal to twice the video duration; when the visual content sample contains a multi-page document, It equals the total number of pages in the document.

[0116] In this embodiment, after fixing the number of temporal samples, the remaining token budget is used to traverse the number of candidate blocks from high to low, and the maximum number of blocks that meet the budget is selected. This maximizes the spatial details of the image while ensuring temporal coverage, effectively improving the information density and processing accuracy of long context multimodal input.

[0117] In one exemplary embodiment, such as Figure 5 As shown, the image data is divided into blocks to obtain the fourth context length corresponding to the image data, including:

[0118] Step S502: For each frame of the image data, obtain the original area and original aspect ratio of the image.

[0119] Wherein, the original area refers to the total number of pixels in the image at its original resolution, which is equal to the width in pixels multiplied by the height in pixels, that is: ,in, The original area is represented by W, where W represents the width in pixels and H represents the height in pixels. The original aspect ratio is the ratio of the image width to the image height, i.e.: , This indicates the original aspect ratio.

[0120] For example, the server reads each image currently being processed, obtains its original width and height pixel values, and calculates the original area and original aspect ratio. These values ​​will serve as a benchmark for selecting subsequent segmentation schemes, measuring the degree of preservation and distortion of the segmented image relative to the original image.

[0121] Step S504: Under the constraint of the number of candidate blocks currently selected, traverse multiple combinations of the number of rows and columns of blocks.

[0122] Among them, the combination of the number of rows and columns in the block refers to the specific way of dividing the image into a number of rows and columns. For example, it can be divided into 12 blocks in the horizontal direction and 12 blocks in the vertical direction. The number of rows and columns can be the same or different.

[0123] For example, based on the current number of candidate blocks, the server first determines the maximum allowed number of blocks for the image, such as 4×4, 6×6, etc. A single frame image supports a maximum block size of 12×12. Next, the server enumerates all combinations of rows and columns that satisfy the condition that the product of the horizontal block number and the vertical block number does not exceed the maximum allowed number of blocks corresponding to the number of candidate blocks. ,in, This indicates the number of blocks the image is divided into horizontally. This indicates the number of blocks into which the image is divided vertically. It refers to the total number of blocks after the image is segmented. This represents the maximum allowed total number of blocks. During the enumeration process, both the horizontal and vertical block counts must be positive integers, and the overall aspect ratio of the divided image should be as close as possible to the original aspect ratio.

[0124] Step S506: Determine the ratio of the area of ​​the image after segmentation to the original area, and the matching degree between the aspect ratio of the image after segmentation and the original aspect ratio, under each combination of the number of rows and columns in the segmentation.

[0125] The image area after segmentation refers to the total pixel area covered by all blocks after dividing the image according to a certain combination of row and column numbers. Since each block has a fixed pixel size, the image area after segmentation is equal to the number of horizontal blocks multiplied by the number of vertical blocks multiplied by the area of ​​a single block, that is: ,in, This represents the area of ​​the image after it has been divided into blocks. This represents the area of ​​a single block. The ratio of the area of ​​the divided image to the original area is... This ratio reflects the extent to which the block covers the original image; the larger the ratio, the more of the original image area is preserved.

[0126] The aspect ratio after segmentation refers to the ratio of the overall width to the height of the image after segmentation, that is: Matching degree refers to the degree of similarity between the aspect ratio of the divided blocks and the original aspect ratio. In this embodiment, the ratio of the two can be calculated, i.e. ,or The smaller value is taken as the matching degree. The closer the matching degree is to 1, the smaller the aspect ratio distortion.

[0127] For example, for each enumerated combination of rows and columns in the block, the total area of ​​the block-formed image is calculated, and the ratio of this area to the original area is calculated. Simultaneously, the overall aspect ratio of the block-formed image is calculated, and its matching degree with the original aspect ratio is also calculated.

[0128] Step S508: Based on the ratio and matching degree, determine the target segmentation scheme for the image.

[0129] The target partitioning scheme refers to the final selected combination of the number of rows and columns in the partitioning.

[0130] For example, for each combination of row and column numbers in the image, the server obtains two key metrics: the ratio of the area of ​​the segmented image to the original area, and the matching degree between the aspect ratio of the segmented image and the original aspect ratio. The matching degree is the smaller of the ratios of the segmented and original aspect ratios. The server compares the area ratio with a preset threshold (e.g., 60%) and takes the smaller of the two as the final determined ratio. Then, it calculates the product between the matching degree and the final ratio. After traversing all candidate combinations, it selects the combination with the largest product value as the target segmentation scheme for the image. If multiple combinations have the same product and are all the maximum value, one of the auxiliary rules can be selected based on the minimum number of blocks or the closest aspect ratio. In one example, the expression for determining the target segmentation scheme is as follows:

[0131]

[0132] The first item is The area of ​​the image after segmentation hour, At this point, the first item takes this ratio, causing the product to become smaller. This is equivalent to penalizing the configuration that doesn't retain enough area, preventing it from being selected due to excessive information being cropped. hour, At this point, the first term is truncated to 0.6, no longer increasing with area, preventing excessive reward for large-area configurations. This allows the model to focus on the second term, avoiding sacrificing aspect ratio due to blindly expanding the area. The second term is... ,when When both ratios are 1, the minimum value is 1. Deviation At any time, regardless Greater than or less than The ratio will decrease symmetrically. This symmetrical attenuation ensures that the penalty for aspect ratio distortion is independent of the distortion direction and only related to the degree of deviation.

[0133] Step S510: Divide the image into blocks according to the target block segmentation scheme to obtain the sub-context length corresponding to the image.

[0134] The sub-context length refers to the total number of tokens generated after the current image is divided into blocks according to the target block segmentation scheme.

[0135] For example, the server uniformly divides the current image into several small blocks according to the number of horizontal and vertical blocks in the target segmentation scheme. Each small block independently extracts features through a visual encoder (such as SigLIP), generating a fixed number of visual tokens. The subcontext length of the image is obtained by summing the tokens generated by all small blocks.

[0136] Step S512: Summarize the lengths of each sub-context to obtain the fourth context length corresponding to the image data.

[0137] For example, each frame of the image data is traversed, each frame is segmented, and the subcontext length of that frame is obtained. Then, all subcontext lengths are summed to obtain the fourth context length.

[0138] In this embodiment, at least 60% of the original area of ​​the image is retained during the segmentation process, and the aspect ratio is kept as consistent as possible with the original image. This maximizes the preservation of the image's integrity and authenticity within a limited visual token budget, avoiding the loss of key information and image distortion caused by traditional fixed segmentation or forced scaling. This helps improve the output accuracy in scenarios with long contextual multimodal information.

[0139] In one exemplary embodiment, such as Figure 6 As shown, multimodal training samples are obtained, including:

[0140] Step S602: Obtain training samples; wherein the training samples include text content samples and visual content samples.

[0141] Training samples refer to raw data units obtained from the dataset that have not yet been labeled or augmented.

[0142] In some embodiments, the open-source datasets cover tasks such as video classification, temporal action localization, video temporal localization, dense video description, temporal segmentation, temporal reasoning, general video response, multi-page documents, video description, and long text. For example, the datasets corresponding to video classification include at least one of the following: Kinetics71, Something-Something-v2, ActivityNet, HACS Segment (Human Action Clips and Segments), COIN (Comprehensive Instructional Video Dataset), HIREST (Hierarchical Retrieval and Step-captioning), FineAction (Fine-Grained Video Dataset for Temporal Action Localization), and PortraitMode-400. The datasets corresponding to temporal action localization include at least one of ActivityNet, HACS Segment, FineAction (Fine-Grained Action Dataset), Ego4D-MQ (Ego4DMulti-Query, First-person Perspective Four-Dimensional Dataset - Multi-Query Subset), COIN, HIREST (Hierarchical Retrievaland STep-captioning, Hierarchical Retrieval and Step-Captioning Dataset), and Perception-Test (Visual Perception Test Dataset).The datasets corresponding to video temporal localization include at least one of the following: Charade-STA (Charade Spatio-Temporal Activity), QVHighlight (Query-based Video HighlightDetection), Ego4D-NIQ (Ego4D Natural InterrogativeQuery), Didemo (Dense Video Captioning and Moment Retrieval), QuerYD (Query-based Video Dataset), MedVidQA (Medical Video Question Answering), Youcook2 (second version of the cooking video dataset), FineVideoParr (fine-grained video parallel understanding dataset), ActivityNet, HACS Segment, FineAction, Ego4D-MQ, COIN, HIREST, Perception-Test, and EgoExoLearn (Ego-Exo Learning Dataset). Datasets for dense video description include at least one of ActivityNet, Youcook2, EgoExoLearn, ViTT (Video Text Tracking), HIREST, COIN, and Breakfast (BreakfastActivity Dataset). Datasets for temporal segmentation include at least one of Breakfast and ViTT. Datasets for temporal reasoning include ActivityNet-RTL (ActivityNet Reasoning over Temporal Length).The datasets corresponding to general video answering include at least one of the following: TVQA (Video Question Answering Dataset), CLEVRER (Comprehensive Language-and-Vision Event Reasoning Dataset), NextQA (Next Video Question Answering Dataset), SportsQA (Sports Video Question Answering Dataset), LLaVA-Video (Large Language and Vision Assistant-Video), FineVideoParr, VideoGPT+ (Video Generation Question Answering Augmentation Dataset), Oops Episode (Unexpected Behavior Video Dataset), Perception-Test, EgoTaskQA (Ego-centric Task Question Answering Dataset), CinePile (Film Clip Dataset), and STAR (Situated Question Answering over Videos). Datasets corresponding to multi-page documents include at least one of the following: SlideVQA (Slide Visual Question Answering), DUDE (Document Understanding Dataset), and MP-DocVQA (Multi-Page Document Visual Question Answering). Datasets corresponding to video descriptions include at least one of the following: ActivityNet, Youcook2, Shot2story (Shot-to-Story Description Dataset), Virpt (Video Script Description Dataset), LLaVA-Video, Momentos (Video Clip Description Dataset), FunQA (Fun Video Question Answering and Description Dataset), S-MIT Monitor (MIT Video Monitoring Dataset), LLaVA-Hound (Image and Text Enhanced Description Dataset), Ego4D-HCap (Ego4D Human Captioning, First-Person Perspective 4D Dataset - Human Description Subset), and EgoExoLearn.The datasets corresponding to long texts include at least one such dataset, such as LongAlign (Long Text Alignment Dataset) and LongReward (Long Text Reward Modeling Dataset).

[0143] For example, the server reads raw training samples from open-source datasets or self-developed datasets. Data sources include the aforementioned open-source datasets and self-developed long-video datasets, covering a wide range of task types from video classification and action localization to long-text understanding, ensuring the model is exposed to rich and diverse visual scenes and text patterns. During the collection process, diversity screening is performed on all candidate videos. That is, the feature vector of each video segment is extracted using a pre-trained CLIP (Contrastive Language-Image Pre-training) model, and the cosine similarity with the features of all videos in the included dataset is calculated one by one. Included videos refer to videos already incorporated into the training sample set. If any included video has a similarity to a candidate video that reaches or exceeds a preset threshold, such as 0.5, the candidate video is considered redundant and discarded; otherwise, it is included in the training sample set. The same screening criterion is followed for self-developed datasets. This strategy avoids overfitting caused by repetitive scenes.

[0144] Step S604: If the visual content sample contains video data, perform a global annotation operation on the video data to generate a first set of prompt information and a corresponding first set of answers.

[0145] Video data consists of continuous frame sequences within visual content samples, typically possessing a clear narrative structure, such as at least one type of film, documentary, or surveillance footage. Global annotation is a top-down annotation method that uses the overall narrative structure of the video as a guide, dividing the video into multiple time segments based on semantic boundaries, generating visual descriptions for each segment, and then summarizing them into question-and-answer data covering the complete video content. The first set of prompts is a set of questions generated by the global annotation operation, with each question relating to the overall plot development, causal relationships, or thematic context of the video. The first set of answers is a set of answers corresponding one-to-one with the questions in the first set of prompts, with the answers derived from the video's global narrative logic.

[0146] For example, when the visual content sample includes video data, the server first invokes a manual or automatic semantic segmentation tool to divide the video into multiple consecutive time segments based on its narrative chapters, such as opening, conflict, climax, and ending, discarding videos with fewer than two time segments to ensure a narrative structure. For each time segment, sampling is performed at a rate of no more than 2 frames per second, with a maximum of 50 frames sampled. Combined with the title of that time segment, a multimodal large model (such as GPT-4o) is invoked to generate a detailed visual description of that time segment. After describing all time segments, the descriptive text of all time segments, the corresponding time intervals, and chapter titles are summarized and provided as input to a plain text large model (such as GPT-4), which is then asked to generate long text question-and-answer pairs that cover the complete video narrative structure. The questions in these question-and-answer pairs constitute the first set of prompt information, and the answers constitute the first set of answers.

[0147] Step S606: Perform local annotation on the video data to generate a second set of prompt information and a corresponding second set of answers. Combine the first set of prompt information, the first set of answers, the second set of prompt information, the second set of answers, the visual content samples, and the text content samples to obtain multimodal training samples.

[0148] The local annotation operation is a bottom-up annotation method that uses short time windows in the video as the basic unit to generate question-answer pairs that focus on specific details within the window, and adds anchor information to the questions. Anchor information includes temporal anchors and contextual anchors. Temporal anchors are explicit time intervals added to the questions, while contextual anchors are background descriptions generated by a multimodal large model that do not contain the answers. The second set of prompts is a set of questions generated by the local annotation operation, with each question focusing on specific content within a continuous time window in the video, such as "What object is the person holding between the fifth and seventh seconds?" The second set of answers is a set of answers that correspond one-to-one with the questions in the second set of prompts, with the answers derived from visual details within the corresponding time window.

[0149] For example, the server processes the same video data in a bottom-up manner. Specifically, multiple consecutive time windows are extracted from the video, with window lengths ranging from a few seconds to tens of seconds. For each time window, sampling is performed at a rate not exceeding 2 frames per second. Several types are randomly selected from a predefined pool of question types (e.g., count, color, action, location, temporal relationship, etc.), for example, five types, guiding the multimodal large model to generate question-and-answer pairs focusing on specific details within that time window. Simultaneously, to localize local information within the overall video, the start and end times (time anchors) of the time window can be proactively added to the generated questions, and the multimodal large model generates a contextual description (context anchor) without the answer, such as "before this, the character had just entered the room," as additional location assistance information. The questions in all generated question-and-answer pairs constitute a second set of prompt information, and the answers constitute a second set of answers.

[0150] Finally, the visual and textual content from the original training samples, along with the generated set of prompts and answers, are combined into a complete multimodal training sample.

[0151] In this embodiment, through a global and local dual-annotation mechanism, the model can simultaneously obtain the overall narrative logic and fine-grained local details of the video, and use time anchors and context anchors to associate segment-level information with the complete video, thereby establishing a multi-level understanding ability from global to local during training, significantly improving the output accuracy in long context multimodal information scenarios.

[0152] In one exemplary embodiment, such as Figure 7 As shown, the upper limit of the context length includes multiple candidate upper limits, each corresponding to a model training phase; the multimodal information processing method also includes:

[0153] Step S702: Sort the multiple candidate upper limit values ​​in ascending order.

[0154] Step S704: Determine the second context length and train the model sequentially based on each candidate upper limit value in ascending order; wherein the model parameters obtained in the previous training stage will be used as the initial parameters of the model in the next training stage.

[0155] Here, the candidate upper limit values ​​are a set of pre-defined upper limits for the context length, such as 32K, 64K, 128K, etc., and each candidate upper limit value corresponds to an independent training phase. Ascending sort means sorting the candidate upper limit values ​​in ascending order.

[0156] For example, the server obtains multiple pre-defined candidate context length upper limits and sorts these candidate values ​​in ascending order to form the execution order of the training phases. Following this order, the current phase's context length upper limit is set as the first (smallest) candidate value in the sorted list, and then the complete model training process is executed: a second context length is calculated based on this upper limit; temporal sampling and image block optimization are performed under the constraint of the second context length; feature sequences are extracted; and model parameters are updated. After the current phase training is completed, the model parameters are saved. Then, the phase corresponding to the next candidate upper limit value (a larger value) is entered. The model parameters saved in the previous phase are loaded as the initial parameters for this phase, and a new second context length is recalculated. Because the upper limit value is larger, the second context length also increases, and sampling and block optimization are re-performed under the new budget constraint, continuing to train the model parameters. This process is repeated until all phases corresponding to all candidate upper limits are completed.

[0157] In this embodiment, by introducing a progressive training strategy, the model first learns to effectively process information in a shorter context, and then gradually expands to a longer context, thereby maintaining stable performance at different lengths, while avoiding the high computational overhead and convergence difficulties caused by using a very large context length at once.

[0158] In one specific embodiment, the multimodal information processing model belongs to the visual-language multimodal model. Based on the LLaVA architecture improvement, it has 8B parameters and can efficiently process extended visual context information, including multiple images, long video sequences, high-resolution media, multi-page documents, or combinations thereof. It supports video sequences of more than 512 frames and ultra-high-definition image input. It is trained through advanced training strategies and data schemes, maintains stable performance under different context lengths, and has a much smaller parameter count than existing multimodal models of the same performance level, such as Qwen2.5-VL-72B and InternVL2.5-78B.

[0159] The core architecture of this model is implemented as follows: The visual encoder uses the SigLIP-so400M model to extract visual features from images, ensuring accurate capture of visual information; the language model is an extension of the Qwen2.5-7B model, serving as the basic language model to guarantee natural language understanding and generation capabilities; an MLP projection layer (input dimension 1024, output dimension 4096) aligns the features extracted by the SigLIP visual encoder with the representation space of the language model, ensuring effective fusion of multimodal information. It supports image block configurations up to 12×12, with a block size of 448×448, and each block corresponds to 256 tokens. The image block strategy supports image processing at arbitrary resolutions, avoiding information loss due to fixed aspect ratio constraints, while avoiding the introduction of a dedicated compression module, maintaining the model's adaptability to different tasks.

[0160] The model's training strategy comprises two core components: information-first sampling and progressive hybrid training, while also including a basic hybrid training scheme and training framework optimization. Information-first sampling ensures the complete preservation of core visual and semantic information, including image region preservation strategies and automatic downsampling strategies. The image region preservation strategy involves dividing the original ultra-high-resolution image into blocks, ensuring that the resulting images... At least retain the original image area 60%, and the proportion of blocks Compared to the original aspect ratio Maintain consistency. For the candidate block proportion set... The optimal configuration can be selected using the following formula:

[0161]

[0162] The formula is in Imposing punishment at the right time, while avoiding punishment Over-rewarding configuration. When At that time, the aspect ratio term reaches its maximum value of 1 and shows a symmetrical decreasing trend.

[0163] Automatic downsampling strategy: Given training samples S={ , }, its maximum sequence length is Calculate the fixed text token length , For 2048, derive a fixed visual token budget. By limiting the visual token budget, the complete text information is ensured to be preserved. When optimizing visual content under constraints, it is divided into two categories, and two key variables are optimized: the number of blocks per image. and the number of samples of time series data .

[0164] The optimization problem is expressed as follows:

[0165]

[0166] The constraints include:

[0167]

[0168]

[0169]

[0170]

[0171] Since the training samples are usually composed of mutually exclusive elements (mainly images or time-series content), this strategy uses a two-stage degradation process to solve the above optimization problem, namely time-series degradation and block-based degradation.

[0172] Temporal degradation: First, the number of image data blocks is fixed at 1, focusing on temporal sampling. The target sampling rate for video is 2 frames / second, and for multi-image documents, all images are used. Each visual input must contain at least [number missing]. A frame, representing the minimum sampling rate, is discarded if this minimum cannot be met within the visual context budget. The maximum number of samples is calculated as follows:

[0173]

[0174] Block Degradation: After determining the maximum number of samples, dynamically adjust the number of blocks to maximize the utilization of available context. Let T = {12, 8, 6, 4, 2, 1} represent the possible block configurations arranged in descending order. Select the highest block configuration that satisfies the following conditions. :

[0175]

[0176] This strategy optimizes the overall learning signal by preserving visual details as much as possible while ensuring the completeness of the text input. This two-stage approach guarantees the complete preservation of text while dynamically adjusting the visual resolution based on the available context, achieving a higher information density compared to static sampling strategies.

[0177] During training, a hybrid training strategy of length-balanced packing is adopted, and combined with the above strategy, the training samples are adaptively adjusted to the current maximum sequence length. It provides a training paradigm that is independent of the number of frames, uniformly optimizes model performance across the entire range of context lengths, and ensures the stability of model performance under different input lengths.

[0178] In addition, regarding the maximum sequence length To address the issues of high computational cost and difficulty in achieving optimal performance in a single training run when the value is large, a stepwise scaling approach is adopted. In this way, sequentially Setting the K values ​​to 32K, 64K, and 128K gradually exposes the model to larger context lengths during training, systematically enhancing the model's ability to handle long contexts, while balancing the distribution of long and short sequences, reducing computational overhead, and safely generating diverse model variants in the middle stages of training.

[0179] This embodiment also optimizes the model's training framework. Specifically, it replaces traditional PyTorch operators with the Triton fusion operator and optimizes GPU (Graphics Processing Unit) memory usage by offloading hidden states from the CPU (Central Processing Unit). Distributed contextual parallelism is implemented based on a two-layer communication group, employing zigzag Llama3-style contextual parallelism to reduce communication latency. Zigzag is a zig-shaped data exchange method derived from the parallel strategy of the Llama 3 model. Fast video metadata (such as duration, frame rate, and encoding format) parsing optimizes video decoding speed and improves training efficiency. During the model inference stage, VLLM (a high-efficiency inference engine for large language models) accelerates inference, significantly reducing memory requirements and improving processing speed.

[0180] In this embodiment, the model's data scheme follows the principle of prioritizing diversity over quality. Data is integrated from open-source datasets and a self-developed dataset. The open-source dataset includes manually annotated data and synthetic video data, specifically designed to improve long-form video understanding. The pre-training corpus contains 11 data categories, covering video classification, temporal action localization, video temporal localization, dense video description, temporal segmentation, temporal reasoning, general video question answering, multi-page documents, video descriptions, and long text. A diversity-driven collection strategy is employed, using multiple video sources and a similarity threshold filtering method—CLIP—to extract video features, calculate the cosine similarity between candidate videos and existing datasets, filter video segments with similarity below 0.5, identify novel segments, maximize content diversity, and avoid data redundancy.

[0181] This embodiment also employs a dual annotation mechanism: top-down story-level annotation (i.e., global annotation) and bottom-up segment-level annotation (i.e., local annotation). Top-down story-level annotation uses manually annotated chapters as meaningful segments, replacing traditional shot-level segmentation and filtering out videos with fewer than two chapters. Each segment is sampled at a rate of up to 2 frames per second, with a maximum of 50 frames sampled. Combined with segment titles, this guides GPT-4o to generate detailed visual descriptions. All segment descriptions, time intervals, and chapter titles are summarized to guide GPT-4 in generating diverse long-text question-and-answer pairs covering various question types, helping the model understand the overall narrative logic of the video. Bottom-up segment-level annotation utilizes GPT-4o to generate focused question-and-answer pairs for short video segments. Five question types are randomly selected from a predefined question type pool, guiding the model to generate corresponding question-and-answer pairs. An anchoring mechanism is also implemented, including temporal anchors and contextual anchors. Temporal anchors directly incorporate time intervals into the question to establish temporal references, while contextual anchors utilize GPT-4o to generate textual contextual anchors. This supplements additional information without revealing the answer, enabling the model to understand both the overall narrative of the video and precise spatiotemporal details, thus facilitating the conversion of segment-level annotations into video-level annotations. For details, please refer to [link / reference]. Figure 8 The bottom-up approach starts with the original video clips, using the GPT-4 model to generate segment-level question-and-answer sessions focusing on local details, and incorporating temporal anchors and texture context generated by GPT-4 as auxiliary localization information. The top-down approach, based on manually annotated story-level clips, first generates corresponding descriptive text, then uses the GPT model to generate story-level question-and-answer sessions covering the complete narrative logic. The results from both approaches are finally integrated through a bidirectional fusion method to form complete training data that includes both overall narrative and fine-grained spatiotemporal information. In one example, the prompts for generating descriptive text and anchors are as follows:

[0182] You are an expert in understanding visual content in video clips. You are requested to create both brief and detailed captions for the current video clip titled "{title}". / / Set the system role, requiring the model to act as a video content understanding expert. The explicit task is to generate both a brief and a detailed description for the current video clip (titled "{title}").

[0183] #### Guidelines For Brief Caption: / / Begin the brief description of the guidelines;

[0184] - Create a concise summary (15-30 words) that captures the essential action, setting, and participants.

[0185] - Focus on the most visually or narratively significant elements of the scene;

[0186] - Use clear, direct language.

[0187] - **IMPORTANT** Treat the video as a complete clip rather than a sequence of frames.

[0188] #### Guidelines For Detailed Caption: / / Begins the detailed guideline section;

[0189] - Begin with a thorough analysis of the visual content shown in the clip.

[0190] - **IMPORTANT** Pay special attention to the progression of actions and movements: / / Important note: Pay special attention to the progress of actions and movements;

[0191] Break down complex actions into their component steps.

[0192] Use transitional words (then, next, afterward, etc.) to show the flow of actions.

[0193] * Describe how one action leads to or connects with the next action;

[0194] * Capture the natural sequence of movements and gestures;

[0195] Note that while the clip may be shown as multiple frames, it should be described as a continuous piece of footage.

[0196] For text that appears clearly in the clip: describe it in its original language and provide an English translation in parentheses. For example: [book in Chinese] [book]. Additionally, explain the meaning of the text within its context.

[0197] **IMPORTANT:** If any text is unclear, partially visible, or too oblurry to read with confidence, simply mention its presence without attempting to specify its content.

[0198] When referring to people, use their characteristics, such as clothing, to distinguish different people.

[0199] - **IMPORTANT** Please provide as many details as possible in your caption, including colors, shapes, and textures of objects, actions and characteristics of humans, as well as scenes and backgrounds.

[0200] - Consider how the visual content provides context and meaning;

[0201] Only output your response in the following format without any additional text, explanations, or notes: / / Output only in the following format, without any additional text, explanations, or notes;

[0202] '''json;

[0203] {"Brief Caption":"concise summary of the video", / / The first field of the JSON object: a brief summary, with an example value of "concise summary of the video";

[0204] "Detailed Caption": "The clip begins with..., progresses by..., and concludes with..."} / / The second field of the JSON object: detailed description, example template description of the beginning, progress, and end; '''.

[0205] In this embodiment, the training parameters of the model are set as follows: batch size of 256, learning rate of 2×10⁻⁶. -4 The training iterations numbered 100,000 steps, utilizing an 8×A100 GPU cluster. Cross-GPU training was achieved through distributed contextual parallelism, keeping single-GPU memory usage below 80GB. Model performance was validated after each iteration, optimizing performance under different context lengths to ensure good performance in short-context tasks. After basic post-mixing training, progressive post-mixing training was performed, first... Increase the batch size to 64K, adjust the batch size to 128, and continue training for 100,000 steps to balance the distribution of long and short sequences, thereby enhancing the model's ability to handle longer contexts. Then... Increase the batch size to 128K, maintain the batch size at 128, and continue training for 100,000 steps until the model converges. Verify model performance after each iteration to ensure the model remains stable under different context lengths, while safely generating diverse model variants during intermediate training phases.

[0206] During the model validation phase, the model's performance was verified through multiple video and image benchmark tests. Table 1 shows the comparison with other models on various video benchmarks. This experiment sampled the video at a default rate of 2 frames per second, disabled the image chunking strategy, and limited the minimum sampling frame count to 8 frames. Table 2 shows the comparison with other models on various image benchmarks. The average score is the average of all benchmark scores, where the OCRbench score needs to be divided by 10.

[0207] Table 1

[0208]

[0209] Among them, LVBench represents the long video understanding benchmark, which evaluates the model's ability to understand long video content (such as plot coherence and chronological events). CG-Bench represents the comprehensive video generation and understanding benchmark, used to test the model's reasoning and description capabilities in complex video scenarios. MMB-Video represents the video evaluation benchmark, a comprehensive evaluation benchmark for video multimodal capabilities, covering tasks such as understanding, question answering, and localization. Video-MME with subtitle is a multimodal evaluation benchmark for videos with subtitles, requiring the model to simultaneously utilize visual frames and subtitle text for question answering. As shown in Table 1, the model in this embodiment achieved comparable or even better results than commercial models such as GPT-4o and Gemini on LVBench (66.4), CG-Bench (55.8), MMB-Video (1.94), and Video-MME (75.7), especially on LVBench, which requires long contextual understanding, significantly exceeding Qwen2.5-VL-8B and LLaVA-Video-8B with the same parameter range.

[0210] Table 2

[0211]

[0212] Among them, DocVQA represents document visual question answering, testing the model's ability to extract information from document images and answer questions. ChartQA represents chart question answering, evaluating the model's ability to understand and reason about chart data such as bar charts, line charts, and pie charts. InfoQA represents information chart question answering, testing the model's ability to answer questions about visually rich infographics such as complex posters and flowcharts. OCRBench represents optical character recognition benchmark, testing the model's ability to recognize text in images, with raw scores ranging from 0 to 1000, and the average score divided by 10 for conversion. TextVQA represents text visual question answering, requiring the model to read text in images (such as road signs and product labels) and perform reasoning. As can be seen from Table 2, the model in this embodiment achieved scores close to or better than current excellent models (such as Qwen2.5-VL-8B and GPT-4o) on DocVQA (94.1), ChartQA (87.5), InfoVQA (80.4), OCRBench (869, i.e., 86.9 on a percentage scale), and TextVQA (83.7). In particular, it reached 869 (about 86.9%) on OCRBench, which exceeded Qwen2.5-VL-8B (864) and GPT-4o (736), indicating that the model has a strong fine-grained visual recognition capability.

[0213] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0214] Based on the same inventive concept, this application also provides a multimodal information processing apparatus for implementing the multimodal information processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more embodiments of the multimodal information processing apparatus provided below can be found in the limitations of the multimodal information processing method described above, and will not be repeated here.

[0215] In one exemplary embodiment, such as Figure 9 As shown, a multimodal information processing device is provided, comprising:

[0216] The information acquisition module 902 is used to acquire multimodal information and prompts for the multimodal information; wherein, the multimodal information includes text content and visual content, and the total context length of the multimodal information exceeds a preset context length threshold.

[0217] Model calling module 904 is used to call a pre-trained multimodal information processing model. The multimodal information processing model is pre-trained based on a strategy that prioritizes retaining text content samples contained in the multimodal training samples, and first optimizes the number of temporal data samples in the visual content samples contained in the multimodal training samples, and then optimizes the number of image data blocks in the visual content samples.

[0218] The multimodal information analysis module 906 is used to input multimodal information and prompt information into the multimodal information processing model, so that the multimodal information processing model can perform contextual analysis on the multimodal information based on the prompt information and output the contextual analysis results corresponding to the multimodal information.

[0219] In one embodiment, the multimodal information processing apparatus further includes a model training module, comprising: a sample acquisition unit for acquiring multimodal training samples; wherein the multimodal training samples include text content samples and visual content samples; a text length determination unit for determining a first context length corresponding to the text content sample, and determining the difference between the upper limit of the context length and the first context length as a second context length corresponding to the visual content sample; a sampling quantity determination unit for determining the sampling quantity of temporal data in the visual content sample, using the second context length as a constraint, while fixing the number of image data segments in the visual content sample; a segmentation quantity determination unit for determining the target number of segments corresponding to the image data based on the sampling quantity and the second context length; a feature acquisition unit for acquiring a temporal feature sequence obtained by sampling the temporal data according to the sampling quantity, and an image sequence feature obtained by segmenting the image data according to the target number of segments; and a model training unit for training an initial multimodal model based on the temporal feature sequence, the image sequence feature, and the text feature sequence corresponding to the text content sample to obtain a multimodal information processing model.

[0220] In one embodiment, the block number determination unit is further configured to: determine the third context length corresponding to the time series data based on the number of samples, and obtain the difference between the second context length and the third context length; obtain multiple candidate block numbers, and divide the image data into blocks according to the order of the candidate block numbers, thereby obtaining the fourth context length corresponding to the image data; compare the fourth context length with the difference, and if the fourth context length does not exceed the difference, determine the currently selected candidate block number as the target block number corresponding to the image data.

[0221] In one embodiment, the block number determination unit is further configured to: for each frame of the image data, obtain the original area and original aspect ratio of the image; under the constraint of the currently selected candidate block number, traverse multiple combinations of block row and column numbers; determine the ratio of the block image area to the original area and the matching degree of the block aspect ratio to the original aspect ratio under each block row and column number combination; determine the target block segmentation scheme of the image based on the ratio and the matching degree; segment the image according to the target block segmentation scheme to obtain the sub-context length corresponding to the image; and summarize the sub-context lengths to obtain the fourth context length corresponding to the image data.

[0222] In one embodiment, the sample acquisition unit is further configured to: acquire initial training samples; wherein the initial training samples include text content samples and visual content samples; if the visual content samples contain video data, perform a global annotation operation on the video data to generate a first set of prompt information and a corresponding first set of answers; perform a local annotation operation on the video data to generate a second set of prompt information and a corresponding second set of answers; and combine the first set of prompt information, the first set of answers, the second set of prompt information, the second set of answers, the visual content samples, and the text content samples to obtain multimodal training samples.

[0223] In one embodiment, the upper limit of the context length includes multiple candidate upper limit values, each candidate upper limit value corresponding to a model training stage; the device is further configured to: sort the multiple candidate upper limit values ​​in ascending order; determine the second context length and perform model training sequentially based on each candidate upper limit value according to the ascending order; wherein the model parameters obtained in the previous training stage will be used as the initial parameters of the model in the next training stage.

[0224] Each module in the aforementioned multimodal information processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0225] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores multimodal information processing data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimodal information processing method.

[0226] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 11As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a multimodal information processing method.

[0227] Those skilled in the art will understand that Figure 10 or Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0228] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0229] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0230] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0231] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0232] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0233] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0234] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multimodal information processing method, characterized in that, The method includes: Acquire multimodal information and prompts for the multimodal information; wherein the multimodal information includes text content and visual content, and the total context length of the multimodal information exceeds a preset context length threshold; Invoke a pre-trained multimodal information processing model; wherein the multimodal information processing model is pre-trained based on a strategy of prioritizing the retention of text content samples contained in the multimodal training samples, and first optimizing the number of temporal data samples in the visual content samples contained in the multimodal training samples, and then optimizing the number of image data blocks in the visual content samples. The multimodal information and the prompt information are input into the multimodal information processing model, so that the multimodal information processing model performs contextual analysis on the multimodal information based on the prompt information and outputs the contextual analysis results corresponding to the multimodal information.

2. The method according to claim 1, characterized in that, The method further includes: Obtain multimodal training samples; wherein, the multimodal training samples include the text content samples and the visual content samples; Determine the first context length corresponding to the text content sample, and determine the difference between the upper limit of the context length and the first context length as the second context length corresponding to the visual content sample; With a fixed number of base blocks for image data in the visual content sample, the number of samples of temporal data in the visual content sample is determined by using the second context length as a constraint. Based on the sampling quantity and the second context length, the number of target blocks corresponding to the image data is determined; Obtain a temporal feature sequence obtained by sampling the temporal data according to the sampling quantity, and an image sequence feature obtained by dividing the image data into blocks according to the target block quantity; Based on the temporal feature sequence, the image sequence features, and the text feature sequence corresponding to the text content sample, the initial multimodal model is trained to obtain the multimodal information processing model.

3. The method according to claim 2, characterized in that, Determining the number of target blocks corresponding to the image data based on the sampling quantity and the second context length includes: The third context length corresponding to the time series data is determined based on the number of samples, and the difference between the second context length and the third context length is obtained. Obtain multiple candidate block counts, and according to the order of the candidate block counts, sequentially divide the image data into blocks based on each candidate block count to obtain the fourth context length corresponding to the image data; The fourth context length is compared with the difference. If the fourth context length does not exceed the difference, the number of candidate blocks currently selected is determined as the number of target blocks corresponding to the image data.

4. The method according to claim 3, characterized in that, The step of segmenting the image data to obtain the fourth context length corresponding to the image data includes: For each frame of the image data, obtain the original area and original aspect ratio of the image; Given the current number of candidate blocks, iterate through various combinations of the number of rows and columns in each block. Determine the ratio of the area of ​​the segmented image to the original area, and the matching degree between the aspect ratio of the segmented image and the original aspect ratio, under each combination of the number of rows and columns of the segmented image; Based on the ratio and the matching degree, the target segmentation scheme of the image is determined; The image is divided into blocks according to the target block segmentation scheme to obtain the sub-context length corresponding to the image; The lengths of each sub-context are summed to obtain the fourth context length corresponding to the image data.

5. The method according to claim 2, characterized in that, The acquisition of multimodal training samples includes: Obtain initial training samples; wherein, the initial training samples include text content samples and visual content samples; If the visual content sample contains video data, a global annotation operation is performed on the video data to generate a first set of prompt information and a corresponding first set of answers; Perform local annotation operations on the video data to generate a second set of prompt information and a corresponding second set of answers; The first set of prompt information, the first set of answers, the second set of prompt information, the second set of answers, the visual content sample, and the text content sample are combined to obtain the multimodal training sample.

6. The method according to claim 2, characterized in that, The upper limit of the context length includes multiple candidate upper limit values, and each candidate upper limit value corresponds to a model training stage. The method further includes: Sort the multiple candidate upper limit values ​​in ascending order; The second context length is determined and the model is trained sequentially based on each candidate upper limit value, according to the ascending sort order; wherein the model parameters obtained in the previous training stage will be used as the initial parameters of the model in the next training stage.

7. A multimodal information processing device, characterized in that, The device includes: An information acquisition module is used to acquire multimodal information and prompts for the multimodal information; wherein the multimodal information includes text content and visual content, and the total context length of the multimodal information exceeds a preset context length threshold. The model invocation module is used to invoke a pre-trained multimodal information processing model. The multimodal information processing model is pre-trained based on a strategy that prioritizes retaining text content samples contained in the multimodal training samples, and first optimizes the number of temporal data samples in the visual content samples contained in the multimodal training samples, and then optimizes the number of image data blocks in the visual content samples. The multimodal information analysis module is used to input the multimodal information and the prompt information into the multimodal information processing model, so that the multimodal information processing model performs contextual analysis on the multimodal information based on the prompt information and outputs the contextual analysis results corresponding to the multimodal information.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images