Multi-modal reasoning task processing method and device based on selection visual token, and storage medium

Abstract

The invention relates to the technical field of artificial intelligence and multi-modal large language models, financial science and technology and medical health, and particularly discloses a multi-modal reasoning task processing method and device based on selection of a visual token, a storage medium and computer equipment, and the method comprises the steps: receiving a to-be-processed image object and a text instruction; encoding the text instruction into a text token sequence, encoding a to-be-processed image object into a plurality of visual tokens, and determining a visual token set according to the visual tokens; an empty visual token subset is initialized, one visual token is iteratively selected from the visual token set every time and added into the visual token subset until a preset condition is met, and during selection every time, determination is carried out based on the semantic coverage increment of the remaining visual tokens; and splicing the selected visual token in the visual token subset with the text token sequence, and executing a multi-modal reasoning task based on a splicing result.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and multimodal large language models, financial technology, and healthcare technology, and in particular to a method and apparatus, storage medium, and computer equipment for processing multimodal reasoning tasks based on the selection of visual tokens. Background Technology

[0002] With the rapid development of multimodal large language model technology, its integration and application in fintech and smart healthcare are deepening. In financial scenarios, models need to process high-definition images containing a large number of charts, invoices, and contract texts to assist in tasks such as risk assessment, compliance detection, and intelligent investment research. In healthcare scenarios, it involves the joint understanding of medical images, reports, and medical records to support assisted diagnosis, disease progression analysis, and medical research. These applications all rely on the model's ability to deeply integrate and reason about visual and textual information. The visual encoder typically converts the input image into a series of visual tokens, which are then input into the model along with text tokens for semantic understanding and decision generation.

[0003] However, existing multimodal inference methods based on visual tokens face significant challenges when dealing with high-resolution images. On the one hand, complex tables, seals, and handwritten annotations in financial documents, as well as subtle lesions, instrument markings, and anatomical structures in medical images, often lead to a sharp increase in the number of visual tokens, resulting in high computational costs and inference latency, making it difficult to meet the stringent efficiency requirements of real-time financial risk control and emergency medical diagnosis. On the other hand, current mainstream methods mostly rely on attention weights to prune and select visual tokens. These methods tend to retain locally salient regions but easily overlook broad semantic connections, leading to problems such as incomplete semantic coverage and loss of contextual information, thereby affecting the accuracy and robustness of the model's judgments. Summary of the Invention

[0004] In view of this, this application provides a multimodal reasoning task processing method and apparatus, storage medium and computer equipment based on the selection of visual tokens. Through the joint optimization mechanism of attention score and semantic coverage increment, it achieves the effect of efficiently and intelligently filtering effective tokens from massive visual tokens, thereby significantly improving the reasoning speed of large models while maintaining or even enhancing their deep semantic understanding ability and decision reliability.

[0005] According to one aspect of this application, a multimodal reasoning task processing method based on the selection of visual tokens is provided, comprising: Receive the image object to be processed and the corresponding text instructions; The text instructions are encoded into a text token sequence, the image object to be processed is encoded into multiple visual tokens, a visual token set is constructed based on the multiple visual tokens, and the attention score corresponding to each visual token is determined; Calculate the semantic similarity between each pair of visual tokens, and initialize an empty subset of visual tokens. Iteratively select one visual token from the set of visual tokens and add it to the subset of visual tokens each time until the number of visual tokens in the subset of visual tokens meets a preset condition. Each time a selection is made, for each remaining visual token in the set of visual tokens, calculate the semantic coverage increment after adding the remaining visual token to the current subset of visual tokens based on the semantic similarity between the remaining visual token and each selected visual token in the current subset of visual tokens. Based on the semantic coverage increment and the corresponding attention score, calculate the joint score of the remaining visual token. Add the remaining visual token with the highest joint score to the current subset of visual tokens and remove it from the set of visual tokens. The selected visual tokens contained in the visual token subset are concatenated with the text token sequence, and a multimodal reasoning task is performed based on the concatenation result to obtain the reasoning result.

[0006] According to another aspect of this application, a multimodal reasoning task processing apparatus based on the selection of visual tokens is provided, comprising: The instruction receiving module is used to receive the image object to be processed and the corresponding text instruction; The encoding module is used to encode the text instructions into a text token sequence, encode the image object to be processed into multiple visual tokens, construct a visual token set based on the multiple visual tokens, and determine the attention score corresponding to each visual token; The visual token selection module is used to calculate the semantic similarity between every two visual tokens, initialize an empty visual token subset, and iteratively select a visual token from the visual token set and add it to the visual token subset each time until the number of visual tokens in the visual token subset meets a preset condition. During each selection, for each remaining visual token in the visual token set, the semantic coverage increment after adding the remaining visual token to the current visual token subset is calculated based on the semantic similarity between the remaining visual token and each selected visual token in the current visual token subset. Based on the semantic coverage increment and the corresponding attention score, the joint score of the remaining visual token is calculated. The remaining visual token with the highest joint score is added to the current visual token subset and removed from the visual token set. The inference module is used to concatenate the selected visual tokens contained in the visual token subset with the text token sequence, and perform a multimodal inference task based on the concatenation result to obtain the inference result.

[0007] According to another aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described multimodal reasoning task processing method based on the selection of a visual token.

[0008] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described multimodal reasoning task processing method based on the selection of visual tokens.

[0009] Using the above technical solution, this application provides a multimodal reasoning task processing method, apparatus, storage medium, and computer device based on the selection of visual tokens. First, it can receive an image object to be processed and its corresponding text instructions. Next, the text instructions are encoded into a text token sequence, and the image object to be processed is divided into multiple image blocks and encoded into a series of visual tokens. These visual tokens constitute a visual token set, and an attention score can be generated for each visual token. Then, the semantic similarity between any two visual tokens is calculated. Simultaneously, an empty subset of visual tokens is initialized to store the selected visual tokens. When iteratively selecting visual tokens from the visual token set, the following operations can be performed each time: For each remaining visual token in the visual token set that has not yet been selected, first, it calculates the extent to which adding this remaining visual token to the current visual token subset increases the semantic coverage of the entire visual token subset to the original image object to be processed (i.e., the semantic coverage increment). Then, this semantic coverage increment is combined with the attention score of the remaining visual token itself to calculate a joint score. Finally, the joint score of each remaining visual token in the visual token set can be calculated. The remaining visual token with the highest joint score is selected and added to the current visual token subset, and then removed from the visual token set. After each addition of a remaining visual token to the current visual token subset, it is checked whether the current visual token subset meets the preset conditions. If the current visual token subset meets the preset conditions, the iteration stops. Finally, the selected visual tokens in the visual token subset obtained through the above optimization are concatenated with the original text token sequence to form the complete input of the multimodal large language model, and the final inference tasks such as question answering, description, or analysis are performed to obtain the final inference result. This application embodiment achieves efficient and intelligent screening of effective tokens from a massive number of visual tokens through a joint optimization mechanism of attention score and semantic coverage increment, thereby significantly improving the inference speed of the large model while maintaining or even enhancing its deep semantic understanding ability and decision reliability.

[0010] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 The diagram illustrates a flowchart of a multimodal reasoning task processing method based on the selection of visual tokens provided in an embodiment of this application. Figure 2 The illustration shows a flowchart of a method for constructing a text token sequence and a visual token set according to an embodiment of this application; Figure 3 This illustration shows a schematic diagram of a multimodal reasoning task processing device based on the selection of visual tokens, according to an embodiment of this application. Figure 4 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0012] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0013] This embodiment provides a multimodal reasoning task processing method based on the selection of visual tokens, such as... Figure 1 As shown, the method includes: Step 101: Receive the image object to be processed and the corresponding text instructions.

[0014] Step 102: Encode the text instructions into a text token sequence, encode the image object to be processed into multiple visual tokens, construct a visual token set based on the multiple visual tokens, and determine the attention score corresponding to each visual token.

[0015] Step 103: Calculate the semantic similarity between every two visual tokens, and initialize an empty subset of visual tokens. Iteratively select one visual token from the set of visual tokens and add it to the subset of visual tokens each time until the number of visual tokens in the subset of visual tokens meets a preset condition. During each selection, for each remaining visual token in the set of visual tokens, calculate the semantic coverage increment after adding the remaining visual token to the current subset of visual tokens based on the semantic similarity between the remaining visual token and each selected visual token in the current subset of visual tokens. Calculate the joint score of the remaining visual token based on the semantic coverage increment and the corresponding attention score. Add the remaining visual token with the highest joint score to the current subset of visual tokens and remove it from the set of visual tokens.

[0016] Step 104: Concatenate the selected visual tokens contained in the visual token subset with the text token sequence, and perform a multimodal reasoning task based on the concatenation result to obtain the reasoning result.

[0017] This application provides a multimodal reasoning task processing method based on selected visual tokens. First, it receives an image object to be processed and a corresponding text instruction. Here, the image object can be an image or a video frame sequence, without limitation. The text instruction can be an instruction indicating what kind of reasoning to perform on the image object, such as "Please help me analyze what's in the image?". For example, in a smart healthcare scenario, the image object to be processed can be a typical lesion image, and the text instruction can be an instruction indicating popular science knowledge describing the typical lesion image. This application embodiment simultaneously receives input from both visual and text modalities for subsequent operations.

[0018] Next, the text instructions are encoded into a sequence of computer-understandable text tokens. This process can be achieved through the word segmenter and embedding layer of a language model. Simultaneously, the image object to be processed is divided into multiple image patches using a visual encoder (such as VisionTransformer) and encoded into a series of visual tokens, forming a visual token set. During processing, the visual encoder can also generate an attention score for each visual token. This attention score reflects the importance of the image patch from the model's perspective; for example, in medical images, visual tokens representing malignant lesions may receive higher attention scores.

[0019] Next, the semantic similarity between any two visual tokens is calculated. Semantic similarity measures how close the two are in the semantic space. For example, in financial document analysis, multiple chart blocks in different locations but all describing debt ratios may have high semantic similarity. Simultaneously, an empty subset of visual tokens is initialized to store the selected visual tokens.

[0020] When iteratively filtering visual tokens from the visual token set, the following operations can be performed at a time: For each remaining visual token in the visual token set that has not yet been selected, firstly, the extent to which adding this remaining visual token to the current visual token subset increases the semantic coverage of the entire visual token subset to the original image object being processed (i.e., semantic coverage increment). This metric ensures that the newly added remaining visual token brings new information that is not fully represented by the existing visual tokens in the current visual token subset. Then, this semantic coverage increment is combined with the attention score of the remaining visual token itself to calculate a joint score. Finally, the joint score of each remaining visual token in the visual token set can be calculated, and the remaining visual token with the highest joint score is selected to be added to the current visual token subset, while the remaining visual token is removed from the visual token set. For example, when processing a medical image containing a tumor core region, a marginal infiltrative region, and normal tissue, this mechanism ensures that after selecting a visual token that clearly shows the tumor core region, subsequent iterations will tend to select remaining visual tokens that cover the unique infiltrative region, rather than redundant remaining visual tokens that are highly similar to the tumor core region.

[0021] After adding a remaining visual token to the current subset of visual tokens each time, you can check whether the current subset of visual tokens meets preset conditions, such as whether the number of existing visual tokens in the current subset meets a preset requirement. If the current subset of visual tokens meets the preset conditions, the iteration stops.

[0022] Finally, the selected visual tokens in the visual token subset obtained through the above optimization are concatenated with the original text token sequence to form the complete input of the multimodal large language model, and the final reasoning tasks such as question answering, description or analysis are performed to obtain the final reasoning result.

[0023] By applying the technical solution of this embodiment, firstly, the image object to be processed and the corresponding text instruction can be received. Next, the text instruction is encoded into a text token sequence, and the image object to be processed is divided into multiple image blocks and encoded into a series of visual tokens. These visual tokens constitute a visual token set, and an attention score can be generated for each visual token. Then, the semantic similarity between any two visual tokens is calculated. Simultaneously, an empty subset of visual tokens is initialized to store the selected visual tokens. When iteratively filtering visual tokens from the visual token set, the following operations can be performed each time: For each remaining visual token in the visual token set that has not yet been selected, firstly, the extent to which adding this remaining visual token to the current visual token subset increases the semantic coverage of the entire visual token subset of the original image object to be processed (i.e., the semantic coverage increment) is calculated. Then, this semantic coverage increment is combined with the attention score of the remaining visual token itself to calculate a joint score. Finally, the joint score of each remaining visual token in the visual token set can be calculated. The remaining visual token with the highest joint score is selected and added to the current visual token subset, and then removed from the visual token set. After each addition of a remaining visual token to the current visual token subset, it is checked whether the current visual token subset meets the preset conditions. If the current visual token subset meets the preset conditions, the iteration stops. Finally, the selected visual tokens in the visual token subset obtained through the above optimization are concatenated with the original text token sequence to form the complete input of the multimodal large language model, and the final inference tasks such as question answering, description, or analysis are performed to obtain the final inference result. This application embodiment achieves efficient and intelligent screening of effective tokens from a massive number of visual tokens through a joint optimization mechanism of attention score and semantic coverage increment, thereby significantly improving the inference speed of the large model while maintaining or even enhancing its deep semantic understanding ability and decision reliability.

[0024] In this embodiment of the application, optionally, step 103, "calculating the semantic coverage increment after adding the remaining visual tokens to the current visual token subset based on the semantic similarity between the remaining visual tokens and each selected visual token in the current visual token subset," includes: obtaining the semantic coverage corresponding to the current visual token subset; adding the remaining visual tokens selected this time to the current visual token subset to form a new visual token subset; traversing each selected visual token in the new visual token subset, and for each traversed selected visual token, calculating the semantic coverage increment after adding the remaining visual tokens to the current visual token subset based on the semantic similarity between the selected visual tokens and each unlabeled visual token in the original visual token set of the image object to be processed. The semantic similarity between visual tokens is determined by filtering target visual tokens from the original set of visual tokens whose semantic similarity to the selected visual tokens is greater than a preset similarity threshold. The number of targets for each target visual token is recorded, and the target visual tokens are marked. After traversal, for each selected visual token, the number of targets is summed to obtain the total semantic coverage. The total semantic coverage is then divided by the total number of all visual tokens of the image object to be processed to obtain the semantic coverage of a new subset of visual tokens. The semantic coverage of the new subset of visual tokens is then subtracted from the semantic coverage of the current subset of visual tokens to obtain the semantic coverage increment for the remaining visual tokens. Accordingly, after "adding the remaining visual token with the highest joint score to the current visual token subset" in step 103, the method further includes: using the semantic coverage corresponding to adding the remaining visual token with the highest joint score to the current visual token subset as the semantic coverage corresponding to the updated current visual token subset.

[0025] In this embodiment, firstly, the semantic coverage corresponding to the current subset of visual tokens is obtained. Here, semantic coverage can be a proportional value used to quantify the extent to which the current subset of visual tokens can represent all the visual information of the original complete image object to be processed. The core idea is that if a region in the original complete image object to be processed is semantically sufficiently similar to any selected visual token in the current subset of visual tokens, then that region is considered to be covered. In the context of smart healthcare, this is similar to assessing whether several key pathological regions of interest (such as cell nuclei and stroma) can adequately represent the global information of the entire biopsy slide image.

[0026] Next, the remaining visual tokens to be evaluated are temporarily added to the current visual token subset, forming a hypothetical new visual token subset. For example, in financial chart analysis, suppose the current visual token subset already contains visual tokens representing "quarterly revenue curve" and "profit margin bar chart", and now it is necessary to evaluate the changes brought about by adding a "cash flow line chart" visual token.

[0027] Then, the process begins to traverse each selected visual token in this new subset of visual tokens. For each selected visual token being traversed, the semantic similarity between them is obtained by checking each unmarked visual token in the original set of visual tokens corresponding to the original image object. Here, semantic similarity is measured by comparing the feature vectors of the two tokens; a higher value indicates greater semantic similarity. Furthermore, all target visual tokens whose similarity to the currently traversed selected visual tokens exceeds a preset similarity threshold are identified, their number is recorded, and these covered visual tokens are marked to prevent duplicate counting when traversing other selected visual tokens in the same new subset of visual tokens. In medical image analysis, this is similar to using a visual token representing a lesion feature to scan the entire image and find all matching, unmarked tissue regions.

[0028] After traversing all selected visual tokens in the new visual token subset, the coverage contributed by each selected visual token is summed to obtain the total semantic coverage of all visual tokens in the original image object that the new visual token subset can cover. Dividing this total semantic coverage by the total number of visual tokens in the original image object yields the semantic coverage of the new visual token subset. This calculation ensures that even if multiple selected visual tokens cover the same area of ​​the original image object, that area is only counted once, thus accurately reflecting the semantic breadth of coverage rather than a simple numerical summation.

[0029] In a specific embodiment, assuming the number of targets corresponding to selected visual token 1 is N1, the number of targets corresponding to selected visual token 2 is N2, and the number of targets corresponding to selected visual token 3 is N3, then the total semantic coverage is N1 + N2 + N3. And assuming the total number of all visual tokens for the image object to be processed is N', then the semantic coverage is (N1 + N2 + N3) / N'.

[0030] Next, the semantic coverage of the new visual token subset is subtracted from the original semantic coverage of the current visual token subset. The difference is the semantic coverage increment brought about by the remaining visual token to be evaluated. This semantic coverage increment intuitively shows how much the semantic representativeness of the entire visual token subset can be improved if this remaining visual token is selected.

[0031] Finally, once the remaining visual token with the highest joint score in this iteration is officially selected and added to the current visual token subset, a state update can be performed. Specifically, the semantic coverage of the new visual token subset corresponding to the previously calculated remaining visual token can be officially set as the semantic coverage of the updated current visual token subset. This provides an accurate baseline for incremental computation in the next iteration, ensuring the consistency of state propagation and computational efficiency throughout the greedy selection process.

[0032] In this embodiment of the application, optionally, the semantic coverage of the new visual token subset is calculated based on the following formula: ; in, This indicates that the remaining visual tokens will be used. v The semantic coverage of the new visual token subset formed after adding it to the current visual token subset S, where V represents the original visual token set of the image object to be processed. This represents the total number of all visual tokens in V. u Represents any visual token in V. s sim( represents any selected visual token in the new subset of visual tokens) u , s ) represents a visual token u With the selected visual token s Semantic similarity between them This indicates the preset similarity threshold.

[0033] In this embodiment, in addition to calculating the semantic coverage of the new visual token subset through the aforementioned traversal method, the semantic coverage of the new visual token subset can also be calculated using the formula above. The numerator in the formula represents the total number of visual tokens in the original visual token set V that have a semantic similarity of at least one selected visual token in the new visual token subset S∪{v} that reaches or exceeds a preset similarity threshold θ. Here, the semantic similarity sim(u,s) measures the proximity of two visual tokens in the feature space, and the preset similarity threshold θ is used to determine whether two visual tokens are semantically sufficient to be considered "similar" or "representative." For example, when analyzing a financial transaction certificate image, if θ is set high, only almost identical numerical regions will be considered to overlap, which helps to finely distinguish key fields such as amount and date.

[0034] The denominator is the size of the original visual token set V, which is the total number of visual tokens obtained after encoding the image object to be processed. Therefore, the semantic coverage calculated by the entire formula is a proportional value between 0 and 1, used to quantify what proportion of the original image object's information the new subset of visual tokens can represent. In the context of smart healthcare, if a lung CT image is encoded with 500 visual tokens, and the new subset of visual tokens can cover 400 of them, then its semantic coverage is 0.8, which means that the subset captures 80% of the semantic content of the original image.

[0035] The core of the formula is to check each original token u. If an s can be found in the new subset of visual tokens, and the two are sufficiently similar (sim(u,s) ≥ θ), then the original token u is included in the coverage area. This mechanism ensures that even if there is a lot of redundant information in the original image object to be processed, as long as there is a selected visual token in the new subset of visual tokens that can represent this kind of information, all these redundant areas will be included in the coverage, thus avoiding information loss.

[0036] Compared to the aforementioned method of calculating semantic coverage by traversing each selected visual token in a new subset of visual tokens and calculating the target number of each selected visual token, the present application embodiment calculates semantic coverage more efficiently using the formula above.

[0037] In this embodiment of the application, optionally, the joint score of the remaining visual tokens is calculated based on the following formula: ; in, Indicates the remaining visual token v The joint score corresponding to the addition to the current visual token subset S. Indicates the remaining visual token v The semantic coverage increment corresponding to adding it to the current visual token subset S, A V Indicates the remaining visual token v The corresponding attention score, Indicates a significant moderating factor.

[0038] In this embodiment, A V Represents the remaining visual token v The corresponding attention score indicates that the image patch is considered important during the initial encoding. It is a preset significance adjustment factor, when When the score is greater than 1, the difference in attention scores can be amplified, thus favoring the selection of visually prominent tokens; when... When <1, this difference will be compressed, thus considering all visual tokens more equally and relying more on semantic coverage increments to make decisions; when When = 1, it is linearly associative. In financial document processing, for an audit report, the visual tokens of the seal and signature areas have extremely high attention scores, which can be adjusted... It allows you to control whether the final selection leans more towards these format-specific salient elements or towards the semantic completeness of covering various data metrics in the report.

[0039] The calculation of joint scores resolves the trade-off between saliency and coverage in visual token pruning. For example, in smart healthcare scenarios, such as rapid screening of mammogram images, this formula ensures that the small number of visual tokens ultimately selected not only effectively capture high-attention lesions like calcifications and lumps, but also comprehensively cover background information crucial for overall diagnosis, such as glandular structures and skin creases. This allows multimodal large models to achieve several times the inference speedup while maintaining a high level of clinical reliability and completeness in the generated auxiliary diagnostic suggestions or report summaries.

[0040] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, a method for constructing a text token sequence and a visual token set is provided, such as... Figure 2 As shown, step 102 includes: Step 102-1: Obtain a predefined text vocabulary, wherein the text vocabulary contains unique identifiers for each preset sub-word unit; segment the text instruction using a preset word segmenter to obtain multiple discrete sub-word units; match each discrete sub-word unit with the text vocabulary to find the unique identifier corresponding to the sub-word unit in the text vocabulary, and map the unique identifier of each sub-word unit to a fixed-dimensional text embedding vector using an embedding vector lookup table stored in a preset text embedding layer; arrange the text embedding vectors corresponding to each sub-word unit according to the order of the sub-word units in the text instruction to form the text token sequence.

[0041] In this embodiment, firstly, a predefined text vocabulary is obtained. This text vocabulary may contain hundreds of thousands of preset "sub-word units" and their corresponding unique identifiers. Sub-word units refer to the foundation of modern large language models for text processing; they can be complete words (such as "cell"), or common prefixes, suffixes, or high-frequency word roots. In the context of smart healthcare, this text vocabulary can cover a large number of professional medical terms, such as "malignant tumor," "CT image," and "pathological staging," ensuring accurate recognition and parsing of users' professional query commands.

[0042] Next, a pre-defined word segmenter is used to segment the user-input text command. The word segmenter, based on a text vocabulary and complex statistical rules, can break down continuous natural language sentences into a series of discrete, model-recognizable sub-word units. For example, in the fintech field, for the text command "Please analyze the company's fourth-quarter cash flow and debt-to-equity ratio," the pre-defined word segmenter might decompose it into ["please," "analyze," "the," "company," "fourth," "quarter," "of," "cash flow," "and," "asset," "debt ratio"]. This process solves the problem that the model cannot directly understand the raw string and transforms it into a structured input basis.

[0043] Then, for each sub-word unit obtained from the preset word segmentation, it is matched against the text vocabulary to find its corresponding unique identifier. This unique identifier can be an index number and has no semantic meaning itself. Afterward, the embedding vector lookup table stored in the preset text embedding layer is called. Through this embedding vector lookup table, each unique identifier is mapped to a fixed-dimensional text embedding vector. This text embedding vector is numerical and its dimensions encode the semantic information of the sub-word unit.

[0044] Finally, the text embedding vectors corresponding to all sub-word units are arranged and combined strictly according to the order in which they appear in the original text instructions, thus forming the final text token sequence. This sequence not only contains the independent semantics of each word, but more importantly, it fully preserves the grammatical relationships and logical structure of the text through its sequential structure.

[0045] Step 102-2: Divide the image object to be processed into multiple non-overlapping image blocks; map each image block to a visual embedding vector through the linear projection layer in the pre-trained visual Transformer encoder to obtain an initial visual token for each image block, and construct an initial visual token sequence based on all initial visual tokens; input the initial visual token sequence into the multi-layer Transformer encoder of the visual Transformer encoder for feature transformation to obtain an updated visual token for each image block; construct a visual token set based on the updated visual tokens; and obtain the attention weight matrix generated by the attention module in at least one layer of the multi-layer Transformer encoder, and determine the attention score corresponding to each visual token based on the attention weight matrix.

[0046] In this embodiment, the input image object to be processed is first divided into a series of regularly arranged, non-overlapping image blocks. For example, a complete medical image, such as a chest X-ray, can be divided into numerous small grid regions, or a complex chart in a financial report can be segmented into multiple local slices. Each image block corresponds to a local region in the original image object to be processed. This division method can transform continuous pixel information into discrete, independently processable units, laying the foundation for subsequent deep feature extraction.

[0047] Next, each image patch is fed into a pre-trained visual Transformer encoder. Specifically, the first step is a linear projection layer, which maps each image patch, composed of pixel values, into a high-dimensional vector called the visual embedding vector through a learnable linear transformation. This vector is the initial visual token for that image patch, containing the primary visual features of the region. Arranging the initial visual tokens of all image patches in spatial order forms the initial visual token sequence.

[0048] Subsequently, the initial visual token sequence is input into a multi-layer Transformer encoder of the visual Transformer encoder. Each layer of the encoder performs deep feature transformation and information fusion on the input token sequence through a self-attention mechanism and a feedforward neural network. After multiple layers of such processing, each initial visual token becomes an updated visual token, whose vector representation not only contains its own visual information but also incorporates its contextual association information with all other regions in the image object to be processed. All these updated visual tokens constitute the set of visual tokens for subsequent processing. At the same time, in the self-attention module of at least one Transformer encoder layer, the attention weight matrix generated by it can be obtained. This matrix quantifies the degree of attention each visual token in the initial visual token sequence pays to another visual token. In this embodiment, the attention weights of special tokens (such as the [CLS] token) used to aggregate global information are extracted from this matrix, and these weight values ​​are determined as the attention scores corresponding to each visual token, which directly reflect the global importance of each image patch in the eyes of the model.

[0049] This application's embodiments, through a Transformer-based architecture, achieve deep understanding and structured representation of image information from local to global, and from representation to semantics. The visual token set constructed based on this architecture is a feature set rich in contextual semantics, while the attention score provides a built-in, data-driven importance metric. This not only encodes each image patch into a semantically rich visual token but also automatically assigns it an attention score, thus providing a reliable basis for the efficient selection of subsequent visual tokens.

[0050] Optionally, in this embodiment, the preset condition is that the number of visual tokens in the subset of visual tokens reaches a preset target number. The preset target number is determined by: obtaining the width resolution and height resolution of the image object to be processed, a preset visual information compression ratio, and a preset image block division size; calculating a first product between the width resolution and the height resolution; calculating a second product between the square of the preset image block division size and the visual information compression ratio; and determining the preset target number based on the ratio between the first product and the second product; or, calculating the preset target number based on the number of multiple visual tokens corresponding to the image object to be processed and a preset retention ratio.

[0051] In this embodiment, the preset condition can be that the number of visual tokens in the subset of visual tokens reaches a preset target number. The target number can be determined dynamically or statically using one of two optional calculation methods to ensure that the screening process is both efficient and adaptable to the needs of different application scenarios. For example, in smart healthcare, the visual information density required to process a high-resolution panoramic pathological image may be drastically different from that required to process a low-resolution ultrasound screenshot. Therefore, a flexible mechanism is needed to determine how many key visual tokens should ultimately be retained for inference.

[0052] The first method involves precise calculations based on the resolution of the image object to be processed and the preprocessing parameters. Specifically, the width and height resolutions of the image object can be obtained, and multiplying them yields the first product, which is the total number of pixels in the image object. Simultaneously, given the size of the image blocks divided by the visual encoder during preprocessing (e.g., 16x16 pixels), the square of this size is calculated to obtain the number of pixels in one image block. This is then multiplied by a preset visual information compression ratio (e.g., 0.1, representing a desired compression of 90% of the visual data) to obtain the second product. Finally, the first product is divided by the second product, and the result is rounded to obtain the preset target number. This allows for the reverse calculation of how many image blocks from the original image object to be divided into as selected visual tokens, based on the desired compression level. In the fintech field, when analyzing a 4K high-definition enterprise architecture diagram, if you want to significantly improve processing speed, you can set a lower visual information compression ratio to calculate that fewer visual tokens need to be retained. Conversely, for a contract signature and seal image that requires detailed review, you can set a higher visual information compression ratio to retain more detailed visual tokens.

[0053] The second method can determine the target number based on the original set of visual tokens of the image object to be processed. Specifically, first, the total number of visual tokens in the set is obtained, and then multiplied by a preset retention ratio (e.g., 0.2, i.e., retaining 20%), thus directly obtaining the target number. This method is computationally simple and does not depend on the physical resolution of the original image object to be processed.

[0054] The two target quantity determination schemes in this application provide quantifiable, configurable, and task-go-related stopping criteria for visual token selection. The first scheme implements adaptive computation linked to image physical properties and compression requirements, making it particularly suitable for scenarios with precise information retention requirements, such as high-precision financial chart analysis. The second scheme provides a simple and efficient control method, ideal for batch tasks requiring standardized processing flows, such as rapidly preprocessing massive amounts of image reports in a medical imaging cloud platform. By flexibly selecting these two schemes, this application can achieve precise control of computing resources and a significant improvement in inference efficiency while ensuring the quality of multimodal inference.

[0055] Furthermore, as Figure 1 In terms of specific implementation, this application provides a multimodal reasoning task processing device based on the selection of visual tokens, such as... Figure 3 As shown, the device includes: The instruction receiving module is used to receive the image object to be processed and the corresponding text instruction; The encoding module is used to encode the text instructions into a text token sequence, encode the image object to be processed into multiple visual tokens, construct a visual token set based on the multiple visual tokens, and determine the attention score corresponding to each visual token; The visual token selection module is used to calculate the semantic similarity between every two visual tokens, initialize an empty visual token subset, and iteratively select a visual token from the visual token set and add it to the visual token subset each time until the number of visual tokens in the visual token subset meets a preset condition. During each selection, for each remaining visual token in the visual token set, the semantic coverage increment after adding the remaining visual token to the current visual token subset is calculated based on the semantic similarity between the remaining visual token and each selected visual token in the current visual token subset. Based on the semantic coverage increment and the corresponding attention score, the joint score of the remaining visual token is calculated. The remaining visual token with the highest joint score is added to the current visual token subset and removed from the visual token set. The inference module is used to concatenate the selected visual tokens contained in the visual token subset with the text token sequence, and perform a multimodal inference task based on the concatenation result to obtain the inference result.

[0056] Optionally, the visual token selection module is used for: Get the semantic coverage corresponding to the current visual token subset; Add the remaining visual tokens selected this time to the current visual token subset to form a new visual token subset; Iterate through each selected visual token in the new subset of visual tokens. For each selected visual token, based on the semantic similarity between the selected visual token and each unlabeled visual token in the original set of visual tokens of the image object to be processed, filter out target visual tokens from the original set of visual tokens whose semantic similarity with the selected visual token is greater than a preset similarity threshold, record the number of targets for the target visual tokens, and label the target visual tokens. After the traversal is completed, for each selected visual token, the number of targets is summed to obtain the total semantic coverage. The total semantic coverage is then divided by the total number of all visual tokens of the image object to be processed to obtain the semantic coverage of the new visual token subset. Subtract the semantic coverage of the current visual token subset from the semantic coverage of the new visual token subset to obtain the semantic coverage increment corresponding to the remaining visual tokens; Accordingly, the device further includes an update module; the update module is configured to: After adding the remaining visual token with the highest joint score to the current visual token subset, the semantic coverage corresponding to adding the remaining visual token with the highest joint score to the current visual token subset is used as the semantic coverage corresponding to the updated current visual token subset.

[0057] Optionally, the semantic coverage of the new visual token subset is calculated based on the following formula: ; in, This indicates that the remaining visual tokens will be used. v The semantic coverage of the new visual token subset formed after adding it to the current visual token subset S, where V represents the original visual token set of the image object to be processed. This represents the total number of all visual tokens in V. u Represents any visual token in V. s sim( represents any selected visual token in the new subset of visual tokens) u , s ) represents a visual tokenu With the selected visual token s Semantic similarity between them This indicates the preset similarity threshold.

[0058] Optionally, the joint score of the remaining visual tokens is calculated based on the following formula: ; in, Indicates the remaining visual token v The joint score corresponding to the addition to the current visual token subset S. Indicates the remaining visual token v The semantic coverage increment corresponding to adding it to the current visual token subset S, A V Indicates the remaining visual token v The corresponding attention score, Indicates a significant moderating factor.

[0059] Optionally, the encoding module is used for: Obtain a predefined text vocabulary, wherein the text vocabulary contains a unique identifier for each preset sub-word unit; The text instruction is segmented using a preset word segmenter to obtain multiple discrete sub-word units; Each discrete sub-word unit is matched with the text vocabulary, and the unique identifier corresponding to the sub-word unit in the text vocabulary is found. Then, the unique identifier of each sub-word unit is mapped to a fixed-dimensional text embedding vector through the embedding vector lookup table stored in the preset text embedding layer. The text embedding vectors corresponding to each sub-word unit are arranged in the order of the sub-word units in the text instruction to form the text token sequence.

[0060] Optionally, the encoding module is further configured to: The image object to be processed is divided into multiple non-overlapping image blocks; By using the linear projection layer in the pre-trained visual Transformer encoder, each image patch is mapped to a visual embedding vector to obtain the initial visual token for each image patch, and an initial visual token sequence is constructed based on all the initial visual tokens. The initial visual token sequence is input into the multi-layer Transformer encoder of the visual Transformer encoder for feature transformation to obtain the updated visual token for each image block. A visual token set is constructed based on the updated visual tokens. In at least one layer of the multi-layer Transformer encoder, the attention weight matrix generated by the attention module is obtained. The attention score corresponding to each visual token is determined based on the attention weight matrix.

[0061] Optionally, the preset condition is that the number of visual tokens in the subset of visual tokens reaches a preset target number; the preset target number is determined as follows: Obtain the width and height resolutions of the image object to be processed, a preset visual information compression ratio, and a preset image block division size; calculate the first product between the width resolution and the height resolution; and calculate the second product between the square of the preset image block division size and the visual information compression ratio; determine the preset target quantity based on the ratio between the first product and the second product; or, The preset target number is calculated based on the number of multiple visual tokens corresponding to the image object to be processed and the preset retention ratio.

[0062] It should be noted that other corresponding descriptions of the functional units involved in the multimodal reasoning task processing device based on the selection of visual tokens provided in this application embodiment can be found in the following references. Figures 1 to 2 The corresponding descriptions in the method will not be repeated here.

[0063] This application also provides a computer device, which may specifically be a personal computer, a server, a network device, etc. Figure 4 As shown, the computer device includes a bus, a processor, memory, and a communication interface, and may also include an input / output interface and a display device. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores location information. The network interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0064] Those skilled in the art will understand that Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0065] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, having stored thereon a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0066] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0067] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0068] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0069] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0070] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for processing multimodal reasoning tasks based on the selection of visual tokens, characterized in that, include: Receive the image object to be processed and the corresponding text instructions; The text instructions are encoded into a text token sequence, the image object to be processed is encoded into multiple visual tokens, a visual token set is constructed based on the multiple visual tokens, and the attention score corresponding to each visual token is determined; Calculate the semantic similarity between each pair of visual tokens, and initialize an empty subset of visual tokens. Iteratively select one visual token from the set of visual tokens and add it to the subset of visual tokens each time until the number of visual tokens in the subset of visual tokens meets a preset condition. Each time a selection is made, for each remaining visual token in the set of visual tokens, calculate the semantic coverage increment after adding the remaining visual token to the current subset of visual tokens based on the semantic similarity between the remaining visual token and each selected visual token in the current subset of visual tokens. Based on the semantic coverage increment and the corresponding attention score, calculate the joint score of the remaining visual token. Add the remaining visual token with the highest joint score to the current subset of visual tokens and remove it from the set of visual tokens. The selected visual tokens contained in the visual token subset are concatenated with the text token sequence, and a multimodal reasoning task is performed based on the concatenation result to obtain the reasoning result.

2. The method according to claim 1, characterized in that, The step of calculating the semantic coverage increment of the remaining visual tokens after adding them to the current visual token subset based on the semantic similarity between the remaining visual tokens and each selected visual token in the current visual token subset includes: Get the semantic coverage corresponding to the current visual token subset; Add the remaining visual tokens selected this time to the current visual token subset to form a new visual token subset; Iterate through each selected visual token in the new subset of visual tokens. For each selected visual token, based on the semantic similarity between the selected visual token and each unlabeled visual token in the original set of visual tokens of the image object to be processed, filter out target visual tokens from the original set of visual tokens whose semantic similarity with the selected visual token is greater than a preset similarity threshold, record the number of targets for the target visual tokens, and label the target visual tokens. After the traversal is completed, for each selected visual token, the number of targets is summed to obtain the total semantic coverage. The total semantic coverage is then divided by the total number of all visual tokens of the image object to be processed to obtain the semantic coverage of the new visual token subset. Subtract the semantic coverage of the current visual token subset from the semantic coverage of the new visual token subset to obtain the semantic coverage increment corresponding to the remaining visual tokens; Accordingly, after adding the remaining visual token with the highest joint score to the current visual token subset, the method further includes: The semantic coverage corresponding to adding the remaining visual token with the highest joint score to the current visual token subset is used as the semantic coverage corresponding to the updated current visual token subset.

3. The method according to claim 2, characterized in that, The semantic coverage of the new visual token subset is calculated based on the following formula: ； in, This indicates that the remaining visual tokens will be used. v The semantic coverage of the new visual token subset formed after adding it to the current visual token subset S, where V represents the original visual token set of the image object to be processed. This represents the total number of all visual tokens in V. u Represents any visual token in V. s sim( represents any selected visual token in the new subset of visual tokens) u , s ) represents a visual token u With the selected visual token s Semantic similarity between them This indicates the preset similarity threshold.

4. The method according to claim 1, characterized in that, The joint score of the remaining visual tokens is calculated based on the following formula: ； in, Indicates the remaining visual token v The joint score corresponding to the addition to the current visual token subset S. Indicates the remaining visual token v The semantic coverage increment corresponding to adding it to the current visual token subset S, A V Indicates the remaining visual token v The corresponding attention score, Indicates a significant moderating factor.

5. The method according to claim 1, characterized in that, Encoding the text instructions into a text token sequence includes: Obtain a predefined text vocabulary, wherein the text vocabulary contains a unique identifier for each preset sub-word unit; The text instruction is segmented using a preset word segmenter to obtain multiple discrete sub-word units; Each discrete sub-word unit is matched with the text vocabulary, and the unique identifier corresponding to the sub-word unit in the text vocabulary is found. Then, the unique identifier of each sub-word unit is mapped to a fixed-dimensional text embedding vector through the embedding vector lookup table stored in the preset text embedding layer. The text embedding vectors corresponding to each sub-word unit are arranged in the order of the sub-word units in the text instruction to form the text token sequence.

6. The method according to claim 1, characterized in that, The step of encoding the image object to be processed into multiple visual tokens, constructing a visual token set based on the multiple visual tokens, and determining the attention score corresponding to each visual token includes: The image object to be processed is divided into multiple non-overlapping image blocks; By using the linear projection layer in the pre-trained visual Transformer encoder, each image patch is mapped to a visual embedding vector to obtain the initial visual token for each image patch, and an initial visual token sequence is constructed based on all the initial visual tokens. The initial visual token sequence is input into the multi-layer Transformer encoder of the visual Transformer encoder for feature transformation to obtain the updated visual token for each image block. A visual token set is constructed based on the updated visual tokens. In at least one layer of the multi-layer Transformer encoder, the attention weight matrix generated by the attention module is obtained. The attention score corresponding to each visual token is determined based on the attention weight matrix.

7. The method according to claim 6, characterized in that, The preset condition is that the number of visual tokens in the subset of visual tokens reaches a preset target number; the preset target number is determined as follows: The width and height resolutions of the image object to be processed, the preset visual information compression ratio, and the preset image block division size are obtained. The first product between the width resolution and the height resolution is calculated, and the second product between the square of the preset image block division size and the visual information compression ratio is calculated. The preset target quantity is determined based on the ratio between the first product and the second product. or, The preset target number is calculated based on the number of multiple visual tokens corresponding to the image object to be processed and the preset retention ratio.

8. A multimodal reasoning task processing device based on the selection of visual tokens, characterized in that, include: The instruction receiving module is used to receive the image object to be processed and the corresponding text instruction; The encoding module is used to encode the text instructions into a text token sequence, encode the image object to be processed into multiple visual tokens, construct a visual token set based on the multiple visual tokens, and determine the attention score corresponding to each visual token; The visual token selection module is used to calculate the semantic similarity between every two visual tokens, initialize an empty visual token subset, and iteratively select a visual token from the visual token set and add it to the visual token subset each time until the number of visual tokens in the visual token subset meets a preset condition. During each selection, for each remaining visual token in the visual token set, the semantic coverage increment after adding the remaining visual token to the current visual token subset is calculated based on the semantic similarity between the remaining visual token and each selected visual token in the current visual token subset. Based on the semantic coverage increment and the corresponding attention score, the joint score of the remaining visual token is calculated. The remaining visual token with the highest joint score is added to the current visual token subset and removed from the visual token set. The inference module is used to concatenate the selected visual tokens contained in the visual token subset with the text token sequence, and perform a multimodal inference task based on the concatenation result to obtain the inference result.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

10. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

Images