Multi-modal visual language model lexical element cutting method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to the fields of financial science and technology and medical science and technology, and discloses a lexical element cutting method, device, equipment and medium for a multi-modal visual language model.The method comprises the steps that a question image and a text are received and converted into a visual lexical element set and a target lexical element set; on the basis of the visual lexical element set and the target lexical element set, a contribution matrix is calculated with the goal of minimizing the optimal transmission cost from the visual lexical elements to the target lexical elements by fusing a visual perception cost function of feature similarity, a spatial relationship and a center distance, so that the contribution degree of each visual lexical element to the generation of the target lexical element is globally quantified; and according to the contribution value sorting and in combination with a preset threshold value, performing one-time cutting on the visual lexical element set, reserving the key Token, and selectively recovering part of the cut Token through uniform sampling so as to adapt to a fine task requirement. According to the method, accurate and efficient Token screening is realized, the calculation and memory overhead is remarkably reduced, an additional calculation step does not need to be introduced, and the reasoning efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology and can be applied to the fields of financial technology and medical technology. In particular, it relates to a method, apparatus, device and medium for word segmentation of a multimodal visual language model. Background Technology

[0002] In the current technological development of Visual Language Models (VLMs), they have demonstrated significant performance in handling multimodal tasks, but at the same time, they place extremely high demands on computing resources. This problem is particularly prominent in visual information processing, especially in fields with extremely high requirements for accuracy and efficiency, such as medical image analysis and insurance claims image review. Specifically, to achieve effective understanding and analysis of image content, existing VLMs (such as LLaVA, InternVL, and other typical models) usually need to convert input images (such as medical X-rays or insurance accident scene photos) into a large number of visual tokens. The number of visual tokens often far exceeds the number of text tokens processed by the model. This difference directly leads to a significant increase in the computational load during the model's inference process, accompanied by a significant increase in memory usage. This not only increases the hardware cost of model deployment but also limits its application in scenarios with limited computing resources, such as medical diagnosis or insurance risk control. To alleviate the aforementioned computational and memory pressures, existing technologies commonly employ visual token pruning strategies. This involves filtering and removing some visual tokens to reduce subsequent computational burden. The core basis of these pruning strategies is often a greedy heuristic standard, with the most common approach being to determine the importance of visual tokens based on attention scores, prioritizing the retention of tokens with higher attention scores and pruning tokens with lower scores. However, such token pruning methods that rely on greedy heuristics have significant shortcomings: On the one hand, attention scores can only reflect the weight of tokens in local attention interactions, and cannot comprehensively and accurately capture the actual contribution of each visual token to the model's overall task (such as lesion recognition or insurance document information extraction). This can easily lead to the erroneous pruning of key tokens or the retention of redundant tokens, thus affecting the model's performance in processing visual-language tasks. On the other hand, to achieve pruning based on greedy heuristics, existing methods usually require additional computational steps outside the model's regular inference process to count, sort, and perform pruning operations on the attention scores of each visual token. These additional steps not only further increase computational overhead but are also difficult to adapt to the model's existing optimization mechanisms, resulting in a failure to effectively improve overall inference efficiency, especially in time-sensitive scenarios such as real-time medical assisted diagnosis or automated insurance processing. In summary, existing vision-language models face the problem of excessive visual tokens leading to a surge in computational costs and memory usage when processing visual information. The current token pruning method based on the greedy heuristic standard has technical drawbacks such as inaccurate importance assessment, the need for additional computational steps, and limited improvement in inference efficiency. There is an urgent need for a better token processing solution to address these issues. Summary of the Invention

[0003] This invention provides a method, apparatus, device, and medium for token trimming in a multimodal visual language model, aiming to solve the problems of inaccurate visual token trimming and the need for additional computational steps in existing visual language models.

[0004] Firstly, a lexical trimming method for a multimodal visual language model is provided, the method comprising: The system receives a question image and question text input by the user, converts the question image into a visual word set through a visual encoder, and processes the question text through a language model to obtain a target word set. Based on the visual word set and the target word set, a contribution matrix is ​​calculated using a preset visual perception cost function with the optimal transmission cost from the visual word to the target word as the objective. The contribution matrix represents the contribution of the visual word to the target word. Calculate the contribution value corresponding to each visual lexical unit based on the contribution matrix; The target visual word set is obtained by pruning the visual word set according to the contribution value of the visual word and the preset contribution threshold.

[0005] Secondly, a lexical trimming device for a multimodal visual language model is provided, including a unit for performing the above-described method.

[0006] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0007] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described method.

[0008] This invention provides a method, apparatus, device, and medium for lexical pruning in a multimodal visual language model. By introducing an optimal transmission theory framework, this invention transforms the visual lexical pruning problem into an optimization problem of minimizing transmission costs. It utilizes a preset visual perception cost function to accurately quantify the contribution of each visual lexical to the target lexical generation task, thereby overcoming the inaccurate pruning defects caused by the reliance on local information in traditional greedy strategies or heuristic rules. At the same time, by calculating the contribution matrix in one go and completing the pruning based on a unified threshold, it avoids the additional computational steps caused by multiple iterations or complex post-processing required by traditional methods, achieving the technical effect of significantly reducing computational load and memory consumption while retaining key visual information. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating the word segmentation method of the multimodal visual language model according to an embodiment of the present invention; Figure 2 yes Figure 1 A flowchart illustrating the sub-steps of S110; Figure 3 yes Figure 1 A flowchart illustrating another sub-step of S110; Figure 4 yes Figure 1 A flowchart illustrating the sub-steps of S120; Figure 5 yes Figure 4 A flowchart illustrating the sub-steps of S121; Figure 6 yes Figure 1 A flowchart illustrating the sub-steps of S130; Figure 7 yes Figure 1 A flowchart illustrating the sub-steps of S140; Figure 8 This is a schematic block diagram of the word segmentation device for the multimodal visual language model according to an embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of a computer device in an embodiment of the present invention; Figure 10 This is another structural schematic diagram of the computer device in an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] With the rapid development and widespread application of multimodal visual language models (VLMs) in various vision-language tasks, the high demand for computational resources has become a key bottleneck restricting their efficient deployment. Especially in the visual information processing stage, to achieve effective parsing of image content, existing models (such as LLaVA and InternVL) often need to convert images into a far greater number of visual tokens than text tokens, directly leading to a significant increase in computational costs and memory usage. To alleviate this pressure, visual token cropping has become a mainstream optimization method. However, existing cropping methods mostly rely on greedy heuristics based on attention scores, which cannot accurately capture the actual contribution of each visual token to the model task, easily leading to miscropping of key tokens or retention of redundant tokens. Furthermore, because they require additional computational steps outside the regular inference process, they are difficult to adapt to the commonly used Flash Attention acceleration technology (which requires additional computational steps and cannot fully utilize its advantages in reducing memory access overhead and improving computational efficiency). At the same time, dynamic cropping also introduces key-value (KV) issues. Frequent changes in cache size not only fail to effectively optimize memory utilization but also further increase memory requirements. These issues collectively limit the inference efficiency and performance improvement of existing visual language models, necessitating a better token pruning solution to overcome these technical bottlenecks.

[0013] The lexical trimming method for multimodal visual language models provided in this invention can be applied to either a client or a server. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0014] Please see Figure 1 As shown, Figure 1 A flowchart illustrating a word segmentation method for a multimodal visual language model provided in an embodiment of the present invention includes the following steps: S110-S140.

[0015] S110. Receive a question image and question text input by the user, convert the question image into a visual word set through a visual encoder, and process the question text through a language model to obtain a target word set; In this embodiment, the question image refers to the image data input by the user for multimodal visual language tasks, which may include visual information such as specific target objects and scenes; the question text refers to the text instructions input by the user related to the question image, such as image description requests, visual question-and-answer questions, etc.; the visual encoder is a model component used to extract image features, which can convert image data into structured feature representations; the visual lexicon set is the set obtained after processing by the visual encoder, which contains feature vectors of multiple preset dimensions, each vector corresponding to the feature information of the image; the language model is a model component used to process text data, which can realize text semantic understanding and feature extraction; the target lexicon set is the set obtained after processing by the language model, which contains semantic feature vectors of multiple preset dimensions, each vector corresponding to the semantic information of the text.

[0016] Specifically, the process first receives the user's input image and text, performing basic validation to ensure their integrity and usability. Then, a pre-defined visual encoder is invoked to extract and encode features from the image, converting its visual information into machine-processable feature vectors, thus forming a visual vocabulary set. Simultaneously, a pre-defined language model is invoked to perform semantic parsing, feature extraction, and encoding on the text, converting its semantic information into semantic feature vectors, thus forming a target vocabulary set. This completes the initial structured transformation of the multimodal input data. Specifically, existing multimodal visual-language models often struggle with effective association in subsequent processing due to the heterogeneity of image and text data formats and inconsistent feature representations. This step addresses this by using a visual encoder and a language model to process the two types of input data separately, converting the heterogeneous images and text into a vocabulary set with unified dimensions. This solves the technical problem of feature alignment difficulties in multimodal data and provides a unified data foundation for subsequent association calculations. This step achieves standardized feature conversion between the question image and the question text, ensuring the compatibility and relevance of multimodal input data in subsequent processing. It lays a reliable data foundation for subsequent computation of the contribution of visual lexical units to target lexical units and for lexical unit pruning, thereby improving the model's adaptability to multimodal input processing.

[0017] In one embodiment, such as Figure 2 As shown, step S110 includes: S111-S115.

[0018] S111. Receive a question image input by the user, preprocess the question image to remove noise interference in the image and adjust the image size to the input size adapted to the visual encoder; S112. Input the preprocessed query image into the preset visual encoder, and extract features from the image through the convolutional layer and pooling layer of the visual encoder to obtain the global visual feature map of the image. S113. The global visual feature map is divided into blocks, and the feature map is divided into multiple local feature blocks of the same size. Each local feature block corresponds to a local region of the image. S114. Perform dimension mapping and feature encoding on each local feature block through the fully connected layer or attention layer of the visual encoder, and convert each local feature block into a feature vector of a preset dimension. S115. Integrate all the feature vectors to form a visual word set containing multiple preset dimension feature vectors.

[0019] In this embodiment, preprocessing refers to the initial optimization operations performed on the query image, including noise removal and size adjustment, with the aim of improving the accuracy of subsequent feature extraction. The global visual feature map is the structured graph data obtained after processing by the convolutional and pooling layers of the visual encoder, which can comprehensively reflect the overall visual features of the image. The local feature block is a sub-image obtained by dividing the global visual feature map into blocks according to preset rules. Each sub-image corresponds to a local region of the image and can capture local detailed features. The fully connected layer is a network layer in the visual encoder used to map the local feature blocks to a preset dimension. The attention layer is a network layer in the visual encoder used to focus on the key information of the local feature blocks. The feature vector is a numerical vector with a preset dimension, which can quantify the visual information of the local feature blocks.

[0020] Specifically, the system first receives the user-input image of the query and initiates a preprocessing process: noise removal algorithms are used to eliminate noise interference in the image, preventing noise from affecting the accuracy of subsequent feature extraction. Simultaneously, the image is scaled or cropped according to the input size requirements of the visual encoder to fit its dimensions. After preprocessing, the image is input into a pre-defined visual encoder. The encoder extracts local features from the image through convolutional layers and then downsamples these features through pooling layers, reducing the amount of data while retaining key features, thus generating a global visual feature map. Next, the global visual feature map is divided according to a pre-defined block division rule, resulting in multiple local feature blocks of the same size. Subsequently, the fully connected layer or attention layer of the visual encoder is called to perform dimension mapping and feature encoding on each local feature block, converting it into a feature vector of a pre-defined dimension. Finally, all feature vectors are integrated to form a visual vocabulary set. Specifically, existing technologies that directly extract features from raw images are susceptible to noise interference and image size mismatch, leading to inaccurate feature extraction and difficulty in effectively capturing local image features. This step addresses these issues by preprocessing to optimize image quality and size, generating a global feature map through convolutional and pooling layers, then segmenting the image into blocks to capture local features, and finally encoding feature vectors through fully connected or attention layers. This solves the technical problems of inaccurate image feature extraction and loss of local features, ensuring that the visual lexicon set comprehensively reflects the global and local visual information of the image. The visual lexicon set generated in this step is of high quality and high representativeness, providing an accurate visual feature foundation for subsequent calculations of the contribution of visual lexicons to target lexicons, reducing subsequent cropping bias caused by inaccurate feature extraction, and improving the model's efficiency in utilizing image information.

[0021] In one embodiment, such as Figure 3 As shown, step S110 further includes: S116-S1110.

[0022] S116. Receive the question text input by the user, preprocess the question text, remove invalid characters from the text and standardize the text format to obtain a uniformly formatted question text to be processed. S117. Input the standardized question text to be processed into the preset language model. Through the text segmentation module of the language model, the question text to be processed is split into multiple continuous text segments according to the preset segmentation rules. S118. Through the embedding layer of the language model, each text segment is semantically encoded, and each text segment is converted into a semantic feature vector of a preset dimension. S119. Perform validity verification on all the semantic feature vectors, remove feature vectors with missing semantic information or noise interference, and retain semantic feature vectors that meet the preset semantic integrity requirements. S1110. Integrate all the retained semantic feature vectors to form a target word set containing multiple preset dimension semantic feature vectors.

[0023] In this embodiment, invalid characters refer to redundant characters in the question text that do not affect semantic expression, such as special symbols and meaningless spaces; text format standardization refers to adjusting the question text to a format that meets the input requirements of the language model, such as unified character encoding and standardized sentence separation; the question text to be processed is preprocessed text data with a unified format and no invalid characters; the text segmentation module is a component in the language model used to split long texts into short text fragments, ensuring the semantic integrity of the fragments according to preset rules; a text fragment is a short text unit with independent semantics obtained after text segmentation; the embedding layer is a network layer in the language model used to convert text fragments into semantic feature vectors; a semantic feature vector is a numerical vector with preset dimensions that can quantify the semantic content of the text fragment; validity verification refers to quality detection of the semantic feature vectors, eliminating vectors with missing semantic information or interference from noise; the semantic integrity requirement is a preset standard for judging whether the semantic feature vectors completely represent the semantics of the text.

[0024] Specifically, the system first receives the user's input question text and initiates a preprocessing process: invalid characters are removed from the text using a character filtering algorithm to prevent them from interfering with the semantic understanding of the language model. Simultaneously, the text format is standardized according to the language model's input format requirements, resulting in a uniformly formatted question text. Next, the question text is input into a pre-defined language model, which calls the model's text segmentation module to split it into multiple continuous text segments according to pre-defined rules (such as sentence boundaries and semantic pauses). Then, each text segment is semantically encoded through the language model's embedding layer, converting it into a semantic feature vector of a pre-defined dimension. Afterward, all semantic feature vectors are validated for validity. Based on pre-defined semantic integrity requirements, vectors lacking semantic information or affected by noise are removed, while valid vectors are retained. Finally, the retained semantic feature vectors are integrated to form a target word set. Specifically, existing technologies for processing query text are prone to semantic comprehension biases in language models due to interference from invalid characters and inconsistent text formats. Furthermore, the generated semantic feature vectors may contain semantic gaps, affecting subsequent calculations of association with visual lexical units. This step addresses these issues by optimizing text quality and format through preprocessing, ensuring semantic integrity of fragments through text segmentation, and filtering valid vectors through validity checks. This resolves the technical problems of inaccurate text processing and low-quality semantic feature vectors, ensuring that the target lexical set accurately reflects the semantic requirements of the query text. The target lexical set generated in this step features high semantic accuracy and validity, providing an accurate semantic benchmark for subsequent calculations of the contribution of visual lexical units to target lexical units. This ensures that the pruning direction of visual lexical units aligns with the user's textual needs, improving the multimodal model's ability to understand and respond to user intent.

[0025] S120. Based on the visual word set and the target word set, a contribution matrix is ​​calculated using a preset visual perception cost function with the optimal transmission cost from the visual word to the target word as the objective. The contribution matrix represents the contribution of the visual word to the target word. In this embodiment, the preset visual perception cost function is a function designed in combination with visual information characteristics (such as feature association, spatial structure, etc.) to calculate the transmission cost between visual words and target words; the optimal transmission cost refers to the optimization objective where the sum of the transmission costs from all visual words to the corresponding target words is the lowest; the contribution matrix is ​​a matrix used to store the degree of contribution of each visual word to each target word, and the value of each element in the matrix directly reflects the importance association between the corresponding visual word and the target word.

[0026] Specifically, the process begins by acquiring the visual lexicon set and target lexicon set obtained from the previous processing. The completeness and dimensional consistency of the two sets are verified to ensure the feasibility of subsequent calculations. Then, a pre-defined visual perception cost function is invoked, and the required input information (i.e., relevant data reflecting the association between visual and target lexicons) is determined. With the optimal transmission cost from visual lexicons to target lexicons as the core optimization direction, an association mapping relationship between visual and target lexicons is established through adapted computational logic. The contribution of each visual lexicon to each target lexicon is quantified one by one, and finally, these contribution levels are organized according to the correspondence between visual and target lexicons to form a contribution matrix. Specifically, existing technologies rely on a greedy heuristic based on attention scores when calculating the contribution of visual lexicons to target lexicons, which cannot comprehensively and accurately capture the true association between the two, easily leading to biased contribution evaluation. This step, by combining a pre-defined visual perception cost function with the optimal transmission cost objective, constructs a systematic contribution calculation framework, eliminating the reliance on greedy strategies and enabling multi-dimensional consideration of the association between visual and target lexicons, thus solving the technical problem of inaccurate contribution calculation. This step generates a contribution matrix that provides an accurate and reliable basis for subsequent judgments on the importance of visual lexical units, laying a core foundation for the accuracy of subsequent lexical unit pruning. At the same time, it avoids introducing additional complex steps in the contribution calculation process, which helps maintain the simplicity and efficiency of the model inference process.

[0027] In one embodiment, such as Figure 4 As shown, step S120 includes: S121-S122.

[0028] S121. Based on the visual word set and the target word set, calculate the transmission cost from each visual word to each target word using the preset visual perception cost function to form a transmission cost matrix; S122. Solve the transmission cost matrix to obtain the contribution matrix with the goal of minimizing the total transmission cost.

[0029] In this embodiment, transmission cost is a quantitative indicator of the degree of correlation between a single visual word and a single target word. Generally, the lower the transmission cost, the stronger the correlation between the two. The transmission cost matrix is ​​a dataset organized according to the row and column correspondence of the transmission costs from all visual words to all target words (rows represent visual words, and columns represent target words). Minimizing the total transmission cost refers to the optimization direction of adjusting the contribution distribution of visual words to target words through optimization algorithms, so that the total transmission cost from all visual words to the corresponding target words is reduced to the lowest level.

[0030] Specifically, the core features of each visual word are first extracted from the visual word set, and the core features of each target word are extracted from the target word set. The two sets of features are then input into a preset visual perception cost function. Following the logic of "each visual word corresponds to each target word", the transmission cost from each visual word to each target word is calculated one by one. After all transmission costs have been calculated, these cost values ​​are filled into the corresponding positions in the matrix according to the index order of visual words and target words to form a transmission cost matrix. With minimizing the total transmission cost as the optimization objective, the optimal transmission solution algorithm is selected. Under the premise that "the contribution allocation of each visual word meets the rationality constraint" (such as the sum of the contributions of each visual word to all target words follows a preset rule), the transmission cost matrix is ​​solved to find the contribution allocation scheme that minimizes the total transmission cost. The matrix corresponding to this scheme is the final contribution matrix. Specifically, in existing technologies, dynamic pruning leads to frequent changes in the KV cache size, and the additional computational steps are incompatible with FlashAttention. This step first generates a fixed-structure transmission cost matrix in one go, and then obtains the contribution matrix through a single optimization. The entire process involves no dynamic adjustments, avoiding cache fluctuation issues. Furthermore, it requires no additional computational steps and is compatible with FlashAttention's memory access optimization mechanism, resolving the technical problems of incompatibility with existing acceleration technologies and increased memory requirements. This step ensures the stability of transmission cost calculation and contribution matrix solving, fully utilizing existing acceleration technologies to improve inference efficiency, while reducing the additional overhead caused by memory fluctuations. It provides structurally stable and data-reliable matrix support for subsequent lexical pruning.

[0031] In one embodiment, such as Figure 5 As shown, step S121 includes: S1211-S1214.

[0032] S1211. Extract the feature vector of each visual word from the visual word set, extract the feature vector of each target word from the target word set, and calculate the similarity between the pairwise feature vectors to obtain the feature similarity parameter. S1212. Obtain the spatial location information of each visual word in the question image, and calculate the relative spatial distance between different visual words and their corresponding target words based on the spatial location information to obtain spatial relationship parameters. S1213. Determine the center coordinates of the question image, and calculate the absolute distance from the center point of each visual word to the center coordinates of the image to obtain the center distance parameter; S1214. Input the feature similarity parameter, spatial relationship parameter and center distance parameter into the preset visual perception cost function, and obtain the transmission cost from each visual word to each target word through function operation.

[0033] In this embodiment, the feature similarity parameter is a parameter obtained by calculating the similarity between the feature vectors of visual words and target words, used to characterize the degree of association between the two at the content level; spatial location information refers to the specific coordinates or area range of the visual word in the question image; relative spatial distance is a distance value that measures the spatial proximity of the associated areas of visual words and target words in the image, the smaller the relative spatial distance, the stronger the spatial association between the two; spatial relationship parameter is a parameter obtained based on the relative spatial distance and used to quantify the spatial association between visual words and target words; image center coordinates are the coordinate values ​​of the geometric center of the image calculated according to the resolution of the question image; center distance parameter is the absolute distance from the center point of the visual word to the image center coordinates, used to determine the image saliency of the area where the visual word is located.

[0034] Specifically, feature vectors for each visual word are extracted from the visual word set, and feature vectors for each target word are extracted from the target word set. Using a pre-defined similarity calculation method (such as cosine similarity algorithm), the similarity between the feature vectors of each visual word and each target word is calculated, and these similarity values ​​are recorded as feature similarity parameters. The spatial location information of each visual word in the question image (such as the coordinates of the top-left and bottom-right corners of the bounding box) is obtained through image analysis tools. Combined with the coordinates of the visual content region associated with the target word, the relative spatial distance between each visual word and the associated region of the corresponding target word is calculated, and these distance values ​​are organized into spatial relationship parameters. The geometric center coordinates of the question image are calculated based on its width and height (e.g., for an image with a resolution of W×H, the center coordinates are (W / 2, H / 2)). Then, the absolute distance from the center point of each visual word to these center coordinates is calculated, and these distance values ​​are used as center distance parameters. Feature similarity parameters, spatial relationship parameters, and center distance parameters are input into a preset visual perception cost function in a preset format (e.g., parameter vector form). The function processes the input parameters according to a preset operational logic (e.g., weighted fusion of the three parameters). The parameter combination for each visual word and each target word corresponds to an output transmission cost value, ultimately yielding the transmission cost from all visual words to all target words. Specifically, existing technologies rely on only a single dimension of information (e.g., attention score) to calculate transmission costs, which cannot fully reflect the structural features of visual information, leading to cost calculation bias. This step extracts parameters from three dimensions—feature similarity, spatial relationship, and center distance—to comprehensively consider the content association, spatial location, and regional saliency of visual words. These parameters are then input into the visual perception cost function for comprehensive calculation, solving the technical problem of a single cost calculation dimension that cannot reflect the true structure of visual information. Through this step, the calculated transmission cost can more accurately and comprehensively reflect the true degree of association between visual words and target words, ensuring the reliability of the transmission cost matrix and improving the accuracy of subsequent contribution matrix solutions. This lays a solid data foundation for subsequent word pruning to retain key information.

[0035] For example, taking the "chest CT image lung nodule detection VQA task" in a medical scenario as an example, assuming the user inputs a chest CT scan image and the question text is "Please indicate the location of the lung nodules in the image and whether they are benign features", the specific process of executing steps S121-S122 and S1211-S1214 is as follows: First, execute S1211: Extract the feature vector of each visual word from the visual word set (visual words are local feature blocks obtained after preprocessing and segmentation of the CT image, such as feature blocks containing lung nodule regions, feature blocks containing normal lung tissue, feature blocks containing vascular textures, etc., each feature vector has a dimension of d=512, including CT value distribution, Features such as texture density are extracted from the target word set to obtain the feature vector of each target word (the target word is the semantic vector obtained after the language model processes the question text, such as "lung nodule location", "benign nodule features", "lung tissue region", etc., with a dimension of 512, containing text semantic information); the cosine similarity algorithm is used to calculate the similarity between pairwise feature vectors. For example, the cosine similarity between the visual word feature vector containing lung nodules and the target word feature vector of "lung nodule location" is 0.89 (highly related content), while the cosine similarity between the visual word containing normal lung tissue and the target word of "lung nodule location" is 0.23 (low related content). These similarity values ​​are the feature similarity parameters. Next, execute S1212: Obtain the spatial location information of each visual term in the CT image using the medical image annotation tool (with the upper left corner of the image as the origin and the coordinate unit as pixels, such as the center coordinates of the visual term containing lung nodules being (320, 450), and those containing normal lung tissue being (180, 320)); Combine the coordinates of the medical anatomical region associated with the target term (such as the coordinate range of the common lung lobe nodule region corresponding to "lung nodule location" being (300-350, 420-480)), calculate the relative spatial distance between each visual term and the region associated with the corresponding target term. For example, the relative spatial distance between the visual term containing lung nodules and the target term "lung nodule location" is 15 pixels (strong spatial correlation), and 120 pixels (weak spatial correlation) for the visual term containing normal lung tissue. These distance values ​​are the spatial relationship parameters. Next, execute S1213: Based on the chest CT image resolution (512×512 pixels), determine the image center coordinates as (256, 256), and calculate the absolute distance from the center point of each visual word to these coordinates. For example, the absolute distance from the center point of a visual word containing lung nodules to the center is 220 pixels, and for those containing vascular textures, it is 180 pixels. These distance values ​​are the center distance parameters. Then, execute S1214: Input the above feature similarity parameters, spatial relationship parameters (normalized to the [0,1] interval), and center distance parameters (normalized to the [0,1] interval) into the preset visual perception cost function. The function is operated by weighted operations (such as...) Obtaining transmission costs, such as visual terms containing lung nodules. To the target term "location of lung nodules" Transmission cost Visual terms containing normal lung tissue arrive Transmission cost This process is repeated to obtain the transmission cost from all visual lexical units to the target lexical unit, forming a transmission cost matrix, thus completing step S121. Finally, step S122 is executed: combining the formula of the contribution matrix... The goal is to solve for the transmission cost matrix by minimizing the total transmission cost. Assume there are two visual lexical units (...). : Contains pulmonary nodules, (including normal lung tissue) and 2 target terms ( Location of lung nodules "Normal lung tissue"), the transmission cost matrix is The solution must satisfy the following conditions: Given the constraints, the contribution matrix that minimizes the total transmission cost is obtained through optimization calculations. ,Right now right The contribution is 1, to =0, right The contribution is 1, to A value of 0 accurately reflects the correlation contribution between visual lexical units and target lexical units in the medical context.

[0036] S130. Calculate the contribution value corresponding to each visual word element based on the contribution matrix; In this embodiment, the contribution value is a quantitative indicator of the degree of contribution of a single visual word to the overall contribution of all target words. Its value directly reflects the importance of the visual word in the multimodal task. The contribution matrix is ​​a matrix obtained in the previous steps that stores the degree of contribution of each visual word to each target word. The rows of the matrix correspond to visual words, the columns correspond to target words, and each element is the contribution coefficient of the corresponding visual word to the target word.

[0037] Specifically, the process begins by obtaining the contribution matrix calculated earlier. The matrix's completeness and data validity are verified to ensure there are no missing or outlier values, thus avoiding any impact on the accuracy of subsequent contribution value calculations. Then, based on the structural characteristics of the contribution matrix (rows corresponding to visual words, columns corresponding to target words), the matrix row data corresponding to each visual word is determined. Through pre-defined mathematical operation logic, the contribution coefficients corresponding to multiple target words in each row are integrated into a single quantified value, which is the preliminary contribution result of the corresponding visual word. Finally, the preliminary contribution results of all visual words are standardized to eliminate the dimensional differences in contribution results between different visual words, obtaining the final contribution value for each visual word. Specifically, existing greedy strategies based on attention scores can only locally determine token importance and cannot comprehensively consider the overall contribution of visual words to all target words, easily leading to a one-sided importance assessment. This step, by extracting the contribution coefficients of visual words to all target words from the contribution matrix and integrating them for calculation, achieves a comprehensive assessment of the overall contribution of visual words, solving the technical problem of one-sided importance assessment and the inability to reflect the global value of the token. This step provides accurate and comprehensive quantification of the importance of each visual token, ensuring that highly important visual tokens can be accurately identified during subsequent token trimming. It avoids mis-trimming of key tokens or retention of redundant tokens due to local evaluation bias, laying the foundation for improving trimming accuracy.

[0038] In one embodiment, such as Figure 6 As shown, step S131 includes: S131-S133.

[0039] S131. Obtain the contribution matrix that represents the degree of contribution of each visual word to the target word, and determine the contribution coefficient of each row element in the matrix to all target words. S132. Perform an aggregation operation on all contribution coefficients in each row of the contribution matrix. The aggregation operation includes any one of summation, weighted summation, or taking the maximum value to obtain the initial contribution value of the visual word element corresponding to each row. S133. Normalize the initial contribution values ​​of all visual words and map them to a preset numerical range to obtain the final contribution value of each visual word.

[0040] In this embodiment, the contribution coefficient is the numerical value of a single element in the contribution matrix, representing the degree of contribution of the corresponding visual word to a single target word. The aggregation operation is a mathematical operation that integrates multiple contribution coefficients corresponding to the same visual word into a single numerical value, including summation, weighted summation, and taking the maximum value. The summation operation is suitable for scenarios that equally consider the contributions of all target words, the weighted summation is suitable for scenarios that assign different importance weights to different target words, and taking the maximum value is suitable for scenarios that prioritize the contribution of visual words to the most relevant target words. The initial contribution value is the visual word contribution result obtained after the aggregation operation without standardization. The normalization process is the process of mapping the initial contribution value to a fixed numerical range (such as [0,1]) according to a preset rule. The purpose is to eliminate the dimensional differences of the initial contribution values ​​of different visual words and facilitate subsequent unified threshold judgment. The preset numerical range is a range that is pre-set according to the requirements of multimodal tasks and is used to standardize the range of contribution values. Usually, [0,1] is selected to simplify the subsequent threshold comparison operation.

[0041] Specifically, the previously generated contribution matrix is ​​first invoked. Through matrix index matching (row indices correspond to visual lexical numbers, column indices correspond to target lexical numbers), the correspondence between each row of data and a single visual lexical is determined, clarifying that each element in each row is the contribution coefficient of that visual lexical to the corresponding target lexical. Based on the requirements of the current multimodal task, an appropriate aggregation operation method is selected. If the task requires equal consideration of the contributions of all target lexicals (such as a general image description task), a summation operation is performed on the contribution coefficients of each row, adding all coefficients to obtain the initial contribution value of the visual lexical in that row. If the task requires highlighting the influence of key target lexicals (such as a specific medical image question-and-answer task), the aggregation operation is performed accordingly. If the task requires prioritizing the association between visual words and the most relevant target words (such as object recognition tasks), then a weighted summation operation is performed on the contribution coefficients of each row. The weights are preset according to the importance of the target word and multiplied by the corresponding coefficients before summing to obtain the initial contribution value. If the task requires prioritizing the association between visual words and the most relevant target words (such as object recognition tasks), then the maximum value operation is performed on the contribution coefficients of each row, and the largest coefficient in that row is used as the initial contribution value. The maximum and minimum values ​​of the initial contribution values ​​of all visual words are calculated, and a linear normalization formula (e.g., final contribution value = (initial contribution value - minimum value) / (maximum value - minimum value)) is used to map the initial contribution value of each visual word to a preset numerical range to obtain the final contribution value. Specifically, existing technologies lack a systematic integration and standardized process for token contributions, resulting in the inability to directly compare the contribution results of different tokens. This step constructs a complete contribution value calculation process through explicit contribution coefficient extraction, appropriate aggregation operation selection, and standardization processing, solving the technical problems of incomparable contribution results and poor adaptability. This step allows for flexible selection of aggregation methods based on different task requirements, improving the method's task adaptability. It also enables unified quantification of contribution values ​​through normalization, facilitating the rapid screening of important visual words using preset thresholds. Simultaneously, it ensures the comparability of contribution values ​​for different visual words, guaranteeing the consistency and accuracy of pruning decisions.

[0042] For example, taking the "Brain MRI Image Tumor Region VQA Task" in a medical scenario as an example, the user inputs a brain MRI image (containing a suspected tumor region), and the question text is "Please identify the location and boundary features of the tumor in the image". The process of executing steps S131-S133 is as follows: First, execute S131 to obtain the contribution matrix calculated based on the optimal transmission method in the early stage. This matrix has 10 rows (corresponding to 10 visual terms, namely the tumor core region, tumor edge region, normal brain tissue region, etc.) and 3 columns (corresponding to 3 target terms, namely "tumor location", "tumor boundary" and "normal brain tissue"). The matrix row index and visual term number are used to identify the tumor location and boundary features of the tumor. The matching process determines the contribution coefficients of the visual word "tumor core region" to the three target words in the first row (0.92, 0.88, and 0.15 respectively), and the contribution coefficients of the visual word "normal brain tissue" to the three target words in the fifth row (0.11, 0.09, and 0.94 respectively). This clarifies that all elements in each row represent the contribution coefficients of a single visual word to all target words. Next, step S132 is executed. Since this task prioritizes highlighting the contribution of visual words to the two key target words "tumor location" and "tumor boundary," a weighted summation is chosen as the aggregation operation method. The weight of the target word "tumor location" is preset to 0.5, and the weight of "tumor boundary" is preset to 0.4. The initial contribution value for the visual word "normal brain tissue" is 0.1. A weighted summation is performed on the contribution coefficients in the first row (0.92×0.5+0.88×0.4+0.15×0.1=0.46+0.352+0.015=0.827) to obtain the initial contribution value for the visual word "tumor core region". A weighted summation is then performed on the contribution coefficients in the fifth row (0.11×0.5+0.09×0.4+0.94×0.1=0.055+0.036+0.094=0.185) to obtain the initial contribution value for the visual word "normal brain tissue". Similarly, the initial contribution values ​​for the remaining eight visual words are calculated. Finally, S133 is executed to statistically analyze the initial values ​​of all visual words. The maximum contribution value was 0.89 (corresponding to the visual term "tumor edge region") and the minimum was 0.12 (corresponding to the visual term "image background"). A linear normalization method was used to map each initial contribution value to a preset numerical range of [0,1]. The final contribution value of the visual term "tumor core region" was (0.827-0.12) / (0.89-0.12)≈0.918, and the final contribution value of the visual term "normal brain tissue" was (0.185-0.12) / (0.89-0.12)≈0.084. This clearly quantifies the importance of different visual terms in medical tasks and provides a basis for subsequent cropping and retention of key tumor-related visual terms.

[0043] S140. The visual word set is cropped according to the contribution value of the visual word and the preset contribution threshold to obtain the target visual word set.

[0044] In this embodiment, the preset contribution threshold is a critical value pre-set according to the requirements of the multimodal visual language task, used to distinguish the importance of visual words. Its value needs to balance the requirements of "reducing the amount of computation" and "preserving key information". Pruning refers to the operation of removing low-importance visual words from the visual word set and retaining high-importance visual words. The target visual word set is the set of visual words obtained after pruning, which only contains visual words that make important contributions to the task, and can reduce the subsequent computational burden.

[0045] Specifically, the contribution value of each visual word element calculated in the previous stage is first obtained, while a preset contribution threshold is retrieved (this threshold can be calibrated based on historical task data or dynamically adjusted according to the real-time task type). Then, the contribution value of each visual word element is compared with the preset contribution threshold one by one, and visual words elements that need to be retained and those that need to be cropped are selected based on the comparison results. Next, the selected visual words elements are checked for reasonableness to ensure that the retained words elements can cover the key visual information required by the task and to avoid information loss due to excessive cropping. Finally, the retained visual words elements are integrated to form the final target visual word element set. After obtaining the target visual word element set, it is input together with the target word element set into the subsequent inference module of the multimodal visual language model to complete the image question answering task. Specifically, existing greedy heuristic-based pruning methods are prone to mispruning key tokens or retaining redundant tokens, and dynamic pruning leads to instability in the KV cache. This step uses a fixed judgment criterion of "contribution value + preset threshold" for pruning, which can accurately identify highly important visual words and avoid mispruning or redundant retention, while maintaining the stability of the KV cache without dynamically adjusting the pruning strategy. This solves the technical problems of inaccurate pruning and incompatibility with KVCache. Through this step, the number of visual words can be effectively reduced, and the amount of computation and memory usage can be reduced, while ensuring that the target visual word set contains the key information required for the task. This provides a guarantee for the efficiency and accuracy of subsequent model inference, and it is also compatible with acceleration technologies such as FlashAttention to further improve inference efficiency.

[0046] In one embodiment, such as Figure 7 As shown, step S140 includes: S141-S144.

[0047] S141. Compare the contribution value of each visual word element with a preset contribution threshold, and select visual words elements with contribution values ​​not lower than the preset contribution threshold as words to be retained, and select visual words elements with contribution values ​​lower than the preset contribution threshold as words to be pruned. S142. Retain all the aforementioned words to be retained to form an initial visual word set after preliminary trimming; S143. Perform uniform sampling on the word units to be cropped, and select the word units to be cropped that are associated with key regions of the image or contain image detail features from the sampling results as word units to be recovered. S144. The word to be recovered is added to the initial visual word set to obtain the final target visual word set.

[0048] In this embodiment, the words to be retained are visual words with a contribution value not lower than a preset contribution threshold, which play an important role in completing the multimodal task; the words to be cropped are visual words with a contribution value lower than the preset contribution threshold, which usually contain redundant information or information with little impact on the task; the initial visual word set after preliminary cropping is a set that only retains the words to be retained, and word restoration has not yet been performed; uniform sampling refers to a sampling method that selects a portion of words from the words to be cropped with equal probability, which can avoid sampling bias; key regions of the image refer to regions in the image that are important to the task, such as regions containing core targets, key textures, or semantic information; image detail features refer to local subtle features in the image that can assist in task judgment, such as edges, textures, local structures, etc.; the words to be restored are words sampled from the words to be cropped that contain key region association information or detail features, and need to be added back to the initial visual word set to avoid information loss.

[0049] Specifically, all visual lexical units and their corresponding contribution values ​​are traversed. Each contribution value is compared with a preset contribution threshold. If the contribution value is greater than or equal to the threshold, the visual lexical unit is marked as a unit to be retained; if the contribution value is less than the threshold, it is marked as a unit to be cropped, ensuring that each visual lexical unit is accurately classified. All visual lexical units marked as units to be retained are directly extracted and arranged according to their spatial or logical association order in the original visual lexical unit set to form an initial visual lexical unit set after preliminary cropping. First, the key regions and detailed features to be retained in the question image are identified using image analysis tools. Then, a uniform sampling algorithm is used to traverse the units to be cropped to determine whether each unit to be cropped is associated with a key region or contains detailed features. If either condition is met, it is marked as a unit to be restored, ensuring that the sampling results can cover potential key information. All units to be restored are supplemented into the initial visual lexical unit set after preliminary cropping according to their original spatial or feature association relationships, and the final target visual lexical unit set is formed after integration. Specifically, existing technologies often suffer from the loss of fine-grained information by completely pruning low-contribution lexical units, impacting performance, especially in tasks requiring detailed information such as OCR and VQA. This step addresses this issue by employing a "prune first, then restore" process. It reduces most redundant lexical units through threshold pruning and restores the lexical units containing key information through uniform sampling, thus resolving the information loss problem caused by complete pruning. This step significantly reduces the number of visual lexical units, lowering computational costs and memory requirements, while avoiding the loss of key details. It ensures that the target visual lexical set is both "lightweight" and "information complete," improving the model's adaptability and performance stability across various multimodal tasks.

[0050] To further understand the method of the embodiments of the present invention, the following description is based on an example scenario.

[0051] For example, taking the "Vehicle Accident Damage Assessment (VQA) Task" in an insurance scenario, the user inputs a multi-angle image of the accident vehicle (including damaged areas such as scratches on the body, broken headlights, and deformed bumpers), and the question text is "Please identify the damaged parts of the vehicle and estimate the range of repair costs." The process of executing steps S110-S140 is as follows: First, S110 is executed, receiving the accident vehicle image and damage assessment question text input by the user. The accident vehicle image is input into a preset visual encoder (such as an encoder based on ResNet), and through feature extraction and encoding, it is converted into a visual word set containing features of various local areas of the vehicle (such as the body, headlights, and bumpers). At the same time, the question text is input into a language model (such as BERT-bass). The e-model, after semantic parsing and encoding, yields a target word set containing semantic information such as "damage location identification" and "repair cost estimation." Then, step S120 is executed, calling a preset visual perception cost function based on the visual word set and the target word set. This function combines feature similarity (e.g., the cosine similarity between the visual word "headlight damage" and the target word "damage location identification"), spatial relationships (e.g., the relative spatial distance between the visual word "damage area" and the key vehicle component area), and center distance (e.g., the absolute distance from the center point of the visual word "damage" to the image center). With the optimal transmission cost from visual word to target word as the objective, a contribution matrix is ​​calculated using the optimal transmission algorithm. The matrix shows the contribution of the visual word "headlight damage" to the target word "damage location identification." The coefficient is 0.91, the contribution coefficient to "estimated repair costs" is 0.85, and the contribution coefficients of the visual word "non-damaged vehicle body area" to both target words are less than 0.2; then S130 is executed, the contribution coefficient of each row corresponding to a single visual word is determined according to the contribution matrix, and the coefficients of each row are aggregated by weighted summation ("damaged part recognition" weight 0.6, "estimated repair costs" weight 0.4), resulting in an initial contribution value of 0.886 for the visual word "damaged headlights" and 0.18 for "non-damaged vehicle body area". Then, all initial contribution values ​​are mapped to the [0,1] interval by linear normalization, resulting in a final contribution value of 0.92 for "damaged headlights" and 0.15 for "non-damaged vehicle body area". Finally, S140 is executed, comparing the final contribution value of each visual word with the preset contribution threshold (calibrated to 0.3 based on historical damage assessment tasks). Words with contribution values ​​not lower than the threshold, such as "damaged headlights" and "deformed bumpers," are selected to be retained, while words with contribution values ​​lower than the threshold, such as "non-damaged areas of the vehicle body" and "background areas," are selected to be clipped. The retained words form a preliminary clipped visual word set. Then, the clipped words related to the details of vehicle edge damage in the unretained words are uniformly sampled (e.g., sampling words related to minor scratches on the vehicle body) as words to be restored and added to the preliminary clipped set to obtain the target visual word set. This word set contains only key visual information related to the damage assessment task, which can support the subsequent model to accurately identify the damaged parts and estimate the repair costs.

[0052] In summary, existing technologies are incompatible with Flash Attention and KV Cache because their visual token pruning relies on a greedy heuristic standard based on attention scores. This strategy requires additional computational steps to count and sort the attention scores of each visual token outside of the model's regular inference process. This cannot meet the requirement of Flash Attention for a simple computational process, making it difficult to fully utilize its advantages in accelerating computation and reducing memory access overhead. At the same time, the dynamic pruning method causes frequent changes in the KV cache size, which undermines cache stability. Instead of effectively utilizing the cache, it increases memory requirements, thus failing to achieve compatibility with both. This invention models the visual token pruning problem as an optimal transmission optimization problem. It introduces a pre-defined visual perception cost function that combines visual token feature similarity, spatial relationships, and center distance to accurately calculate the contribution matrix of each visual word to the target word. Then, based on the contribution matrix, it obtains the contribution value of each visual word, replacing the greedy heuristic criterion that relies on attention scores in existing technologies. This avoids the problem of misjudging token importance due to local attention weights, thus solving the problem of inaccurate visual token pruning. Furthermore, this invention performs token pruning only once during the inference phase through a fixed process of calculating the contribution matrix → determining the contribution value → pruning based on a threshold and selectively restoring, without adding extra calculation steps beyond regular inference. This not only adapts to the memory access optimization mechanism of Flash Attention but also avoids changes in the KV cache size caused by dynamic pruning, maintaining KV cache stability and solving the problems of requiring additional calculation steps and compatibility with Flash Attention and KV... The cache incompatibility issue was addressed. Ultimately, by reducing unimportant visual tokens, computational load and memory usage were significantly reduced, especially during inference, effectively shortening inference time and achieving the technical effect of "improving inference efficiency." Furthermore, the one-time token pruning not only reduced memory usage but also maintained the stability of the KV cache, thereby improving the model's memory utilization and achieving the technical effect of "memory saving."

[0053] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0054] This invention also provides a lexical trimming device for a multimodal visual language model, which corresponds one-to-one with the lexical trimming method for the multimodal visual language model described in the above embodiments. For example... Figure 8As shown, the lexical trimming device 200 of the multimodal visual language model includes: a conversion unit 201, a transmission cost unit 202, a contribution calculation unit 203, and a trimming unit 204. Detailed descriptions of each functional unit are as follows: The conversion unit 201 is used to receive a question image and a question text input by the user, convert the question image into a visual word set through a visual encoder, and process the question text through a language model to obtain a target word set; The transmission cost unit 202 is used to calculate a contribution matrix based on the visual word set and the target word set using a preset visual perception cost function with the optimal transmission cost from the visual word to the target word as the objective. The contribution matrix represents the contribution of the visual word to the target word. The contribution calculation unit 203 is used to calculate the contribution value corresponding to each visual word element according to the contribution matrix. The cropping unit 204 is used to crop the visual word set according to the contribution value of the visual word and the preset contribution threshold to obtain the target visual word set.

[0055] In one embodiment, the conversion unit 201 is further configured to: receive a question image input by a user; preprocess the question image to remove noise interference and adjust the image size to an input size adapted to the visual encoder; input the preprocessed question image to a preset visual encoder; extract features from the image through the convolutional and pooling layers of the visual encoder to obtain a global visual feature map of the image; perform block processing on the global visual feature map, dividing the feature map into multiple local feature blocks of the same size, each local feature block corresponding to a local region of the image; perform dimension mapping and feature encoding on each local feature block through the fully connected layer or attention layer of the visual encoder, converting each local feature block into a feature vector of a preset dimension; and integrate all the feature vectors to form a visual lexicon set containing multiple preset dimension feature vectors.

[0056] In one embodiment, the conversion unit 201 is further configured to: receive a question text input by a user; preprocess the question text by removing invalid characters and standardizing the text format to obtain a uniformly formatted question text to be processed; input the standardized question text to be processed into a preset language model; split the question text to be processed into multiple continuous text segments according to preset segmentation rules through the text segmentation module of the language model; perform semantic encoding on each text segment through the embedding layer of the language model, converting each text segment into a semantic feature vector of preset dimensions; perform validity verification on all semantic feature vectors, remove feature vectors with missing semantic information or noise interference, and retain semantic feature vectors that meet preset semantic integrity requirements; and integrate all retained semantic feature vectors to form a target word set containing multiple preset dimension semantic feature vectors.

[0057] In one embodiment, the transmission cost unit 202 is further configured to: calculate the transmission cost from each visual word to each target word using the preset visual perception cost function based on the visual word set and the target word set, forming a transmission cost matrix; and solve the transmission cost matrix to obtain the contribution matrix with the objective of minimizing the total transmission cost.

[0058] In one embodiment, the transmission cost unit 202 is further configured to: extract feature vectors of each visual word from the visual word set, extract feature vectors of each target word from the target word set, and calculate the similarity between pairwise feature vectors to obtain feature similarity parameters; obtain spatial location information of each visual word in the question image, and calculate the relative spatial distance between different visual words and their corresponding target words based on the spatial location information to obtain spatial relationship parameters; determine the center coordinates of the question image, and calculate the absolute distance from the center point of each visual word to the center coordinates of the image to obtain center distance parameters; input the feature similarity parameters, spatial relationship parameters, and center distance parameters into the preset visual perception cost function, and obtain the transmission cost from each visual word to each target word through function operation.

[0059] In one embodiment, the contribution calculation unit 203 is further configured to: obtain a contribution matrix characterizing the degree of contribution of each visual word to the target word, and determine the contribution coefficient of each row element in the matrix to all target words; perform aggregation operation on all contribution coefficients in each row of the contribution matrix, the aggregation operation including summation, weighted summation or taking the maximum value, to obtain the initial contribution value of the visual word corresponding to each row; and normalize the initial contribution values ​​of all visual words to map the initial contribution values ​​to a preset numerical range to obtain the final contribution value of each visual word.

[0060] In one embodiment, the cropping unit 204 is further configured to: compare the contribution value of each visual word with a preset contribution threshold, select visual words with contribution values ​​not lower than the preset contribution threshold as words to be retained, and select visual words with contribution values ​​lower than the preset contribution threshold as words to be cropped; retain all the words to be retained to form an initial visual word set after preliminary cropping; uniformly sample the words to be cropped, and select the words to be cropped that are associated with key regions of the image or contain image detail features from the sampling results as words to be restored; supplement the words to be restored to the initial visual word set to obtain the final target visual word set.

[0061] Specific limitations regarding the lexical trimming device for multimodal visual language models can be found in the limitations of the lexical trimming method for multimodal visual language models described above, and will not be repeated here. Each module in the aforementioned lexical trimming device for multimodal visual language models can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0062] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the server-side functions or steps of a lexical trimming method for a multimodal visual language model.

[0063] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a lexical segmentation method for a multimodal visual language model. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the lexical trimming method of the multimodal visual language model described above.

[0064] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the lexical trimming method for the multimodal visual language model described above.

[0065] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0066] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0067] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0068] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A lexical trimming method for a multimodal visual language model, characterized in that, The method includes: The system receives a question image and question text input by the user, converts the question image into a visual word set through a visual encoder, and processes the question text through a language model to obtain a target word set. Based on the visual word set and the target word set, a contribution matrix is ​​calculated using a preset visual perception cost function with the optimal transmission cost from the visual word to the target word as the objective. The contribution matrix represents the contribution of the visual word to the target word. Calculate the contribution value corresponding to each visual lexical unit based on the contribution matrix; The target visual word set is obtained by pruning the visual word set according to the contribution value of the visual word and the preset contribution threshold.

2. The method according to claim 1, characterized in that, The step of converting the question image into a visual word set using a visual encoder includes: Receive a question image input by the user, preprocess the question image to remove noise interference in the image and adjust the image size to the input size adapted to the visual encoder; The preprocessed query image is input into a preset visual encoder. The convolutional and pooling layers of the visual encoder are used to extract features from the image to obtain a global visual feature map of the image. The global visual feature map is divided into blocks, which are then divided into multiple local feature blocks of the same size. Each local feature block corresponds to a local region of the image. Each local feature block is dimensionally mapped and its features are encoded by the fully connected layer or attention layer of the visual encoder, and each local feature block is converted into a feature vector of a preset dimension. All the aforementioned feature vectors are integrated to form a visual lexicon set containing feature vectors of multiple preset dimensions.

3. The method according to claim 1, characterized in that, The step of processing the question text using a language model to obtain the target word set includes: Receive the question text input by the user, preprocess the question text, remove invalid characters from the text and standardize the text format to obtain a uniformly formatted question text to be processed; The standardized question text to be processed is input into a preset language model. The text segmentation module of the language model splits the question text into multiple continuous text segments according to the preset segmentation rules. Through the embedding layer of the language model, each text segment is semantically encoded, and each text segment is converted into a semantic feature vector of a preset dimension; The validity of all the semantic feature vectors is verified, and feature vectors with missing semantic information or noise interference are removed, while semantic feature vectors that meet the preset semantic integrity requirements are retained. All retained semantic feature vectors are integrated to form a target word set containing semantic feature vectors of multiple preset dimensions.

4. The method according to claim 1, characterized in that, The step of calculating the contribution matrix based on the visual lexicon set and the target lexicon set using a preset visual perception cost function with the optimal transmission cost from visual lexicon to target lexicon as the objective includes: Based on the visual word set and the target word set, the transmission cost from each visual word to each target word is calculated using the preset visual perception cost function, forming a transmission cost matrix; The contribution matrix is ​​obtained by solving the transmission cost matrix with the objective of minimizing the total transmission cost.

5. The method according to claim 4, characterized in that, The step of calculating the transmission cost from each visual word to each target word using the preset visual perception cost function based on the visual word set and the target word set, and forming a transmission cost matrix, includes: The feature vector of each visual word is extracted from the visual word set, the feature vector of each target word is extracted from the target word set, and the similarity between each pair of feature vectors is calculated to obtain the feature similarity parameter. The spatial location information of each visual word in the question image is obtained, and the relative spatial distance between different visual words and their corresponding target words is calculated based on the spatial location information to obtain spatial relationship parameters. Determine the center coordinates of the question image, and calculate the absolute distance from the center point of each visual word to the center coordinates of the image to obtain the center distance parameter; The feature similarity parameters, spatial relationship parameters, and center distance parameters are input into the preset visual perception cost function, and the transmission cost from each visual word to each target word is obtained through function operation.

6. The method according to claim 1, characterized in that, The step of calculating the contribution value corresponding to each visual lexical unit based on the contribution matrix includes: Obtain a contribution matrix that represents the degree of contribution of each visual word to the target word, and determine the contribution coefficient of each element in the matrix to all target words. An aggregation operation is performed on all contribution coefficients in each row of the contribution matrix. The aggregation operation includes any one of summation, weighted summation, or taking the maximum value to obtain the initial contribution value of the visual word corresponding to each row. The initial contribution values ​​of all visual words are normalized and mapped to a preset numerical range to obtain the final contribution value of each visual word.

7. The method according to claim 1, characterized in that, The step of pruning the visual word set according to the contribution value of the visual word and a preset contribution threshold to obtain the target visual word set includes: The contribution value of each visual word element is compared with a preset contribution threshold. Visual words elements with contribution values ​​not lower than the preset contribution threshold are selected as words to be retained, and visual words elements with contribution values ​​lower than the preset contribution threshold are selected as words to be pruned. All the aforementioned words to be retained are retained to form an initial visual word set after preliminary trimming; The words to be cropped are uniformly sampled, and the words to be cropped that are associated with key regions of the image or contain detailed features of the image are selected from the sampling results as words to be recovered. The lexical units to be recovered are added to the initial visual lexical set to obtain the final target visual lexical set.

8. A word segmentation device for a multimodal visual language model, characterized in that, Includes a unit for performing the method according to any one of claims 1-7.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images