Method, apparatus, equipment, and medium for creating image-descriptive text based on visual lexical pruning
By constructing a visual word semantic distance matrix and adopting a greedy selection strategy, the problems of high computational cost and performance degradation in the pruning process of visual question answering models are solved, and the efficient deployment of visual question answering models in resource-constrained environments is realized.
Patent Information
- Application Number
- CN202611327589.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-28
- Publication Date
- 2026-11-17
AI Technical Summary
Existing visual question answering models suffer from high computational costs, long processing times, lack of versatility and flexibility during the pruning process, and severe performance degradation under high pruning rates, making them particularly difficult to deploy in resource-constrained environments.
By constructing a semantic distance matrix between visual lexical units and employing a greedy selection strategy, visual lexical units with the largest minimum semantic distance or the smallest semantic difference are selected round by round to form a pruned set of visual lexical units, which is then directly input into a large language model to generate descriptive text.
Without relying on additional training or calibration, the number of visual lexical units is significantly reduced after pruning, which reduces the computational complexity and memory usage of large language models, shortens inference time, and enables the effective deployment of visual question answering models in resource-constrained environments.
Smart Images

Figure CN123117341A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and medium for creating image-descriptive text based on visual lexical pruning. Background Technology
[0002] Visual question answering models have made significant progress in computer vision and natural language processing in recent years, enabling them to perform complex tasks such as image description, visual question answering, and video understanding. Visual question answering models first encode the input image or video into hundreds or thousands of visual words, then concatenate these visual words with text words before inputting them into a large language model for processing. However, since the computational complexity and memory consumption of large language models are quadratic of the length of their input sequence, the introduction of a large number of visual words leads to severe inference latency and memory consumption problems, severely limiting the deployment of visual question answering models in real-time applications and resource-constrained environments.
[0003] While various visual lexical pruning methods exist, they often tend to retain a large number of similar lexical units, or require additional calibration datasets or model fine-tuning to adapt to the pruned structure. These processes are computationally expensive, time-consuming, and need to be repeated for different models, lacking versatility and flexibility. Current visual lexical pruning methods either require costly training or calibration processes, or, at high pruning rates, suffer severe performance degradation due to excessive redundancy in the selected lexical units.
[0004] In summary, how to retain a highly representative subset of visual lexical units and avoid information redundancy among selected lexical units at extremely high pruning rates without relying on additional training or calibration is a problem that needs to be solved in this field. Summary of the Invention
[0005] The purpose of this invention is to provide a method, apparatus, device, and medium for creating image-descriptive text based on visual lexical pruning. This method retains a highly representative subset of lexical units in the visual information and avoids information redundancy between selected lexical units, even at extremely high pruning rates, without relying on additional training or calibration. The specific solution is as follows: In a first aspect, this invention discloses an image description text creation method based on visual lexical pruning, applied to a target visual question answering model, wherein the target visual question answering model includes a visual encoder and a large language model, wherein the method includes: A visual encoder is used to obtain multiple visual words of the input image and text words corresponding to the input image; A distance matrix is constructed based on the semantic distance between each visual word, and the set of visual words is determined as the candidate word set. The selected word set is initialized as an empty set. In the first round, the first minimum semantic distance between each visual word in the candidate word set and other visual words is determined from the distance matrix. The visual word corresponding to the maximum value of each of the first minimum semantic distances is determined as the target visual word in this round by a greedy selection strategy. The target visual word is then moved to the selected word set and the process jumps to the next round. In non-first rounds, the second minimum semantic distance between each visual word in the candidate word set and each target visual word in the selected word set is determined from the distance matrix. The visual word corresponding to the maximum value of each second minimum semantic distance is determined as the target visual word in this round through a greedy selection strategy. The target visual word is then moved to the selected word set and the process jumps to the next round until the preset selection stop condition is met. The candidate word set is discarded, and the target visual word in the selected word set is determined as the pruned visual word. The pruned visual lexical units are concatenated with the text lexical units to obtain the input lexical unit sequence; The input word sequence is fed into a large language model so that the large language model generates descriptive text corresponding to the input image.
[0006] Optionally, the step of using a visual encoder to obtain multiple visual units of the input image and text units corresponding to the input image includes: The input image is divided into multiple non-overlapping image blocks, and each image block is flattened into a one-dimensional vector. Each of the one-dimensional vectors is input into a visual encoder, and the visual encoder outputs the feature vector corresponding to each of the image blocks. Each of the aforementioned feature vectors is input into the projection layer, and the projection layer maps each of the aforementioned feature vectors to the same dimension as the input embedding space of the large language model, thereby obtaining multiple visual lexical units; The input text corresponding to the input image is embedded through the text embedding layer of the large language model to obtain text lexical units.
[0007] Optionally, before determining the first minimum semantic distance between each visual word in the candidate word set and other visual words from the distance matrix, the method further includes: Obtain the total number of visual words corresponding to the input image and the preset pruning rate; The theoretical number of visual words after pruning is determined by multiplying the total number by the preset pruning rate. The theoretical number is rounded down to obtain the first pruning threshold. A preset selection stop condition is established based on the first pruning quantity threshold; Accordingly, the jump to the next round includes: If the number of target visual words in the selected word set is less than the first pruning threshold in the preset selection stop condition, then proceed to the next round.
[0008] Optionally, before determining the first minimum semantic distance between each visual word in the candidate word set and other visual words from the distance matrix, the method further includes: Obtain the total number of visual lexical units corresponding to the input image; The complexity of the input image is evaluated to obtain a complexity index of the input image; Determine the target pruning rate that matches the complexity of the input image based on the complexity index; The second pruning number threshold is determined by multiplying the total number by the target pruning rate; A preset selection stop condition is established based on the second pruning quantity threshold; Accordingly, the jump to the next round includes: If the number of target visual words in the selected word set is less than the second pruning threshold in the preset selection stop condition, then proceed to the next round.
[0009] Optionally, constructing a distance matrix based on the semantic distance between each of the visual lexical units includes: Each visual word is normalized using the L2 norm to obtain a normalized vector corresponding to each visual word. A normalized matrix is constructed based on each of the normalized vectors, and a similarity matrix is constructed based on the product of the normalized matrix and its transpose. Each element in the similarity matrix is subtracted from a preset constant to convert the similarity into semantic distance, and a distance matrix is constructed based on the semantic distance.
[0010] Optionally, determining the set of visual lexical units as the candidate lexical unit set includes: Obtain the attention score corresponding to each visual word, and sort the visual words in descending order of the attention scores to obtain a sorted visual word sequence. Candidate visual words are selected from the head of the sorted visual word sequence according to a preset screening ratio, and the set of selected candidate visual words is determined as the candidate word set.
[0011] Optionally, moving the target visual lexical unit to the selected lexical unit set includes: If there are multiple target visual words in this round, the second minimum semantic distance of the target visual words is determined as the diversity score of the target visual words. Obtain the attention score corresponding to each target visual word and the semantic relevance score between each target visual word and the input text corresponding to the input image; The attention score, semantic relevance score, and diversity score of each target visual word are normalized to obtain the normalized attention score, normalized semantic relevance score, and normalized diversity score of each target visual word. The weights corresponding to the normalized attention score, the normalized semantic relevance score, and the normalized diversity score are determined, and the normalized attention score, the normalized semantic relevance score, and the normalized diversity score are weighted and summed to obtain the comprehensive score of each target visual word. The comprehensive scores are sorted in descending order, and the target visual word with the largest comprehensive score is selected from the target visual word and moved to the selected word set.
[0012] Secondly, this invention discloses an image description text creation device based on visual lexical pruning, applied to a target visual question answering model, wherein the target visual question answering model includes a visual encoder and a large language model, wherein the device includes: The lexical acquisition module is used to acquire multiple visual lexical units of the input image and text lexical units corresponding to the input image using a visual encoder; The set determination module is used to construct a distance matrix based on the semantic distance between each visual word element, determine the set of visual word elements as a candidate word element set, and initialize the selected word element set as an empty set; The first selection module is used to determine, in the first round, the first minimum semantic distance between each visual word in the candidate word set and other visual words from the distance matrix, determine the visual word corresponding to the maximum value of each of the first minimum semantic distances as the target visual word in this round through a greedy selection strategy, move the target visual word to the selected word set, and jump to the next round. The second selection module is used to determine, in non-first rounds, the second minimum semantic distance between each visual word in the candidate word set and each target visual word in the selected word set from the distance matrix, and to determine the visual word corresponding to the maximum value of each second minimum semantic distance as the target visual word in the current round through a greedy selection strategy, and to move the target visual word to the selected word set, and jump to the next round, until the preset selection stop condition is met; The pruning module is used to discard the candidate word set and determine the target visual word in the selected word set as the pruned visual word; The lexical concatenation module is used to concatenate the pruned visual lexical units with the text lexical units to obtain the input lexical unit sequence; The text generation module is used to input the input word sequence into a large language model so that the large language model generates descriptive text corresponding to the input image.
[0013] Thirdly, the present invention discloses an electronic device, comprising: Memory, used to store computer programs; A processor for executing a computer program to implement the steps of the aforementioned disclosed method for creating image-descriptive text based on visual lexical pruning.
[0014] Fourthly, the present invention discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed image description text creation method based on visual lexical pruning.
[0015] Therefore, this invention is applied to a target visual question answering model, which includes a visual encoder and a large language model. The method includes: acquiring multiple visual lexical units of an input image and corresponding text lexical units using the visual encoder; constructing a distance matrix based on the semantic distance between each visual lexical unit, and determining the set of visual lexical units as a candidate lexical unit set, initializing the selected lexical unit set as an empty set; in the first round, determining the first minimum semantic distance between each visual lexical unit in the candidate lexical unit set and other visual lexical units from the distance matrix, using a greedy selection strategy to determine the visual lexical unit corresponding to the maximum value among the first minimum semantic distances as the target visual lexical unit in this round, moving the target visual lexical unit to the selected lexical unit set, and jumping to the next round. Next, in rounds other than the first round, the second minimum semantic distance between each visual word in the candidate word set and each target visual word in the selected word set is determined from the distance matrix. A greedy selection strategy is used to identify the visual word corresponding to the maximum value of each of the second minimum semantic distances as the target visual word in this round, and the target visual word is moved to the selected word set, proceeding to the next round, until a preset selection stopping condition is met. The candidate word set is discarded, and the target visual word in the selected word set is identified as pruned visual words. The pruned visual words are concatenated with the text words to obtain an input word sequence. The input word sequence is input into a large language model to generate descriptive text corresponding to the input image.
[0016] The beneficial effects are as follows: This invention constructs a distance matrix based on the semantic distance between each visual word and performs greedy selection based on this distance matrix. This maximizes the minimum semantic distance between the target visual word selected in the first round and all other visual words, and maximizes the minimum semantic distance between the target visual word selected in each round after the first round and each target visual word in the currently selected word set. This ensures that the final pruned visual words are as dispersed as possible in the semantic space and have the greatest difference between each other, avoiding the redundancy problem caused by retaining a large number of similar words in traditional pruning methods. This allows the few visual words retained after pruning to cover the diverse visual information in the original image to the greatest extent. Because the greedy selection strategy only relies on the semantic distance between visual words... Distance-based methods require no additional labeled data, no fine-tuning of model parameters, and no reliance on calibration datasets. Therefore, they can be executed directly as a standalone preprocessing step before model inference, achieving plug-and-play functionality. The number of visual lexical units retained is far less than the number of original visual lexical units, significantly reducing the length of the lexical sequence input to the large language model. Since the computational complexity and memory usage of the large language model are proportional to the square of its input sequence length, this invention directly reduces the number of floating-point operations in attention calculation and feedforward networks of the large language model, shortens the inference time required for the model to generate descriptive text, and reduces the GPU memory space occupied by key-value cache during inference, enabling the visual question-answering model to be deployed and run in hardware environments with limited GPU memory. Attached Figure Description
[0017] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 A flowchart of an image description text creation method based on visual lexical pruning is provided for an embodiment of the present invention; Figure 2 A specific flowchart for creating image description text is provided in this embodiment of the invention; Figure 3 A greedy selection flowchart provided for an embodiment of the present invention; Figure 4 A flowchart illustrating a specific method for creating image-descriptive text based on visual lexical pruning, as provided in this embodiment of the invention; Figure 5 A schematic diagram of an image description text creation device based on visual lexical pruning provided in an embodiment of the present invention; Figure 6This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.
[0020] Visual question answering models have made significant progress in computer vision and natural language processing in recent years, enabling them to perform complex tasks such as image description, visual question answering, and video understanding. Visual question answering models first encode the input image or video into hundreds or thousands of visual words, then concatenate these visual words with text words before inputting them into a large language model for processing. However, since the computational complexity and memory consumption of large language models are quadratic of the length of their input sequence, the introduction of a large number of visual words leads to severe inference latency and memory consumption problems, severely limiting the deployment of visual question answering models in real-time applications and resource-constrained environments.
[0021] While various visual lexical pruning methods exist, they often tend to retain a large number of similar lexical units, or require additional calibration datasets or model fine-tuning to adapt to the pruned structure. These processes are computationally expensive, time-consuming, and need to be repeated for different models, lacking versatility and flexibility. Current visual lexical pruning methods either require costly training or calibration processes, or, at high pruning rates, suffer severe performance degradation due to excessive redundancy in the selected lexical units.
[0022] The terms "comprising" and "having," and any variations thereof, in the specification and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may include steps or units not listed.
[0023] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0024] Next, we will describe in detail an image description text creation scheme based on visual lexical pruning provided by an embodiment of the present invention. Figure 1 A flowchart of an image description text creation method based on visual lexical pruning is provided in this embodiment of the invention. This method is applied to a target visual question answering model, which includes a visual encoder and a large language model. The method includes: Step S11: Use a visual encoder to obtain multiple visual words of the input image and text words corresponding to the input image.
[0025] In this embodiment, the step of obtaining multiple visual lexical units of an input image and corresponding text lexical units using a visual encoder includes: dividing the input image into multiple non-overlapping image blocks and flattening each image block into a one-dimensional vector; inputting each one-dimensional vector into a visual encoder and outputting a feature vector corresponding to each image block through the visual encoder; inputting each feature vector into a projection layer and mapping each feature vector to the same dimension as the input embedding space of the large language model through the projection layer to obtain multiple visual lexical units; and embedding the input text corresponding to the input image through the text embedding layer of the large language model to obtain text lexical units.
[0026] like Figure 2 As shown, the target visual question answering model specifically includes a CLIP visual encoder, a projection layer, a Large Language Model (LLM), and a text encoder. It acquires the input image and the corresponding input text, divides the input image into multiple non-overlapping image patches, and flattens each patch into a one-dimensional vector. Each one-dimensional vector is input to the CLIP (Contrastive Language-Image Pre-training) visual encoder, which outputs the corresponding feature vector. Each feature vector is then input to a projection layer consisting of two MLP (Multilayer Perceptron) layers. The projection layer maps each feature vector to the same dimension as the input embedding space of the Large Language Model, obtaining multiple visual lexical units. Finally, the input text is embedded through the text embedding layer of the Large Language Model to obtain text lexical units.
[0027] Specifically, for example, if the input image is a 336×336 RGB (Red, Green, Blue) image and the input text is "Please describe the content of this image", the input image is divided into multiple non-overlapping image patches, each 14×14 pixels in size. Since the input image resolution is 336×336 pixels, the image is divided into 24×24 patches, totaling 576 patches. Each patch is flattened into a 14×14×3 one-dimensional vector, totaling 588 dimensions. Each one-dimensional vector is input to the CLIP visual encoder, which outputs 576 corresponding feature vectors. Each feature vector is then input to the projection layer, which maps each feature vector to the same 4096-dimensional embedding space as the large language model, resulting in 576 4096-dimensional visual lexical units. The input text is then embedded through the text embedding layer of the large language model to obtain text lexical units.
[0028] Step S12: Construct a distance matrix based on the semantic distance between each visual word, determine the set of visual words as the candidate word set, and initialize the selected word set as an empty set.
[0029] Pre-compute any two lexical units v among all 576 visual lexical units. i and v j The cosine distance d(v) between them i v j The calculated pairwise distances are constructed into a 576×576 distance matrix D, which is completed on the GPU (Graphics Processing Unit) through a single matrix multiplication.
[0030] The set of all 576 visual lexical units is determined as the candidate lexical unit set, or a partial set of visual lexical units is determined as the candidate lexical unit set. The candidate lexical unit set is denoted as R={1,2,…,576}. At the same time, an empty set of selected lexical units Ev= is initialized. Text terms are denoted as Et.
[0031] Step S13: In the first round, determine the first minimum semantic distance between each visual word in the candidate word set and other visual words from the distance matrix. Using a greedy selection strategy, determine the visual word corresponding to the maximum value of each of the first minimum semantic distances as the target visual word in this round, and move the target visual word to the selected word set, and jump to the next round.
[0032] In the first round, for each visual word index i in the candidate word set R, the semantic distance between visual word i and all other visual words j in the candidate word set is obtained from the distance matrix, where j∈R and j≠i. The minimum value min is then found. j≠i {D[i][j]} represents the minimum semantic distance from each visual word to all other visual words. After obtaining all 576 minimum values, we find the word index k corresponding to the maximum value among these minimum values, where k = argmax. i (min j≠i {D[i][j]}) uses a greedy selection strategy based on the Max-Min Diversity Problem (MMDP) to determine the visual word corresponding to index k as the target visual word for this round. The target visual word k is removed from the candidate word set R and added to the selected word set Ev, i.e., Ev={k}. If the number of k is 1, then the size of R is 575, and then the next round is entered.
[0033] In a specific embodiment of the stopping condition establishment, before determining the first minimum semantic distance between each visual word in the candidate word set and other visual words from the distance matrix, the method further includes: obtaining the total number of visual words corresponding to the input image and a preset pruning rate; determining the theoretical number of pruned visual words based on the product of the total number and the preset pruning rate; rounding the theoretical number to obtain a first pruning number threshold; and establishing a preset selection stopping condition based on the first pruning number threshold. For example, if the target pruning rate is set to 90%, that is, only 10% of visual words are retained, the total number of visual words corresponding to the input image is obtained as 576, and the preset pruning rate is 90%. The product of the total number of 576 and the preset pruning rate of 90%, 576 × (100% - 90%) = 57.6, is determined as the theoretical number of visual words after pruning. The theoretical number of 57.6 is rounded to obtain 57 or 58 words. In actual implementation, it can be rounded up to obtain the first pruning number threshold. Based on the first pruning number threshold, a preset selection stop condition is established, that is, the iterative selection stops when the number of words in the selected word set reaches the first pruning number threshold.
[0034] In one specific implementation of the jump, the jump to the next round includes: if the number of target visual words in the selected word set is less than the first pruning threshold in the preset selection stop condition, then jump to the next round. After the first round ends, check whether the number of target visual words in the selected word set is less than the first pruning threshold. If the number of target visual words in the selected word set is less than the first pruning threshold, then jump to the next round to continue the selection. If the number of target visual words in the selected word set reaches the first pruning threshold, then stop the iteration.
[0035] In another specific embodiment of the stopping condition establishment, before determining the first minimum semantic distance between each visual word in the candidate word set and other visual words from the distance matrix, the method further includes: obtaining the total number of visual words corresponding to the input image; evaluating the complexity of the input image to obtain a complexity index; determining a target pruning rate matching the complexity of the input image based on the complexity index; determining a second pruning number threshold based on the product of the total number and the target pruning rate; and establishing a preset selection stopping condition based on the second pruning number threshold. The steps are as follows: obtaining the total number of visual words corresponding to the input image; evaluating the complexity of the input image to obtain a complexity index; determining a target pruning rate matching the complexity of the input image based on the complexity index (higher complexity results in a lower target pruning rate and more retained words); determining a second pruning number threshold based on the product of the total number and the target pruning rate; and establishing a preset selection stopping condition based on the second pruning number threshold. For example, after evaluating the complexity of a solid-color background image containing a single object, a low complexity index is obtained. Based on this, the target pruning rate is determined to be 95%, and the total number of visual words is 576. The product is 576 × (100% - 95%) = 28.8. After rounding, the second pruning threshold is 29 words. That is, the preset selection stopping condition is to stop when the number of selected words reaches 29. After evaluating the complexity of a complex street scene image containing multiple objects and rich textures, a high complexity index is obtained. Based on this, the target pruning rate is determined to be 80%, and the product is 576 × (100% - 80%) = 115.2. After rounding, the second pruning threshold is 115 words. That is, the preset selection stopping condition is to stop when the number of selected words reaches 115.
[0036] In another specific implementation of the jump, the jump to the next round includes: if the number of target visual words in the selected word set is less than the second pruning threshold in the preset selection stop condition, then jump to the next round. After the first round ends, check whether the number of target visual words in the selected word set is less than the second pruning threshold. If the number of target visual words in the selected word set is less than the second pruning threshold, then jump to the next round to continue the selection. If the number of target visual words in the selected word set reaches the second pruning threshold, then stop the iteration.
[0037] Step S14: In non-first rounds, determine the second minimum semantic distance between each visual word in the candidate word set and each target visual word in the selected word set from the distance matrix. Using a greedy selection strategy, determine the visual word corresponding to the maximum value of each second minimum semantic distance as the target visual word in this round, and move the target visual word to the selected word set. Jump to the next round until the preset selection stop condition is met.
[0038] like Figure 3 As shown, in rounds other than the first round, for each word index i in the candidate word set, the semantic distance between that word and each target visual word in the currently selected word set is obtained from the distance matrix. For the same candidate visual word i, i∈R, the minimum value min among its semantic distances with all selected words is found. j∈Ev {D[i][j]} represents the minimum semantic distance from the candidate word to the currently selected word set. After obtaining the minimum value for each candidate word in the candidate word set, find the word index k corresponding to the maximum value among these minimum values, where k = argmax. i ∈R(min j∈Ev {D[i][j]}) determines the visual word corresponding to index k as the target visual word for this round, removes the target visual word from the candidate word set and adds it to the selected word set, checks whether the number of target visual words in the selected word set meets the preset selection stop condition. If it meets the condition, the iteration stops; otherwise, it jumps to the next round to continue the selection. For example, if the number of target visual words in the selected word set is less than the first pruning threshold or less than the second pruning threshold, it jumps to the next round to continue the selection. If the number of target visual words in the selected word set is not less than the first pruning threshold or not less than the second pruning threshold, the iteration stops.
[0039] Understandably, since the distance matrix D has been pre-calculated, the minimum value is calculated in each iteration. j∈Ev{D[i][j]} only needs to read the values of the selected columns from D and find the minimum value. On a GPU, parallel reduction operations can be used to speed it up. For 576 words, the entire selection process usually takes less than 10 milliseconds on a GPU, which is negligible compared to the inference time of LLM.
[0040] Step S15: Discard the candidate word set and determine the target visual word in the selected word set as the pruned visual word.
[0041] When the number of target visual lexical units in the selected lexical unit set reaches the preset selection stopping condition, the iteration stops. All unselected visual lexical units in the candidate lexical unit set are discarded and not input into the large language model. All target visual lexical units in the selected lexical unit set are identified as pruned visual lexical units and used as the visual part input into the large language model. For example, the index set Ev of the 57 finally selected visual lexical units is retained, and the remaining 576-57=519 visual lexical units are discarded.
[0042] Furthermore, the specific process of determining the target visual lexical units in the selected lexical unit set as pruned visual lexical units is as follows: All target visual lexical units in the selected lexical unit set are constructed into an initial solution set. This initial solution set is simultaneously determined as both the current optimal solution set and the current solution set, and the tabu list is initialized to empty. Tabu search optimization is performed on the current solution set, iteratively executing discard and add operations to obtain a new solution set for the current round. The discard operation involves randomly removing a visual lexical unit from the current solution set and adding the removed visual lexical unit to the tabu list. The addition operation involves selecting a visual lexical from the candidate lexical set and adding it to the current solution set. After each iteration, the diversity objective function value of the new solution set in the current round is determined. When the diversity objective function value is greater than the diversity objective function value of the current optimal solution set, the new solution set in the current round is taken as the new current optimal solution set. At the same time, the new solution set in the current round is taken as the current solution set for the next iteration. This process is repeated until a preset iteration stopping condition is met, and all visual lexicals in the final retained current optimal solution set are determined as the pruned visual lexicals. By introducing tabu search to perform secondary optimization on the greedy selection results, and continuing to search for more diverse lexical combinations based on the greedy solution, the final retained visual lexical subset is more evenly distributed in the semantic space. The tabu list avoids repeatedly selecting recently discarded lexicals in the short term, preventing the search from getting trapped in local optima. Since the distance matrix has been pre-calculated before the search begins, each evaluation of the diversity objective function value of the new solution set only requires reading the corresponding elements from the distance matrix, without recalculation. Therefore, this optimization process significantly improves the diversity quality of the pruned lexical subset while increasing a small amount of computational overhead.
[0043] Step S16: Concatenate the pruned visual lexical units with the text lexical units to obtain the input lexical unit sequence.
[0044] The pruned visual lexical units are concatenated with the original text lexical units to form an input lexical unit sequence. For example, if the final selected pruned visual lexical units are 57 and the selected text lexical units are 15, concatenating the 57 pruned visual lexical units with the 15 text lexical units yields an input lexical unit sequence of length 72. The original number of visual lexical units is 576, and the original sequence length after concatenating the original text lexical units is 591. The length of the input lexical unit sequence is reduced to approximately 87.8% of the original.
[0045] Step S17: Input the input word sequence into the large language model so that the large language model generates descriptive text corresponding to the input image.
[0046] The concatenated input word sequence is fed into the large language model for autoregressive generation. As the length of the input sequence is reduced from 591 to 72, the computational complexity of the Transformer's attention is reduced from O(591²) to O(72²). The theoretical computational cost is reduced to about 1.48% of the original, and the actual computational cost is about 15% of the original model. The large language model finally generates descriptive text for the input image.
[0047] This invention constructs a distance matrix based on the semantic distance between visual words and performs greedy selection based on this matrix. This maximizes the minimum semantic distance between the target visual word selected in the first round and all other visual words, and maximizes the minimum semantic distance between the target visual word selected in each round after the first round and all target visual words in the currently selected word set. This ensures that the final pruned visual words are as dispersed as possible in the semantic space and have the greatest difference between them, avoiding the redundancy problem caused by retaining a large number of similar words in traditional pruning methods. This allows the few visual words retained after pruning to cover the diverse visual information in the original image to the greatest extent possible. Because the greedy selection strategy relies only on the semantic distance between visual words... No additional labeled data, no fine-tuning of model parameters, and no reliance on calibration datasets are required. Therefore, it can be executed directly as a standalone preprocessing step before model inference, achieving plug-and-play functionality. The number of visual lexical units retained is far less than the number of original visual lexical units, significantly reducing the length of the lexical sequence input to the large language model. Since the computational complexity and memory usage of the large language model are proportional to the square of its input sequence length, this invention directly reduces the number of floating-point operations in attention calculation and feedforward network of the large language model, shortens the inference time required for the model to generate descriptive text, and reduces the GPU memory space occupied by key-value cache during inference, enabling the visual question answering model to be deployed and run in hardware environments with limited GPU memory.
[0048] like Figure 4 As shown, this embodiment of the invention discloses a specific method for creating image-descriptive text based on visual lexical pruning. Compared to the previous embodiment, this embodiment further explains and optimizes the technical solution. Applied to a target visual question-answering model, the target visual question-answering model includes a visual encoder and a large language model, wherein the method includes: Step S21: Use a visual encoder to obtain multiple visual words of the input image and text words corresponding to the input image.
[0049] Step S22: Perform L2 norm normalization on each visual word to obtain a normalized vector corresponding to each visual word; construct a normalization matrix based on each normalized vector, and construct a similarity matrix based on the product of the normalization matrix and its transpose; subtract each element in the similarity matrix from a preset constant to convert the similarity into semantic distance, and construct a distance matrix based on the semantic distance.
[0050] To accelerate the subsequent diversity maximization selection process, a pairwise cosine distance matrix D of size 576 × 576 is first pre-computed among all visual lexical units. For any two lexical units v i and v j The cosine distance is defined as: d(v i v j )=1-(v i ·v j ) / (||v i ||×||v j ||); Where · represents the vector dot product, ||·|| represents the L2 norm, and the distance ranges from [0, 2], where 0 represents identical vectors and 2 represents identical vectors.
[0051] Specifically, L2 norm normalization is performed on all visual lexical units, that is, each visual lexical unit vector is divided by its own L2 norm to obtain the normalized vector corresponding to each visual lexical unit; all normalized vectors are arranged in rows to construct a normalization matrix, and the product of the normalization matrix and its transpose is calculated to obtain the similarity matrix. Each element in the similarity matrix represents the cosine similarity between two corresponding visual lexical units; each element in the similarity matrix is subtracted from the constant 1 to convert the cosine similarity into cosine distance, and a distance matrix is constructed based on all the calculated pairwise cosine distances.
[0052] Step S23: Determine the set of visual lexical elements as the candidate lexical element set, and initialize the selected lexical element set as an empty set.
[0053] In one specific embodiment, the set of all 576 visual words obtained after processing by the visual encoder and the projection layer is determined as the candidate word set. Each visual word in the candidate word set is a 4096-dimensional vector. The candidate word set will gradually decrease in subsequent rounds as the target visual word is removed.
[0054] In another specific embodiment, determining the set of visual lexical units as a candidate lexical unit set includes: obtaining the attention score corresponding to each visual lexical unit, and sorting the visual lexical units in descending order of the attention scores to obtain a sorted visual lexical unit sequence; selecting candidate visual lexical units from the head of the sorted visual lexical unit sequence according to a preset screening ratio, and determining the set of selected candidate visual lexical units as the candidate lexical unit set. Specifically, obtaining the attention score corresponding to each visual lexical unit, sorting all visual lexical units in descending order of the attention scores to obtain a sorted visual lexical unit sequence; selecting a portion of visual lexical units from the head of the sorted visual lexical unit sequence as candidate visual lexical units according to a preset screening ratio, for example, a preset screening ratio of retaining the first N×2 or the first N×3 lexical units as a candidate pool, and determining the set of selected candidate visual lexical units as the candidate lexical unit set, subsequently performing a greedy selection based on the maximum-minimum diversity problem on this candidate lexical unit set.
[0055] Step S24: In the first round, determine the first minimum semantic distance between each visual word in the candidate word set and other visual words from the distance matrix. Using a greedy selection strategy, determine the visual word corresponding to the maximum value among the first minimum semantic distances as the target visual word in this round, and move the target visual word to the selected word set, and jump to the next round.
[0056] Step S25: In non-first rounds, determine the second minimum semantic distance between each visual word in the candidate word set and each target visual word in the selected word set from the distance matrix. Using a greedy selection strategy, determine the visual word corresponding to the maximum value of each second minimum semantic distance as the target visual word in this round, and move the target visual word to the selected word set. Jump to the next round until the preset selection stop condition is met.
[0057] In this embodiment, moving the target visual word unit to the selected word unit set includes: if there are multiple target visual words units in the current round, determining the second minimum semantic distance of the target visual words unit as the diversity score of the target visual words unit; obtaining the attention score corresponding to each target visual word unit and the semantic relevance score between each target visual word unit and the input text corresponding to the input image; and normalizing the attention score, semantic relevance score, and diversity score of each target visual word unit to obtain each target visual word unit. The normalized attention score, normalized semantic relevance score, and normalized diversity score of each target visual word are determined. The weights corresponding to these scores are then determined, and a weighted sum is performed to obtain a comprehensive score for each target visual word. These comprehensive scores are then sorted in descending order, and the target visual word with the highest comprehensive score is selected and moved to the selected word set.
[0058] If there are multiple target visual words in this round, the second minimum semantic distance of the target visual words is determined as the diversity score of the target visual words; the attention score and semantic relevance score between each target visual word and the input text are obtained; the attention score, semantic relevance score and diversity score of each target visual word are normalized respectively to obtain the normalized attention score, normalized semantic relevance score and normalized diversity score of each target visual word; the weights corresponding to the normalized attention score, normalized semantic relevance score and normalized diversity score are determined, and the three normalized scores are weighted and summed to obtain the comprehensive score of each target visual word; the comprehensive scores of all target visual words are sorted in descending order, and the target visual word with the largest comprehensive score is selected from all target visual words and moved to the selected word set.
[0059] For example, suppose there are three target visual nouns A, B, and C in this round, with diversity scores of 0.8, 0.7, and 0.9, corresponding attention scores of 0.9, 0.6, and 0.7, and semantic relevance scores with the input text of 0.5, 0.8, and 0.6, respectively. Normalize these scores for each dimension. Assume that after normalization, the scores for A are 0.8, 1.0, and 0.5; for B, 0.7, 0.6, and 1.0; and for C, 1.0, 0.7, and 0.6. The weights for the three dimensions are 0.3, 0.4, and 0.3, respectively. Therefore, the overall score for A is 0.8×0.3+1.0×0.4+0.5×0.3=0.79, for B it is 0.7×0.3+0.6×0.4+1.0×0.3=0.75, and for C it is 1.0×0.3+0.7×0.4+0.6×0.3=0.76. The three overall scores are sorted from highest to lowest as A (0.79), C (0.76), and B (0.75). The target visual word element A with the highest overall score is then moved to the selected word element set.
[0060] Step S26: Discard the candidate word set and determine the target visual word in the selected word set as the pruned visual word.
[0061] Step S27: Concatenate the pruned visual lexical units with the text lexical units to obtain the input lexical unit sequence.
[0062] Step S28: Input the input word sequence into the large language model so that the large language model generates descriptive text corresponding to the input image.
[0063] As can be seen, this invention is a plug-and-play pure inference stage optimization method that requires no additional calibration datasets, model fine-tuning, or architectural modifications, thus lowering the barrier to entry and deployment costs. By maximizing the diversity of selected lexical units, this invention can maintain most of the original model's performance even under extreme pruning rates, and it does not depend on specific large language model architectures or visual encoders. Therefore, it can be applied not only to target visual question answering models but also to large-scale image and video multimodal models. By reducing the number of input lexical units, it reduces the inference latency and GPU memory consumption of large multimodal models, making it suitable for real-time applications and resource-constrained environments.
[0064] A greedy selection algorithm based on the maximum-minimum diversity problem for visual lexical pruning iteratively selects lexicals that are furthest from the selected set, ensuring optimal global diversity in the final subset. The maximum-minimum diversity problem solving algorithm is treated as a preprocessing step, directly manipulating visual lexicals before model inference. This process does not involve any model parameter updates, fine-tuning, or dependence on additional calibration data. Cosine distance is used to evaluate the differences between visual lexicals, thus combining with diversity pruning methods. This approach is compatible with key-value caching inference optimization techniques for large language models and enables efficient parallel processing on hardware accelerators (such as GPUs and FPGAs) by pre-compiling the distance matrix.
[0065] Figure 5 This is a schematic diagram of an image description text creation device based on visual lexical pruning, provided in an embodiment of the present invention. The device is applied to a target visual question answering model, which includes a visual encoder and a large language model. The device includes: The lexical acquisition module 11 is used to acquire multiple visual lexical units of the input image and text lexical units corresponding to the input image using a visual encoder; The set determination module 12 is used to construct a distance matrix based on the semantic distance between each visual word element, determine the set of visual word elements as a candidate word element set, and initialize the selected word element set as an empty set; The first selection module 13 is used to determine, in the first round, the first minimum semantic distance between each visual word in the candidate word set and other visual words from the distance matrix, determine the visual word corresponding to the maximum value of each of the first minimum semantic distances as the target visual word in this round through a greedy selection strategy, and move the target visual word to the selected word set and jump to the next round. The second selection module 14 is used to determine, in non-first rounds, the second minimum semantic distance between each visual word in the candidate word set and each target visual word in the selected word set from the distance matrix, and to determine the visual word corresponding to the maximum value of each second minimum semantic distance as the target visual word in the current round through a greedy selection strategy, and to move the target visual word to the selected word set, and jump to the next round, until the preset selection stop condition is met; The pruning module 15 is used to discard the candidate word set and determine the target visual word in the selected word set as the pruned visual word; The lexical concatenation module 16 is used to concatenate the pruned visual lexical units with the text lexical units to obtain the input lexical unit sequence; The text generation module 17 is used to input the input word sequence into the large language model so that the large language model generates descriptive text corresponding to the input image.
[0066] Furthermore, embodiments of the present invention also disclose an electronic device, Figure 6 This is a structural diagram of an electronic device according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of the invention. Specifically, the electronic device may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the image description text creation method based on visual lexical pruning disclosed in any of the foregoing embodiments. Furthermore, the electronic device in this embodiment may specifically be a computer.
[0067] In this embodiment, the power supply 23 is used to provide operating voltage for various hardware devices on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this invention, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0068] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0069] The operating system 221 is used to manage and control the various hardware devices on the electronic device and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the image description text creation method based on visual lexical pruning disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.
[0070] Furthermore, the present invention also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned image description text creation method based on visual lexical pruning. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.
[0071] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0072] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0073] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0074] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0075] The technical solution provided by the present invention has been described in detail above. Specific examples have been used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core idea of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the idea of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A method for creating image-descriptive text based on visual lexical pruning, characterized in that, Applied to a target visual question answering model, the target visual question answering model including a visual encoder and a large language model, wherein the method includes: A visual encoder is used to obtain multiple visual words of the input image and text words corresponding to the input image; A distance matrix is constructed based on the semantic distance between each visual word, and the set of visual words is determined as the candidate word set. The selected word set is initialized as an empty set. In the first round, the first minimum semantic distance between each visual word in the candidate word set and other visual words is determined from the distance matrix. The visual word corresponding to the maximum value of each of the first minimum semantic distances is determined as the target visual word in this round by a greedy selection strategy. The target visual word is then moved to the selected word set and the process jumps to the next round. In non-first rounds, the second minimum semantic distance between each visual word in the candidate word set and each target visual word in the selected word set is determined from the distance matrix. The visual word corresponding to the maximum value of each second minimum semantic distance is determined as the target visual word in this round through a greedy selection strategy. The target visual word is then moved to the selected word set and the process jumps to the next round until the preset selection stop condition is met. The candidate word set is discarded, and the target visual word in the selected word set is determined as the pruned visual word. The pruned visual lexical units are concatenated with the text lexical units to obtain the input lexical unit sequence; The input word sequence is fed into a large language model so that the large language model generates descriptive text corresponding to the input image.
2. The image description text creation method based on visual lexical pruning according to claim 1, characterized in that, The step of acquiring multiple visual terms and corresponding text terms of an input image using a visual encoder includes: The input image is divided into multiple non-overlapping image blocks, and each image block is flattened into a one-dimensional vector. Each of the one-dimensional vectors is input into a visual encoder, and the visual encoder outputs the feature vector corresponding to each of the image blocks. Each of the aforementioned feature vectors is input into the projection layer, and the projection layer maps each of the aforementioned feature vectors to the same dimension as the input embedding space of the large language model, thereby obtaining multiple visual lexical units; The input text corresponding to the input image is embedded through the text embedding layer of the large language model to obtain text lexical units.
3. The image description text creation method based on visual lexical pruning according to claim 1, characterized in that, Before determining the first minimum semantic distance between each visual word in the candidate word set and other visual words from the distance matrix, the method further includes: Obtain the total number of visual words corresponding to the input image and the preset pruning rate; The theoretical number of visual words after pruning is determined by multiplying the total number by the preset pruning rate. The theoretical number is rounded down to obtain the first pruning threshold. A preset selection stop condition is established based on the first pruning quantity threshold; Accordingly, the jump to the next round includes: If the number of target visual words in the selected word set is less than the first pruning threshold in the preset selection stop condition, then proceed to the next round.
4. The image description text creation method based on visual lexical pruning according to claim 1, characterized in that, Before determining the first minimum semantic distance between each visual word in the candidate word set and other visual words from the distance matrix, the method further includes: Obtain the total number of visual lexical units corresponding to the input image; The complexity of the input image is evaluated to obtain a complexity index of the input image; Determine the target pruning rate that matches the complexity of the input image based on the complexity index; The second pruning number threshold is determined by multiplying the total number by the target pruning rate; A preset selection stop condition is established based on the second pruning quantity threshold; Accordingly, the jump to the next round includes: If the number of target visual words in the selected word set is less than the second pruning threshold in the preset selection stop condition, then proceed to the next round.
5. The image description text creation method based on visual lexical pruning according to any one of claims 1 to 4, characterized in that, The step of constructing a distance matrix based on the semantic distance between each visual word includes: Each visual word is normalized using the L2 norm to obtain a normalized vector corresponding to each visual word. A normalized matrix is constructed based on each of the normalized vectors, and a similarity matrix is constructed based on the product of the normalized matrix and its transpose. Each element in the similarity matrix is subtracted from a preset constant to convert the similarity into semantic distance, and a distance matrix is constructed based on the semantic distance.
6. The image description text creation method based on visual lexical pruning according to claim 1, characterized in that, The step of determining the set of visual lexical units as the candidate lexical unit set includes: Obtain the attention score corresponding to each visual word, and sort the visual words in descending order of the attention scores to obtain a sorted visual word sequence. Candidate visual words are selected from the head of the sorted visual word sequence according to a preset screening ratio, and the set of selected candidate visual words is determined as the candidate word set.
7. The image description text creation method based on visual lexical pruning according to claim 1, characterized in that, Moving the target visual word to the selected word set includes: If there are multiple target visual words in this round, the second minimum semantic distance of the target visual words is determined as the diversity score of the target visual words. Obtain the attention score corresponding to each target visual word and the semantic relevance score between each target visual word and the input text corresponding to the input image; The attention score, semantic relevance score, and diversity score of each target visual word are normalized to obtain the normalized attention score, normalized semantic relevance score, and normalized diversity score of each target visual word. The weights corresponding to the normalized attention score, the normalized semantic relevance score, and the normalized diversity score are determined, and the normalized attention score, the normalized semantic relevance score, and the normalized diversity score are weighted and summed to obtain the comprehensive score of each target visual word. The comprehensive scores are sorted in descending order, and the target visual word with the largest comprehensive score is selected from the target visual word and moved to the selected word set.
8. An image-descriptive text creation device based on visual lexical pruning, characterized in that, An apparatus for use in a target visual question answering model, the target visual question answering model comprising a visual encoder and a large language model, wherein the apparatus comprises: The lexical acquisition module is used to acquire multiple visual lexical units of the input image and text lexical units corresponding to the input image using a visual encoder; The set determination module is used to construct a distance matrix based on the semantic distance between each visual word element, determine the set of visual word elements as a candidate word element set, and initialize the selected word element set as an empty set; The first selection module is used to determine, in the first round, the first minimum semantic distance between each visual word in the candidate word set and other visual words from the distance matrix, determine the visual word corresponding to the maximum value of each of the first minimum semantic distances as the target visual word in this round through a greedy selection strategy, move the target visual word to the selected word set, and jump to the next round. The second selection module is used to determine, in non-first rounds, the second minimum semantic distance between each visual word in the candidate word set and each target visual word in the selected word set from the distance matrix, and to determine the visual word corresponding to the maximum value of each second minimum semantic distance as the target visual word in the current round through a greedy selection strategy, and to move the target visual word to the selected word set, and jump to the next round, until the preset selection stop condition is met; The pruning module is used to discard the candidate word set and determine the target visual word in the selected word set as the pruned visual word; The lexical concatenation module is used to concatenate the pruned visual lexical units with the text lexical units to obtain the input lexical unit sequence; The text generation module is used to input the input word sequence into a large language model so that the large language model generates descriptive text corresponding to the input image.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the image description text creation method based on visual lexical pruning as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image description text creation method based on visual lexical pruning as described in any one of claims 1 to 7.
