Processing method of model input sequence, processing device of model input sequence, and medium

Abstract

This application discloses a method, device, and medium for processing model input sequences, relating to the field of Internet technology. The method includes: extracting feature vectors of the modality to be input; calculating the importance assessment result of each text token in the feature vectors; obtaining the resource utilization rate of the current system; selecting a compression strategy based on the system resource utilization rate; compressing each text token after it has been ranked according to the importance assessment results based on the compression strategy to obtain the target text token; and outputting the target text token to a preset processing model. Based on this, by extracting semantic features of modal data, a basis for token importance assessment is provided. After calculating the importance score, a suitable compression strategy is used to compress the content ranked based on the importance assessment, retaining high-value tokens while filtering out redundant low-value tokens.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to methods, devices and media for processing model input sequences. Background Technology

[0002] The core architecture of current large language models is the Transformer architecture, whose computational complexity increases quadratically with the number of tokens (the smallest unit of data processed by the model). This characteristic makes the model suffer from processing efficiency bottlenecks in scenarios with long sequence inputs.

[0003] To alleviate this problem, lightweight scenarios typically employ token compression strategies, reducing computational overhead and improving processing efficiency by decreasing the total number of tokens input to the model. However, existing compression strategies suffer from insufficient value discrimination accuracy, easily leading to the accidental deletion of high-value tokens and redundant retention of low-value tokens. Even with compression, the efficiency of long-sequence inference in the model remains difficult to substantially improve.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a method, device and medium for processing model input sequences, which aims to solve the technical problem that high-value tokens are mistakenly deleted or low-value tokens are not effectively filtered, making it difficult to improve the efficiency of long sequence inference in the model.

[0006] To achieve the above objectives, this application proposes a method for processing model input sequences, the method comprising: Extract the feature vector of the input modality and calculate the importance evaluation result of each text token in the feature vector; Obtain the current system resource utilization rate, and select a compression strategy based on the system resource utilization rate; Based on the compression strategy, each text token after being sorted by the importance evaluation results is compressed to obtain the target text token; The target text token is output to the preset processing model.

[0007] In one embodiment, after the step of compressing each text token sorted by importance assessment results based on the compression strategy to obtain the target text token, the method for processing the model input sequence further includes: The semantic association information of the target text Token is learned by a multilayer perceptron (MLP), and the semantic gaps of the target text Token are updated. Based on the target text token updated by semantic detection, a lightweight model is trained by comparing the learning loss function and the semantic reconstruction loss function to obtain the target text token with semantic consistency.

[0008] In one embodiment, the input modality includes a text modality, an image modality, and a speech modality. The step of extracting the feature vector of the input modality and calculating the importance evaluation result of each text token in the feature vector includes: Extract the text token embedding vector of the text modality, the visual feature vector of the image modality, and the audio feature vector of the audio modality from the input modality; Standardize the text token embedding vector, the visual feature vector, and the audio feature vector; Calculate the internal importance evaluation result of each text token in the text token embedding vector, and calculate the visual importance evaluation result between each text token and the visual feature vector and / or calculate the audio importance evaluation result between each text token and the audio feature vector; Based on the internal importance assessment results of each text token, and according to the visual importance assessment results and / or the audio importance assessment results, the importance assessment results of each text token are calculated.

[0009] In one embodiment, the steps of calculating the visual importance assessment result between each text token and the visual feature vector and / or calculating the audio importance assessment result between each text token and the audio feature vector include: Calculate the cosine similarity between the text token and the image embedding vector, and take the maximum value of the cosine similarity as the evaluation value corresponding to the visual importance evaluation result; and / or Calculate the keyword matching degree between each text token and the speech keywords in the audio feature vector, and the sentiment matching degree between each text token and the audio feature vector; The weighted fusion score between the keyword matching degree and the sentiment matching degree is used as the evaluation value of the audio importance assessment result.

[0010] In one embodiment, before the steps of extracting the feature vector of the input modality and calculating the importance evaluation result of each text token in the feature vector, the method for processing the model input sequence further includes: Obtain the text information after text writing based on the audio modality; Based on a preset algorithm, core prompt words are extracted from the text information, and the core prompt words are used as the text token of the text modality, and the text token is used as the input modality.

[0011] In one embodiment, the step of compressing each text token after it has been sorted by importance assessment results based on the compression strategy to obtain the target text token includes: Determine the current compression ratio corresponding to the compression strategy, wherein the current compression ratio is the sum of the adjustment parameter calculated by the system resource utilization and the base compression ratio; Calculate the number of text tokens to be retained based on the current compression ratio; The number of text tokens that are retained after being sorted in reverse order according to the importance assessment results is selected as the target text token.

[0012] In one embodiment, after the step of selecting the number of text tokens to be retained after being sorted in reverse order of the importance evaluation results as the target text token, the method for processing the model input sequence further includes: Identify the compressed text token sequence; Based on the multimodal guided verification algorithm, the target tokens in the compressed text token sequence are determined to have a correlation matching degree between the text tokens and other modalities that is greater than a preset matching degree. Set the target token to the target text token to retain the target token.

[0013] In one embodiment, the input modality is a text modality, and the step of extracting the feature vector of the input modality and calculating the importance evaluation result of each text token in the feature vector includes: Obtain the text token of the text modality, and extract the text token embedding vector of the text token based on the text encoder; The internal importance assessment result of each text token in the text token embedding vector is determined as the importance assessment result.

[0014] Furthermore, to achieve the above objectives, this application also proposes a processing device for a model input sequence, the X device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the processing method for the model input sequence as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the processing method for the model input sequence as described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: First, the feature vector of the input modality is extracted, and the importance assessment result of each text token in the feature vector is calculated. This achieves accurate quantitative differentiation of the semantic priority of each token, avoiding the accidental deletion of high-value tokens due to the lack of clear assessment criteria, and providing a judgment standard for the targeted screening of low-value tokens. Next, an appropriate compression strategy is selected based on the resource utilization of the current system, so that the compression operation can match different operating resource scenarios. Then, based on this strategy, the text tokens after importance ranking are compressed to obtain the target text token, so as to retain the core high-value tokens and filter redundant low-value tokens. Finally, the target text token is output to the preset processing model for processing. This makes the input sequence of the model streamline the invalid content while maintaining the integrity of the core semantics. It breaks through the efficiency bottleneck caused by the absence of high-value tokens or the redundancy of low-value tokens when the model is in long sequence inference, and allows the model to complete the inference process more efficiently under different resource conditions. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating the first embodiment of the method for processing the input sequence of the model in this application; Figure 2 A schematic diagram of the token compression process for processing the input sequence of the model in this application; Figure 3 A flowchart illustrating the second embodiment of the method for processing the input sequence of the model in this application; Figure 4 A schematic diagram of the token importance evaluation process for the model input sequence processing method provided in the second embodiment of this application; Figure 5 This is a schematic diagram of the token importance evaluation process obtained by combining various embodiments of the processing method for the input sequence of the model in this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the processing method of the model input sequence in the embodiments of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] The main solution of this application embodiment is: extracting the feature vector of the input modality and calculating the importance evaluation result of each text token in the feature vector; Obtain the current system resource utilization rate, and select a compression strategy based on the system resource utilization rate; Based on the compression strategy, each text token after being sorted by the importance evaluation results is compressed to obtain the target text token; The target text token is output to the preset processing model.

[0023] In this embodiment, for ease of description, the following description will focus on the processing device of the model input sequence.

[0024] The core architecture of current large language models is the Transformer architecture, whose computational complexity increases quadratically with the number of tokens. This characteristic makes the model suffer from processing efficiency bottlenecks in scenarios with long input sequences.

[0025] To alleviate this problem, token compression strategies are typically employed, which reduce computational overhead and improve processing efficiency by decreasing the total number of tokens input to the model. However, existing compression strategies suffer from insufficient accuracy in value discrimination, easily leading to the accidental deletion of high-value tokens and redundant retention of low-value tokens. Even with compression, the efficiency of long-sequence inference in the model remains difficult to improve substantially.

[0026] Based on this, this application provides a solution that first extracts the feature vector of the input modality and calculates the importance evaluation result of each text token in the feature vector, realizing the accurate quantitative differentiation of the semantic priority of each token. This avoids the situation where high-value tokens are mistakenly deleted due to the lack of clear evaluation criteria, and also provides a judgment standard for the targeted screening of low-value tokens. Next, an appropriate compression strategy is selected based on the resource utilization of the current system, so that the compression operation can match different operating resource scenarios. Then, based on this strategy, the text tokens after importance ranking are compressed to obtain the target text token, so as to retain the core high-value tokens and filter redundant low-value tokens. Finally, the target text token is output to the preset processing model, so that the input sequence of the model is both streamlined with invalid content and maintains the integrity of the core semantics. This breaks through the efficiency bottleneck caused by the absence of high-value tokens or the redundancy of low-value tokens when the model is performing long sequence inference, and allows the model to complete the inference process more efficiently under different resource conditions.

[0027] It should be noted that the processing device for the model input sequence in this embodiment can be a computing service device with data processing, network communication and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or a wearable electronic device that can achieve the above functions.

[0028] In this embodiment, the processing device for the model input sequence includes a feature extraction module for extracting semantic features from the input multimodal data such as text, vision, and audio; a token importance assessment module for calculating the importance score of each text token based on the multimodal features, providing a quantitative basis for compression decisions; a dynamic compression strategy module for adaptively selecting a compression method based on the token importance score and the current resource scenario, realizing on-demand compression; and a reconstruction and fine-tuning module for semantic completion and consistency fine-tuning of the compressed token sequence to solve the semantic breakage problem that may occur during the compression process.

[0029] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0030] This application provides a method for processing model input sequences, referring to... Figure 1 , Figure 1 A flowchart illustrating the first embodiment of the method for processing the input sequence of the model in this application.

[0031] In this embodiment, the method for processing the model input sequence includes steps S10 to S40: Step S10: Extract the feature vector of the input modality and calculate the importance evaluation result of each text token in the feature vector.

[0032] In this embodiment, the input modality can be a text modality, or a combination of a text modality and other modalities, such as an image modality or an audio modality. The feature vector is a high-dimensional numerical vector representing the semantic information of the data, obtained by encoding the original modality data through an encoder. The importance evaluation result is a score or level that quantifies the semantic priority of each text token. Specifically, in speech-derived scenarios, text tokens are discrete text units extracted from prompts derived from speech-to-text transcription, or text units extracted from the complete text after speech-to-text transcription. In non-speech-derived text input scenarios, text tokens are text units corresponding to the complete text input from other devices. The following uses a speech-derived scenario as an example to illustrate the speech-derived scenario in a lightweight context.

[0033] As an optional implementation, when the input modality only contains a text modality, the text modality is the complete text content obtained after the currently received audio is transcribed into text. After receiving the speech processing task, the received audio data can be transcribed into text, and the text token embedding vectors of all words in the transcribed complete text can be extracted. For example, in the lightweight application scenario of smartwatch cycling navigation, the user sends a navigation command to the smartwatch via voice, and the audio content is "Help me plan a cycling route from home to the company, prioritizing sections with cycling lanes and avoiding construction areas"; after the smartwatch's audio receiving module captures the voice signal, it transcribes it using the built-in ASR automatic speech recognition algorithm to obtain the complete text content "Help me plan a cycling route from home to the company, prioritizing sections with cycling lanes and avoiding construction areas". Subsequently, word segmentation processing is performed on the complete text, and the text token sequence of all words is extracted according to the principle of semantic integrity, specifically: [help, me, plan, from home, to company, cycling route, priority, walk, have, cycling lane, of, section, avoid, construction area].

[0034] Next, the encoder performs semantic encoding on each independent text token in the token sequence, outputting a high-dimensional semantic feature vector corresponding to each token. It's understandable that the text token embedding vector is merely a quantified carrier of the text token's semantics; it objectively carries the token's semantic information but cannot directly reflect its value priority for model inference. Therefore, after obtaining the semantic feature vector, the token semantics implicit in the vector can be converted into quantifiable scores in the token importance evaluation module, i.e., calculating its importance evaluation result. Since there are no other dimensional parameters when calculating the importance evaluation result, the importance evaluation is based solely on the text's internal features. Therefore, after obtaining the text tokens of the text modality and extracting the text token embedding vectors based on the text encoder, the internal importance evaluation result of each text token in the text token embedding vector can be directly determined as the overall importance evaluation result.

[0035] For example, the calculation process of the internal importance assessment result Score_text is as follows: Score_text=ω1·(1-Entropy(Attention_weights))+ω2·Norm(Gradient) Where Attention_weights is the attention weight vector of the target large language model for this token; Entropy is the entropy calculation function, Entropy(Attention_weights)=-Σ(wi·log2(wi)), where wi is the attention weight; Gradient is the gradient vector of the token embedding vector to the model output loss; Norm is the L2 normalization function, which converts the gradient vector into a value in the interval [0,1], and ω1 and ω2 are weight coefficients, with the default ω1=0.7 and ω2=0.3, which can be adaptively adjusted through training.

[0036] As an alternative implementation, when the input modality includes text and other modalities (the other modalities at least include speech), the waveform signal of the audio modality can first be transcribed into speech. Core prompt words can then be extracted from the transcribed complete text and split into a sequence of text tokens. Simultaneously, these tokens are converted into feature vectors using encoders corresponding to the other modalities, and the dimensions of the multi-dimensional feature vectors are standardized, for example, to obtain a 768-dimensional feature vector corresponding to all modalities.

[0037] After obtaining the feature vectors, it is necessary to calculate the internal importance evaluation result of each text token in the text token embedding vector, as well as the association importance evaluation result between each text token and other modal feature vectors. Finally, the final importance evaluation result is calculated through weighted fusion.

[0038] In non-voice-derived text input scenarios, the method for obtaining the importance assessment results is the same as the two methods mentioned above, and will not be elaborated upon in this application.

[0039] This embodiment evaluates the importance of multimodal features using tokens, providing a precise priority basis for subsequent compression.

[0040] Step S20: Obtain the current system resource utilization rate and select a compression strategy based on the system resource utilization rate.

[0041] In this embodiment, system resource utilization refers to the proportion of CPU, GPU, or memory resources currently occupied by the device, which can be read through the device's own program or hardware. The compression strategy is a token filtering / merging rule determined based on resource conditions. Different compression ratios can be selected under different resource scenarios to adapt to the resource carrying capacity of different devices. In high-resource scenarios, a high retention rate strategy ensures the semantic integrity of the input sequence. In medium-to-low-resource lightweight scenarios, a reasonable compression ratio strategy reduces redundant tokens to lower computational overhead, while avoiding the defects of erroneous deletion of high-value tokens and redundant retention of low-value tokens. This achieves dual optimization of the model's long sequence inference efficiency and semantic accuracy.

[0042] Specifically, the processing device for the model input sequence can collect CPU utilization and memory usage data in real time through its built-in hardware monitoring tools. Simultaneously, it selects the appropriate strategy based on the resource utilization and corresponding resource usage scenarios, such as high-resource scenarios and low-resource scenarios. For example, if CPU utilization is <50% and memory usage is <40%, it is determined to be a high-resource scenario, corresponding to a high retention rate strategy; if CPU utilization is 50%-80% and memory usage is 40%-70%, it is determined to be a medium-resource scenario, corresponding to a balanced strategy; if CPU utilization is >80% or memory usage is >70%, it is determined to be a low-resource scenario, corresponding to a high compression rate strategy. Finally, the corresponding compression strategy is selected based on the actual resource utilization. For instance, if the server hardware monitoring shows CPU utilization of 25% and memory usage of 35%, it is determined to be a high-resource scenario, and a high retention rate compression strategy is selected.

[0043] This embodiment selects the corresponding compression strategy based on resource utilization, thereby balancing semantic integrity and resource consumption in different resource scenarios.

[0044] Step S30: Based on the compression strategy, each text token after being sorted by importance assessment results is compressed to obtain the target text token.

[0045] In this embodiment, compression refers to the operation of filtering or merging the list of text tokens, while the target text token is the core token retained after compression.

[0046] As an optional implementation, the current compression rate corresponding to the compression strategy can be determined first. This compression rate is obtained by summing the adjustment parameters calculated from the system resource utilization rate and the base compression rate. Then, the number of text tokens to be retained is calculated based on the current compression rate. Finally, the text tokens with the highest retention rate, sorted in reverse order of importance assessment results, are selected as the target text tokens. The calculation process of the current compression rate, Compression_Rate, is as follows: Compression_Rate=Base_Rate+δ·(1-Resource_Availability).

[0047] Here, Base_Rate is the base compression rate, usually 10%, which can be set according to actual needs, and δ is the adjustment coefficient with a default of 0.4; Resource_Availability is the system resource utilization rate, which can be calculated by weighting CPU and memory, and its range is [0, 1]. After calculating Compression_Rate, the number of tokens to be retained can be deduced in reverse, that is, the number of tokens to be retained = the original number of tokens × (1 - Compression_Rate)).

[0048] Specifically, all text tokens can be sorted from highest to lowest importance based on their evaluation results, and then a corresponding number of tokens can be retained based on the compression ratio corresponding to the compression strategy. Please refer to [link / reference]. Figure 2 After inputting the importance score and resource status (GPU or CPU utilization and memory utilization), the system determines the resource scenario. In a high-resource scenario, the compression strategy is a high retention rate strategy, where 80% of the tokens are retained, thus retaining the top-80% of important tokens (the first 80% of the sorted tokens) to prioritize semantic integrity. If a balanced strategy is adopted, where 50%-60% of the tokens are retained, the system truncates the top 50%-60% of the tokens. If a high compression rate strategy is adopted, the system retains the top 30%-40% of important tokens. Additionally, the system can greedily merge the most representative tokens based on the Rouge score to maximize inference speed.

[0049] Furthermore, after compression, when the input modality includes multiple modalities, a multimodal guided verification method can be used to extract tokens from the compressed text token sequence whose association matching degree with other modalities is greater than a preset matching degree. These tokens are then retained as target text tokens, ensuring that core semantic tokens are not mistakenly deleted. Specifically, the multimodal guided verification calculates the semantic matching degree between the text tokens and features of other modalities such as images / audio. Tokens with a matching degree that meets the standard are forcibly retained, preventing the mistaken deletion of such cross-modal core semantic tokens. This overcomes the limitations of relying solely on single-text importance scores for selection, ensuring that the core semantics of the compressed sequence are not lost. Finally, if the target compression rate is met, the compressed token sequence is output; otherwise, selection continues based on the reverse sequence.

[0050] As an alternative implementation, compression can also be performed based on sorting and merging. Specifically, the tokens are first sorted by importance and divided into a high-importance layer (top 50%), a medium-importance layer (middle 20%), and a low-importance layer (bottom 30%). Then, if the compression strategy is a balanced strategy, the high-importance layer tokens are retained, and semantically related tokens in the medium-importance layer are merged, such as merging "intersection ahead" and "cycling lane" into "cycling lane at intersection ahead". Finally, the low-importance layer tokens are filtered to obtain the target text token.

[0051] Step S40: Output the target text token to the preset processing model.

[0052] In this embodiment, after obtaining the target text token, it is output to the corresponding processing model so that the model can perform the relevant processing tasks.

[0053] This embodiment provides a method for processing model input sequences. By evaluating the importance of text tokens, it avoids the accidental deletion of high-value tokens and the redundancy of low-value tokens. At the same time, it selects an appropriate resource compression strategy based on the current resource utilization rate to achieve accurate compression of text tokens. It balances semantic integrity and resource consumption under different resource scenarios. After compression, a multimodal guided verification algorithm is used to retain tokens with strong correlations between modalities to avoid accidental deletion of strongly correlated tokens. Finally, an output sequence for direct inference of large models is obtained, thereby effectively improving the efficiency of long sequence inference of the model while maintaining the semantic accuracy of the inference results.

[0054] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the first embodiment can be referred to the above description, and will not be repeated hereafter. In this regard, the input modality includes text modality, image modality, and voice modality; please refer to... Figure 3Step S10 also includes steps S11 to S14: Step S11: Extract the text token embedding vector of the text mode, the visual feature vector of the image mode, and the audio feature vector of the audio mode from the input modes to be processed.

[0055] In this embodiment, a text token embedding vector can be extracted by a text encoder. The encoder uses the embedding layer of a pre-trained language model such as BERT, RoBERTa, or a large language model to convert the input text sequence into a fixed-dimensional text token embedding vector, thereby capturing the semantic relationships within the text.

[0056] Meanwhile, the visual encoder uses a pre-trained visual model to divide the image modality into regions and extract the visual feature vectors of each region, focusing on capturing salient regions in the image such as the main target and key scene information; at the same time, the speech encoder uses a pre-trained speech model to convert the input speech signal into a frame-level feature vector and extract core information such as keywords and emotional tone such as excitement, calmness, and anger in the speech to obtain audio feature vectors.

[0057] In this multimodal scenario, the text token is a discrete text unit extracted from the prompt words derived from speech-to-text transcription. Therefore, before step S11, the text information after the audio modality is transcribed can be obtained first. Then, core prompt words are extracted from the text information based on a preset algorithm, and these core prompt words are used as the text token of the text modality, and the text token is used as the input modality. For example, a user issues a command to the smart glasses via voice: "I want to go to the company from the entrance of my community. Please plan a cycling route for me, preferably one with a dedicated cycling lane, and avoid the construction area that is temporarily closed today." First, the audio modality is transcribed to obtain the corresponding text information: "I want to go to the company from the entrance of my community. Please plan a cycling route for me, preferably one with a dedicated cycling lane, and avoid the construction area that is temporarily closed today." Then, a preset TF-IDF keyword extraction algorithm is used to extract core prompt words from the text information, ultimately obtaining core keywords such as "from the entrance of the community," "to the company," "cycling route," "dedicated cycling lane," "avoid," and "temporarily closed construction area," which are directly used as the text token of the text modality.

[0058] Step S12: Standardize the text token embedding vector, visual feature vector, and audio feature vector.

[0059] In this embodiment, due to the differences in the numerical range and distribution of feature vectors from different modalities, direct cross-modal semantic matching is not possible. Therefore, it is necessary to standardize the output feature vectors of each sub-encoder using features such as LayerNorm to unify the dimensions before inputting these data into the Token Importance Assessment module for processing. For example, mapping all vectors to the same numerical space and standardizing them to 768 dimensions can eliminate numerical biases between modalities and provide a fair computational benchmark for subsequent cross-modal importance assessment.

[0060] Step S13: Calculate the internal importance evaluation result of each text token in the text token embedding vector, and calculate the visual importance evaluation result between each text token and the visual feature vector and / or calculate the audio importance evaluation result between each text token and the audio feature vector.

[0061] In this embodiment, the internal semantic value of the text token can be calculated first, and then the cross-modal importance assessment result can be calculated. The matching degree between each text token embedding vector and the visual feature vector is calculated by the cosine similarity algorithm to obtain the visual importance assessment result. At the same time, the overlap degree between the text token embedding vector and the keyword features in the audio feature vector is calculated by the keyword feature matching algorithm to obtain the audio importance assessment result.

[0062] Specifically, the calculation method for the internal importance assessment result of each text token is as shown in the first embodiment, and will not be repeated here.

[0063] When calculating the visual importance assessment result between each text token and visual feature vector, the cosine similarity between the text token and the image embedding vector is typically calculated, and the maximum value of the cosine similarity is taken as the assessment value corresponding to the visual importance assessment result. The calculation process of the visual importance assessment result, Score_image, is as follows: Score_image=Max(CosineSimilarity(Token_emb, Image_region_emb)).

[0064] Wherein, Token_emb is the embedding vector of the text token, Image_region_emb is the embedding vector of each image region output by the visual encoder, CosineSimilarity is the cosine similarity calculation function, and finally the maximum similarity between the token and all image regions is taken as the visual matching score.

[0065] When calculating the audio importance assessment result between each text token and the audio feature vector, the keyword matching degree of each text token and the sentiment matching degree of each text token and the audio feature vector can be calculated. Finally, the weighted fusion score between the keyword matching degree and the sentiment matching degree is used as the evaluation value of the audio importance assessment result. The calculation process of the evaluation value core_audio of the audio importance assessment result is as follows: core_audio=ω_a·Keyword_Match+ω_e·Emotion_Match.

[0066] Here, Keyword_Match represents the matching degree between the text token and the speech keyword. If they match, it is 1; if they do not match, it is 0; and if they partially match, it is 0.5. Emotion_Match represents the matching degree between the emotional tendency of the text token and the emotional state of the speech. For example, if the text is "happy" and the speech is excited, the matching degree is 1. ω_a and ω_e are weight coefficients, with the default values ​​of ω_a=0.6 and ω_e=0.4.

[0067] Step S14: Based on the internal importance assessment results of each text token, calculate the importance assessment results of each text token according to the visual importance assessment results and / or audio importance assessment results.

[0068] In this embodiment, weighting coefficients can be set according to the actual application scenario to weight and fuse the internal importance evaluation results, visual importance evaluation results, and audio importance evaluation results of the text tokens to obtain the comprehensive importance evaluation result of each text token. In the lightweight scenario, the weight of cross-modal importance can be appropriately increased to prioritize the retention of core tokens that are strongly associated with image and audio semantics. In the high-resource scenario, the weight of internal importance can be increased to maximize the preservation of the integrity of text semantics.

[0069] Specifically, the evaluation value of the importance assessment result is Score = α·Score_text + β·Score_image + γ·Score_audio, where α, β and γ are different weighting coefficients, α+β+γ=1, and the weighting coefficient is 0 when there is no corresponding modality.

[0070] For example, to help understand the implementation flow of the model input sequence processing method obtained by combining this embodiment with the first embodiment described above, please refer to... Figure 4 , Figure 4 A schematic diagram of the token importance evaluation process for processing model input sequences is provided, specifically: The process begins by inputting multimodal features, specifically text token embeddings, visual region features, and audio frame features. If the current scene lacks a visual or audio modality, the calculations for that modality are skipped. Two operations are then performed simultaneously: calculating the intra-text importance score (Score_text) using a combination of attention entropy and gradient normalization; and calculating cross-modal matching scores. The visual matching score (Score_image) is obtained by calculating the cosine similarity between the text token and the visual region features, while the audio matching score (Score_audio) is determined by combining the results of keyword matching and sentiment matching.

[0071] Finally, the text importance score, visual matching score, and audio matching score are weighted and fused according to preset weighting coefficients (α, β, γ) to obtain the comprehensive importance score calculated by the formula: Score = α·Score_text + β·Score_image + γ·Score_audio. The comprehensive importance score for all text tokens is then output.

[0072] This embodiment provides a method for processing model input sequences. Through feature extraction, standardization, multi-dimensional evaluation, and fusion scoring, it transforms raw data from different modalities such as text, images, and audio into high-dimensional feature vectors that the model can compute. This fully preserves the core semantic information of each modality, providing a quantitative carrier for subsequent importance assessment. Then, the features of each modality are mapped to the same semantic space, ensuring fairness and accuracy in cross-modal evaluation and avoiding interference from numerical differences in single-modal features in determining token value. Based on the multi-dimensional importance assessment method, the accuracy of identifying high-value tokens is improved, resulting in a more comprehensive token importance ranking and providing a precise basis for subsequent resource-adaptive compression strategies.

[0073] Based on any embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, after step S30, the method for processing the model input sequence further includes steps S50~S60: Step S50: Learn the semantic association information of the target text Token through a multilayer perceptron (MLP) and update the semantic gaps of the target text Token.

[0074] In this embodiment, the compressed target text token may have semantic breaks. Therefore, a reconstruction and fine-tuning module is needed to perform semantic completion and consistency fine-tuning on the compressed token sequence. The reconstruction and fine-tuning module adopts a lightweight network architecture (such as a 2-layer MLP + LayerNorm), with the input being the compressed token embedding sequence and the output being a semantically coherent standardized embedding sequence.

[0075] Specifically, the target text token sequence is used as the input layer data of the MLP. Then, the MLP performs non-linear transformation on the input vector through 2-3 hidden layers, calculates the pairwise association weights of the token embedding vectors one by one, accurately captures the logical relationship between text tokens, quantifies the semantic dependency strength between tokens, and finally fine-tunes and updates the embedding vector of each target token based on the learned semantic association weights, supplements its semantic association features with the overall sequence, and ultimately fills the semantic gaps caused by compression, making the semantic connection of the target token sequence more coherent.

[0076] For example, the compressed token sequence is [cycling route, dedicated cycling lane, avoid construction area]. Originally, there was a semantic gap between "cycling route" and "avoid construction area". After the MLP learns the association logic of "cycling route planning needs to avoid construction area", it fine-tunes the embedding vectors of the two, reduces the semantic distance, and fills this gap.

[0077] Step S60: Based on the target text token updated by semantic detection, the lightweight model is trained by comparing the learning loss function and the semantic reconstruction loss function to obtain the semantically consistent target text token.

[0078] In this embodiment, the contrastive learning loss function is used to bring semantically similar token vectors closer together and semantically unrelated ones further apart. The semantic reconstruction loss function measures the ability of the optimized token to reconstruct the core semantics of the original complete text. The lightweight model is an independent model that requires training. Semantic consistency refers to the high degree of matching between the optimized target token and the core semantics of the original complete text.

[0079] Specifically, the updated target token embedding vector is used as the sample to be optimized, and the original complete text (i.e., the uncompressed token embedding vector) is used as the baseline sample. Simultaneously, negative samples are constructed. When calculating the contrastive learning loss, the sample to be optimized, the baseline sample, and the negative sample are input into the InfoNCE loss function. By calculating cosine similarity, the similarity between the sample to be optimized and the baseline sample is maximized, and the similarity with the negative sample is minimized, thus avoiding semantic shift of the target token. Next, the semantic reconstruction loss is calculated. The sample to be optimized is input into the decoder of the lightweight model to attempt to reconstruct the core semantic content of the original complete text, and the mean squared error between the reconstruction result and the original core semantics is calculated. Then, the two loss functions are fused to obtain the total loss function. Based on the total loss function, the AdamW gradient descent algorithm is used to backpropagate and update the parameters of the lightweight model. Iterative training continues until the total loss converges, ultimately ensuring that the target text token output by the trained lightweight model retains the concise characteristics of compression while maintaining a high degree of consistency with the core semantics of the original complete text.

[0080] For example, the original complete text is "Help me plan a cycling route from my neighborhood to my company, take the dedicated cycling lane, and avoid construction areas." After the compressed target token is updated in step S50, it is made closer to the original core semantic vector through comparative learning loss, and it can restore the core semantics of "planning a cycling route, taking the cycling lane, and avoiding construction areas" through semantic reconstruction loss. The target token obtained after training is both concise and semantically complete.

[0081] For example, to help understand the implementation flow of the model input sequence processing method obtained by combining the above embodiments, please refer to... Figure 5 , Figure 5 A flowchart illustrating a method for processing model input sequences is provided, specifically: The device's input layer receives three modalities of data: text, image, and audio. Text data is converted into token sequences, image data is input as pixel matrices, and audio data is input as waveforms. Then, in the multimodal feature extraction layer, text encoders, visual encoders, and speech encoders extract text token embeddings, image region features, and audio spectral features, respectively. These three types of features are then standardized into a 768-dimensional feature vector to eliminate numerical biases between different modalities, placing them within the same semantic space.

[0082] Next, the process moves to the Token Importance Evaluation Layer. Here, the semantic information of each text token is combined with its cross-modal correlation with image region features and audio spectral features to calculate a comprehensive importance score for each text token. After obtaining the score, the token enters the Dynamic Compression Layer. Based on the device resource scenario, the top-K proportion of high-importance tokens are retained, while multimodal guided verification ensures that core semantic tokens are not omitted, generating a compressed token sequence. Following this, the compression enhancement layer uses a lightweight MLP model to learn the semantic association logic of the compressed token sequence, filling in the semantic gaps caused by token simplification. Semantic consistency calibration is used to adjust the token embedding vectors, resulting in a semantically coherent fine-tuned sequence. Finally, the fine-tuned token sequence is connected to the large language model at the output layer, completing the entire multimodal token processing flow.

[0083] For example, continuing with the smart glasses cycling navigation scenario, the keyword sequence extracted after transcribing the user's voice commands is [from the community, to the company, cycling route, dedicated cycling lane, avoid construction areas]; the extracted audio modality is a waveform containing complete voice content, while the image modality is a real-time image of the community entrance taken by the smart glasses' camera and a real-scene image of the target road segment cached by the device.

[0084] Next, the text encoder directly extracts a 768-dimensional embedding vector for each keyword from the keyword sequence; the visual encoder processes the visual feature vector of "blue markings for cycling lane" extracted from the community entrance image, and extracts the visual feature vectors of "red construction barriers" and "dedicated cycling lane signs" from the target road segment image, and standardizes them to 768 dimensions; the speech encoder parses the audio waveform: extracts the emphasis tone features of the keyword "avoid" and the core semantic spectrum features of "cycling route", and standardizes them to 768-dimensional audio feature vectors.

[0085] After entering the Token importance evaluation layer, the semantic weight of the text keyword "dedicated cycling lane" is calculated by combining multimodal features. The weight is 0.8+, the cosine similarity with "cycling lane markings / signs" in the image is 0.9+, and the matching degree with the keyword "cycling" in the audio is 0.85, with a total score of 0.88. In "avoid construction areas", the semantic weight of the text is 0.85+, the similarity with "construction fence" in the image is 0.92+, and the matching degree with the accent of "avoid" in the audio is 0.9, with a total score of 0.89.

[0086] The smart glasses are designed for a medium-resource scenario. Since all keywords are core semantics, the complete keyword sequence is preserved. Multimodal guided verification (the matching degree between "dedicated cycling lane" and image features > 0.7) confirms that no additional adjustments are needed. Next, a lightweight MLP is used to detect that the embedding vectors of "cycling route," "dedicated cycling lane," and "avoid construction area" are slightly far apart (cosine similarity 0.42). The spatial association between "cycling lane" and "construction fence" in the image is learned, and the embedding vectors of the three are fine-tuned. After fine-tuning, the similarity between "cycling route" and "dedicated cycling lane" increases to 0.86; the similarity between "cycling route" and "avoid construction area" increases to 0.84, filling potential semantic gaps. Finally, semantic consistency calibration ensures that the keyword sequence is fully aligned with the core requirements of the original instructions.

[0087] Finally, the fine-tuned keyword sequence [from the residential area to the company, cycling route, dedicated cycling lane, avoid construction area] is connected to the large language model of the smart glasses to generate the result "A cycling route from the residential area to the company has been planned, and the dedicated cycling lane will be selected first, and the construction area of ​​XX section will be automatically avoided".

[0088] It should be noted that the above examples are for illustrative purposes only and are not intended to limit this application.

[0089] This application provides a processing device for a model input sequence, the processing device for a model input sequence comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the processing method for the model input sequence in the first embodiment described above.

[0090] The following is for reference. Figure 6 This document illustrates a schematic diagram of a processing device suitable for implementing the model input sequence in the embodiments of this application. The processing device for the model input sequence in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The processing device for the model input sequence shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0091] like Figure 6As shown, the processing device for the model input sequence may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for the operation of the processing device for the model input sequence. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the model input sequence processing device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a model input sequence processing device with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0092] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0093] The model input sequence processing device provided in this application, employing the model input sequence processing method in the above embodiments, can solve the technical problem that high-value tokens are mistakenly deleted or low-value tokens are not effectively filtered, leading to difficulty in improving the inference efficiency of long model sequences. Compared with the prior art, the beneficial effects of the model input sequence processing device provided in this application are the same as those of the model input sequence processing method provided in the above embodiments, and other technical features in this model input sequence processing device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0094] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0095] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0096] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the model input sequence processing method in the above embodiments.

[0097] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM, or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0098] The aforementioned computer-readable storage medium may be included in the processing device for the model input sequence; or it may exist independently and not be assembled into the processing device for the model input sequence.

[0099] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a processing device for a model input sequence, cause the processing device for the model input sequence to: xxx.

[0100] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0102] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0103] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described model input sequence processing method. This solves the technical problem that high-value tokens are mistakenly deleted or low-value tokens are not effectively filtered, leading to difficulties in improving the efficiency of long-sequence inference in the model. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the model input sequence processing method provided in the above embodiments, and will not be elaborated upon here.

[0104] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for processing a model input sequence, characterized in that, The processing method for the model input sequence includes: Extract the feature vector of the input modality and calculate the importance evaluation result of each text token in the feature vector; Obtain the current system resource utilization rate, and select a compression strategy based on the system resource utilization rate; Based on the compression strategy, each text token after being sorted by the importance evaluation results is compressed to obtain the target text token; The target text token is output to the preset processing model.

2. The method for processing the model input sequence as described in claim 1, characterized in that, After the step of compressing each text token after it has been sorted by importance assessment results based on the compression strategy to obtain the target text token, the method for processing the model input sequence further includes: The semantic association information of the target text Token is learned by a multilayer perceptron (MLP), and the semantic gaps of the target text Token are updated. Based on the target text token updated by semantic detection, a lightweight model is trained by comparing the learning loss function and the semantic reconstruction loss function to obtain the target text token with semantic consistency.

3. The method for processing the model input sequence as described in claim 1, characterized in that, The input modality includes text modality, image modality, and speech modality. The step of extracting the feature vector of the input modality and calculating the importance evaluation result of each text token in the feature vector includes: Extract the text token embedding vector of the text modality, the visual feature vector of the image modality, and the audio feature vector of the audio modality from the input modality; Standardize the text token embedding vector, the visual feature vector, and the audio feature vector; Calculate the internal importance evaluation result of each text token in the text token embedding vector, and calculate the visual importance evaluation result between each text token and the visual feature vector and / or calculate the audio importance evaluation result between each text token and the audio feature vector; Based on the internal importance assessment results of each text token, and according to the visual importance assessment results and / or the audio importance assessment results, the importance assessment results of each text token are calculated.

4. The method for processing the model input sequence as described in claim 3, characterized in that, The steps of calculating the visual importance assessment result between each text token and the visual feature vector and / or calculating the audio importance assessment result between each text token and the audio feature vector include: Calculate the cosine similarity between the text token and the image embedding vector, and take the maximum value of the cosine similarity as the evaluation value corresponding to the visual importance evaluation result; and / or Calculate the keyword matching degree between each text token and the speech keywords in the audio feature vector, and the sentiment matching degree between each text token and the audio feature vector; The weighted fusion score between the keyword matching degree and the sentiment matching degree is used as the evaluation value of the audio importance assessment result.

5. The method for processing the model input sequence as described in claim 3, characterized in that, Before the steps of extracting the feature vector of the input modality and calculating the importance evaluation result of each text token in the feature vector, the processing method of the model input sequence further includes: Obtain the text information after text writing based on the audio modality; Based on a preset algorithm, core prompt words are extracted from the text information, and the core prompt words are used as the text token of the text modality, and the text token is used as the input modality.

6. The method for processing the model input sequence as described in claim 1, characterized in that, The step of compressing each text token after it has been sorted by importance assessment results based on the compression strategy to obtain the target text token includes: Determine the current compression ratio corresponding to the compression strategy, wherein the current compression ratio is the sum of the adjustment parameter calculated by the system resource utilization and the base compression ratio; Calculate the number of text tokens to be retained based on the current compression ratio; The number of text tokens that are retained after being sorted in reverse order according to the importance assessment results is selected as the target text token.

7. The method for processing the model input sequence as described in claim 6, characterized in that, Following the step of selecting the number of text tokens to retain after being sorted in reverse order of the importance assessment results as the target text token, the method for processing the model input sequence further includes: Identify the compressed text token sequence; Based on the multimodal guided verification algorithm, the target tokens in the compressed text token sequence are determined to have a correlation matching degree between the text tokens and other modalities that is greater than a preset matching degree. Set the target token to the target text token to retain the target token.

8. The method for processing the model input sequence as described in claim 1, characterized in that, The input modality is a text modality. The steps of extracting the feature vector of the input modality and calculating the importance evaluation result of each text token in the feature vector include: Obtain the text token of the text modality, and extract the text token embedding vector of the text token based on the text encoder; The internal importance assessment result of each text token in the text token embedding vector is determined as the importance assessment result.

9. A processing device for a model input sequence, characterized in that, The processing device for the model input sequence includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the processing method for the model input sequence as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the processing method for the model input sequence as described in any one of claims 1 to 8.

Images