Information processing method and apparatus

Abstract

The application provides an information processing method and device, and relates to the technical field of artificial intelligence. The information processing method comprises the following steps: obtaining input information, wherein the input information comprises a target image; processing the target image to obtain a first visual token sequence; processing the first visual token sequence based on the similarity between visual tokens in the first visual token sequence to obtain a target visual token sequence; the number of visual tokens in the target visual token sequence is less than the number of visual tokens in the first visual token sequence; and generating a response result of the input information based on the target visual token sequence by using a target model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an information processing method and apparatus. Background Technology

[0002] With the widespread application of models in image understanding tasks, models typically need to convert input images into visual word sequences and process them. However, when the input image has a high resolution or contains rich details, the number of generated visual words is large, resulting in low processing efficiency of the model. Summary of the Invention

[0003] In view of this, this application provides an information processing method and apparatus.

[0004] According to a first aspect of this application, an information processing method is provided, comprising: obtaining input information, the input information including a target image; processing the target image to obtain a first visual word sequence; processing the first visual word sequence based on the similarity between visual words in the first visual word sequence to obtain a target visual word sequence; the number of visual words in the target visual word sequence being less than the number of visual words in the first visual word sequence; and generating a response result of the input information based on the target visual word sequence using a target model.

[0005] A second aspect of this application provides an information processing apparatus, comprising: a data acquisition module for acquiring input information, the input information including a target image; an image processing module for processing the target image to obtain a first visual word sequence; a data processing module for processing the first visual word sequence based on the similarity between visual words in the first visual word sequence to obtain a target visual word sequence; wherein the number of visual words in the target visual word sequence is less than the number of visual words in the first visual word sequence; and a result generation module for generating a response result of the input information based on the target visual word sequence using a target model.

[0006] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0007] The above and other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0008] Figure 1 A schematic diagram of an example environment in which the method according to an embodiment of this application can be applied is shown;

[0009] Figure 2The flowchart illustrating one of the information processing methods provided in the embodiments of this application is shown in the illustration;

[0010] Figure 3 The second flowchart illustrating an information processing method provided in an embodiment of this application is shown in the illustration.

[0011] Figure 4 The third flowchart illustrating an information processing method provided in an embodiment of this application is shown in the illustration.

[0012] Figure 5 The fourth flowchart illustrating an information processing method provided in an embodiment of this application is shown in the illustration.

[0013] Figure 6 The fifth flowchart illustrating an information processing method provided in an embodiment of this application is shown in the illustration.

[0014] Figure 7 The sixth flowchart illustrating an information processing method provided in an embodiment of this application is shown in the illustration.

[0015] Figure 8 The seventh flowchart illustrating an information processing method provided in an embodiment of this application is shown in the illustration.

[0016] Figure 9 The eighth flowchart illustrating an information processing method provided in an embodiment of this application is shown in the illustration.

[0017] Figure 10 This illustration schematically shows a flowchart of a multimodal information processing method provided in an embodiment of this application.

[0018] Figure 11 A block diagram of a processing system provided in an embodiment of this application is shown schematically. Detailed Implementation

[0019] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0020] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0021] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0022] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0023] Figure 1 A schematic diagram of an example environment in which the method according to an embodiment of this application can be applied is shown. In this example environment, application 125 is installed on terminal device 110. User 140 can interact with application 125 via terminal device 110 and / or an attached device of terminal device 110.

[0024] In some embodiments, application 125 can be downloaded and installed on terminal device 110. In some embodiments, application 125 can also be accessed in other ways, such as through a web page. Figure 1 In this environment, in response to the launch of application 125, terminal device 110 can display the interface 150 of application 125.

[0025] In some embodiments, terminal device 110 can communicate with server 130 to provide services to application 125. Terminal device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 110 can also support any type of user-facing interface. Server 130 can be various types of computing systems / servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, and computing devices in cloud environments.

[0026] In some embodiments, application 125 may provide multimodal model-based interactive functionality. Application 125 may include applications dedicated to providing multimodal services, or applications integrated with multimodal processing capabilities. Although Figure 1 The image shows a single application, but in reality, multiple applications can be installed on the terminal device 110.

[0027] In embodiments of this application, multiple target models 160 can be deployed locally on the terminal device 110 or remotely. In the case of remote deployment, the terminal device 110 can directly invoke the target model 160, or it can invoke the target model 160 via the server 130. Exemplarily, the target model 160 can be a visual language model with the ability to process multimodal information such as images and text. The terminal device 110 provides an interface 150 that can present interactions with the target model 160. In the interface 150, the user 140 can initiate a request to the target model 160 by inputting input information including a target image. Optionally, the user 140 can upload online or offline image files and interact with the target model 160 by combining text input (e.g., prompt text, natural language questions, or instructions) to request the target model 160 to complete various tasks such as image understanding, image question answering, visual reasoning, and image description generation.

[0028] In embodiments of this application, during interaction with user 140, target model 160 can process the target image in response to input information including the target image provided by user 140. During target image processing, the target image is first processed to obtain a first visual word sequence. Then, based on the similarity between visual words in the first visual word sequence, the first visual word sequence is processed again to obtain a target visual word sequence, wherein the number of visual words in the target visual word sequence is less than the number of visual words in the first visual word sequence. In this way, target model 160 can reduce the number of visual words that need to be processed, thereby reducing computational overhead and improving inference efficiency. Finally, target model 160 generates a response result based on the target visual word sequence and presents it to user 140 through interface 150. In some embodiments, during processing, target model 160 can invoke one or more tools 165 to assist in task execution and the provision of task results as needed. These tools 165 can be of any type, such as text generation tools, image processing tools, information search tools, online or offline databases, chart generation tools, etc.

[0029] In some embodiments, the environment may further include a management node for multiple target models 160, which can interact with the multiple target models 160. In some examples, the management node may, in response to a task request from user 140, determine the task requirements corresponding to the task request. The management node may then, based on the task requirements, assign the task request to the target model 160 that matches the task requirements, thereby requesting the target model 160 to perform the task.

[0030] In some embodiments, the target model 160 may be constructed based on one or more machine learning models. In some embodiments, the target model 160 may include at least a visual encoder and a language model. The visual encoder is used to encode the target image into a first visual word sequence, and the language model is used to process the target visual word sequence and generate a response result. In some embodiments, the target model 160 may be a multimodal large model capable of handling multiple modal inputs, such as text input, visual input (e.g., images, videos), audio input, etc. These machine learning models may include content-generative models capable of generating corresponding outputs based on model inputs. In some embodiments, the target model 160 may receive text-modal model inputs (e.g., natural language and / or machine language) and / or non-text-modal model inputs (e.g., images, speech, videos, etc.), and is capable of generating corresponding model outputs based on the model inputs, thereby completing the task execution.

[0031] It should be understood that the structure and function of the various elements in the environment are described for illustrative purposes only and are not intended to limit the scope of this application in any way.

[0032] The following will be based on Figure 1 The following describes the information processing method of the embodiments of this application in detail, based on the described scenario.

[0033] Figure 2 The flowchart illustrating one of the information processing methods provided in the embodiments of this application is shown.

[0034] In Example 1, as Figure 2 As shown, the information processing method may specifically include the following operations.

[0035] Operation S210 obtains input information, including the target image;

[0036] Operation S220 processes the target image to obtain the first visual word sequence;

[0037] Operation S230: Based on the similarity between visual words in the first visual word sequence, process the first visual word sequence to obtain the target visual word sequence; the number of visual words in the target visual word sequence is less than the number of visual words in the first visual word sequence;

[0038] Operation S240 generates a response result based on the target visual word sequence using the target model to generate the input information.

[0039] In operation S210, input information refers to the processing data provided to the model.

[0040] For example, the input information may include, but is not limited to: target image, prompt text, user instructions, task type identifier, etc.

[0041] Optionally, the input information includes a target image, which refers to image data that the model needs to perform visual understanding on, for extracting visual features and generating visual lexical units.

[0042] For example, the target image may include, but is not limited to, natural scene images, document images, scanned copies, screenshots, etc. The target image may be a single image or one or more frames from a video frame sequence.

[0043] Optionally, the target image can be a high-resolution image to preserve fine visual details. High-resolution images generate a large number of visual units after being processed by a visual encoder, which is the main reason why visual units need to be compressed in this embodiment.

[0044] In operation S220, the first visual lexical sequence refers to the sequence of visual feature representations extracted from the target image by the visual encoder. It can be understood as an intermediate representation that converts the image content into a lexical form that the model can process, for subsequent lexical compression processing and language model inference.

[0045] For example, the first visual word sequence may include, but is not limited to: a visual word sequence output by a visual transformer, a visual feature sequence extracted and transformed by a convolutional neural network, etc.

[0046] In one feasible implementation, the target image can be input into a visual encoder, which divides the target image into multiple image blocks, encodes each image block to obtain a corresponding visual word, and arranges all visual words according to the spatial position of the image blocks to form a first visual word sequence.

[0047] For example, when a visual transformer is used as a visual encoder, the target image is segmented into image blocks of fixed size. Each image block is processed by linear projection and positional encoding to obtain initial words. After processing through several transformation layers, the first visual word sequence is output.

[0048] Optionally, the visual encoder may include multiple cascaded transform layers. The first visual word sequence may be the output of one of the transform layers in the visual encoder, or it may be the output of the last transform layer. The visual word sequences output by different transform layers capture different levels of visual semantic information.

[0049] In operation S230, the target visual word sequence refers to the visual word sequence obtained after compressing the first visual word sequence, which can be understood as the visual feature representation after redundancy elimination or information simplification.

[0050] For example, the target visual lexical sequence may include, but is not limited to: a compressed visual lexical sequence obtained by lexical merging, a visual lexical sequence obtained by lexical filtering, and a visual lexical sequence obtained by a combination of merging and filtering.

[0051] In one feasible implementation, the similarity between different visual words in the first visual word sequence can be calculated. Based on the similarity, the degree of redundancy between visual words can be determined. Visual words with high redundancy are compressed to reduce the number of visual words and obtain the target visual word sequence. For example, visual words with high similarity can be merged into one visual word, or some visual words can be selected and retained.

[0052] In another feasible implementation, the first visual word sequence can be processed based on a similarity threshold. Visual words with a similarity higher than the threshold can be merged or filtered to obtain the target visual word sequence. The value of the similarity threshold affects the final compression ratio; the higher the similarity threshold, the more visual words are retained, and the lower the compression ratio.

[0053] It should be noted that the number of visual words in the target visual word sequence is less than the number in the first visual word sequence, thus achieving visual word compression. The compressed target visual word sequence significantly reduces the computational load of subsequent language model processing while retaining the main visual information. Since the computational complexity of self-attention in a language model is positively correlated with the square of the word sequence length, reducing the number of visual words can significantly improve the model's inference efficiency.

[0054] Optionally, similarity can be calculated based on the feature vector of the visual word itself, or based on the key representation or query representation of the visual word in the transformation layer. Different representations capture different semantic characteristics of visual words, and the similarity calculated based on the key representation can reflect the degree of matching of visual words in attention calculation.

[0055] Optionally, the length of the target visual word sequence can be adaptively adjusted according to the information density of the target image. For images with higher information density, the similarity between visual words in the first visual word sequence is relatively low, and a relatively large number of visual words are retained after compression. Conversely, for images with lower information density, the similarity between visual words is relatively high, and a relatively small number of visual words are retained after compression.

[0056] It should be noted that the compression process in this application does not require the introduction of an additional training module. It can achieve word compression simply by analyzing the similarity relationship between visual words. Therefore, it can be directly applied to a trained model without the need for retraining or fine-tuning the model parameters.

[0057] In operation S240, the response result refers to the output content generated by the target model based on the input information.

[0058] For example, the response results may include, but are not limited to: textual descriptions of the image, question-and-answer responses to the image content, textual content identified from the image, analysis results of the image content, and inference conclusions based on the image content.

[0059] In one feasible implementation, the target visual word sequence can be input into the target model, which processes and infers from the sequence to generate a response. For example, the target model takes the target visual word sequence as part of the input and generates output words one by one through autoregression, ultimately forming a complete response text.

[0060] In another feasible implementation, the target visual word sequence and the text word sequence can be concatenated and input into the target model. The target model processes both visual and text information simultaneously to generate a response. The text word sequence originates from prompts or user instructions in the input information. The target model combines the visual content with the text instructions to complete a specific task.

[0061] Optionally, the target model may include a visual encoder and a language model. The visual encoder is used to extract visual features and generate visual word sequences, while the language model is used to process visual and text words and generate response results. The language model typically employs a large-scale language model based on a transformer architecture, possessing powerful semantic understanding and reasoning capabilities.

[0062] Optionally, the target model can be a large multimodal language model capable of processing both visual and textual information simultaneously. The target model achieves unified representation and joint processing of visual and textual information by projecting visual lexical units into the same embedding space as textual lexical units.

[0063] Optionally, during the response generation process, the target model assigns attention weights to each visual word in the target visual word sequence, with different visual words contributing differently to the response. Through word compression, the target visual word sequence retains visual information that is more relevant to the task, helping the target model generate more accurate response results.

[0064] In one example, the system receives a high-resolution chest CT scan image as input. The system processes the CT scan image into a visual encoder to generate a first visual word sequence. Since many normal lung tissue areas in the CT scan image exhibit similar grayscale values ​​and texture features, the corresponding visual words have high similarity. The system calculates the similarity between visual words, merging high-similarity visual words corresponding to normal tissue areas, while retaining the visual words corresponding to the nodular lesion in the right lower lobe due to significant differences from surrounding tissues and low similarity, thus obtaining the target visual word sequence. The target model infers based on the compressed target visual word sequence, generating the response: "A nodular high-density shadow with a diameter of approximately 8 mm was found in the right lower lobe; further examination is recommended." Through this processing, the number of visual words is significantly reduced while retaining key lesion information, improving inference efficiency.

[0065] In another example, the system receives an ultra-high-resolution satellite remote sensing image as input, covering a large area of ​​agricultural land. The system inputs the remote sensing image into a visual encoder for processing, generating a first visual word sequence. Because large areas of farmland, forest, and water in the remote sensing image exhibit high consistency in color and texture, the corresponding visual words have high similarity. By calculating the similarity between visual words, highly similar visual words within the same land area are merged, while visual words corresponding to the boundaries of land areas, roads, buildings, and other features are retained due to significant feature differences and low similarity, thus obtaining the target visual word sequence. The target model infers based on the compressed target visual word sequence, generating the response: "The image contains approximately 60% farmland, 25% forest, and 15% water; three suspected new buildings were detected." Through this processing, key feature information is preserved while significantly reducing memory usage, enabling the system to process ultra-high-resolution remote sensing images that originally required high-performance servers on ordinary computing devices.

[0066] By adopting the technical solution of this application, after processing the target image to obtain a first visual word sequence, the first visual word sequence is further processed based on the similarity between visual words in the first visual word sequence. This process can identify and process redundant visual words with high similarity, resulting in a target visual word sequence with fewer words. Since the number of visual words in the target visual word sequence is less than that in the first visual word sequence, the number of visual words that the target model needs to process when generating response results based on the target visual word sequence is reduced. This reduces the computational complexity of the model in processes such as self-attention mechanisms, reduces the computational overhead caused by processing redundant information, and thus improves the model's inference efficiency. Furthermore, since the processing is based on the similarity between visual words, key information of the image can be preserved while reducing the number of words, avoiding a significant impact on the quality of the response results, and achieving a balance between model inference efficiency and output quality.

[0067] Figure 3 The second flowchart illustrating an information processing method provided in an embodiment of this application is shown.

[0068] like Figure 3 As shown, based on the above embodiments, Embodiment 2 may also include one of the following operations.

[0069] Operation S310 involves merging the first visual word sequence based on the similarity between visual words in the first visual word sequence to obtain the target visual word sequence.

[0070] Operation S320 involves filtering visual words in the first visual word sequence based on the similarity between visual words in the first visual word sequence to obtain the target visual word sequence.

[0071] In operation S310, the merging process refers to the process of integrating multiple visual lexical units with similar features in the first visual lexical unit sequence into a smaller number of visual lexical units.

[0072] For example, the merging process may include, but is not limited to: mean fusion of visual words with similarity exceeding a certain threshold, weighted fusion of visual words with similar features, and selective fusion of redundant visual words.

[0073] In one feasible implementation, the similarity between visual words in the first visual word sequence can be calculated, visual word pairs or groups with high similarity can be identified, and these visual words can be merged to obtain new visual words. Visual words not involved in the merging remain unchanged, and all visual words together constitute the target visual word sequence. For example, for two visual words with a similarity exceeding a preset threshold, the average of their feature vectors can be calculated as the new visual word after merging.

[0074] In another feasible implementation, the merging process can be performed iteratively, processing a pair or group of visual words with the highest similarity in each merging operation, until the length of the target visual word sequence reaches the expected requirement or the similarity between all visual words is below the threshold.

[0075] Optionally, the merging process can calculate similarity based on the feature representations of visual lexical units. These feature representations include the embedding vector of the visual lexical unit itself, the corresponding key representation, or the corresponding query representation. Different feature representations reflect the semantic information of visual lexical units at different processing stages of the model.

[0076] It should be noted that the merging process can effectively reduce the number of visual lexical units while preserving the information of each visual lexical unit involved in the merging. It is suitable for situations where there are a large number of similar or redundant lexical units in the first visual lexical unit sequence. The length of the target visual lexical unit sequence after merging is shorter than that of the first visual lexical unit sequence, and each lexical unit may carry information from multiple original lexical units.

[0077] In operation S320, the word filtering process refers to the process of selecting and retaining some visual words from the first visual word sequence and removing other visual words. It can be understood as a compression method that reduces the sequence length by selectively retaining important words.

[0078] For example, lexical filtering processing may include, but is not limited to: retaining visual lexical units with low similarity, retaining visual lexical units with high information content, and removing visual lexical units with high redundancy.

[0079] In one feasible implementation, visual lexical units to be retained can be determined based on the similarity distribution among visual lexical units. Lexical units with low similarity to other visual lexical units are retained, while those with high similarity to other visual lexical units are filtered out or selectively retained. The retained visual lexical units constitute the target visual lexical unit sequence. For example, for regions with highly similar lexical units, only one representative lexical unit can be retained, while other similar lexical units are removed.

[0080] In another feasible implementation, a retention ratio or target word number can be set, and the first visual word sequence can be sorted or grouped according to the similarity features of visual words. Visual words can be selected according to the retention ratio or target number to form the target visual word sequence.

[0081] Optionally, the word filtering process can incorporate the spatial location information of visual words to avoid significant spatial unevenness in the filtered target visual word sequence. The spatial location information comes from the position of the image patch corresponding to the visual word in the target image.

[0082] Optionally, the lexical filtering process can maintain the original order of the first visual lexical sequence, and the relative order of the retained visual lexical units in the target visual lexical sequence is consistent with their relative order in the first visual lexical sequence.

[0083] It should be noted that merging and word filtering can be used individually or in combination. When used in combination, merging can be performed first, followed by word filtering, or vice versa, or the two processing methods can be performed alternately. Different processing orders and combinations are suitable for different image features and task requirements.

[0084] By adopting the above technical solution, based on the similarity between visual words in the first visual word sequence, either merging or word filtering can be used to compress the first visual word sequence. Merging reduces the number of words by integrating similar words, while word filtering reduces the number of words by selectively retaining them. Both methods can reduce the length of the target visual word sequence, thereby reducing the computational load of subsequent model processing.

[0085] Figure 4 The third flowchart illustrating an information processing method provided in an embodiment of this application is shown.

[0086] like Figure 4 As shown, based on the above embodiment two, embodiment three may also include the following operations.

[0087] Operation S410 divides the first visual word sequence into a first word group and a second word group;

[0088] Operation S420: Determine target word pair, which includes at least one first word in a first word group and at least one second word in a second word group, wherein the first similarity between the first word and the second word is greater than the first similarity threshold.

[0089] Operation S430 merges the target word pairs to obtain the target visual word sequence.

[0090] In operation S410, the first lexical group and the second lexical group refer to the two visual lexical subsets obtained by partitioning the first visual lexical sequence.

[0091] For example, the partitioning rules may include, but are not limited to: partitioning according to the position of visual words in the sequence, partitioning according to odd and even indices, partitioning according to the spatial position of the image blocks corresponding to the visual words, etc.

[0092] In one feasible implementation, visual lexical units with odd-numbered indices in the first visual lexical unit sequence can be divided into a first lexical unit group, and visual lexical units with even-numbered indices can be divided into a second lexical unit group, so that the number of lexical units in the two lexical units is basically equal.

[0093] In another feasible implementation, visual lexical units can be alternately assigned to the first lexical group and the second lexical group in the order of their positions in the sequence. The first lexical unit is assigned to the first lexical group, the second lexical unit is assigned to the second lexical group, the third lexical unit is assigned to the first lexical group, and so on.

[0094] Optionally, the partitioning operation can maintain the assignment of adjacent words in the first visual word sequence to different word groups, so that the words in the first word group and the words in the second word group are spatially interspersed.

[0095] In operation S420, target word pair refers to visual word pairings from different word groups that meet the similarity condition.

[0096] For example, target word pairs may include, but are not limited to: a pair consisting of a word in the first word group and the word in the second word group that is most similar to it, or any cross-group word pair where the first similarity exceeds the first similarity threshold.

[0097] In one feasible implementation, for each first word element in the first word element group, the first similarity between it and each second word element in the second word element group can be calculated. The second word element with the largest first similarity is selected to form a candidate word element pair with the first word element. It is then determined whether the first similarity of the candidate word element pair is greater than the first similarity threshold. If it is greater, the candidate word element pair is determined as the target word element pair.

[0098] For example, for the i-th word in the first word group, calculate its first similarity with all words in the second word group, find the j-th word with the highest similarity, and if the similarity value is greater than the first similarity threshold, then the i-th word and the j-th word constitute the target word pair.

[0099] In another feasible implementation, all possible word pairings between the first word pair and the second word pair can be traversed, the first similarity of each pair can be calculated, and the pairings with the first similarity greater than the first similarity threshold can be identified as the target word pair.

[0100] Optionally, the first similarity can be calculated based on the feature representation of visual lexical units, including the key representation, query representation, or value representation corresponding to the visual lexical unit. The first similarity calculated using the key representation can reflect the degree of matching of visual lexical units in the attention calculation.

[0101] Optionally, the first similarity can be calculated using methods such as cosine similarity or Euclidean distance.

[0102] Optionally, a first word element can form a target word element pair with one or more second word elements, and a second word element can also form a target word element pair with one or more first word elements.

[0103] In operation S430, the merging process refers to the process of fusing visual lexical units in the target lexical pair to obtain a new visual lexical unit.

[0104] For example, the merging process may include, but is not limited to: calculating the arithmetic mean of the feature vectors of each word in the target word pair, calculating the weighted average, and selecting the feature of one word as the merging result.

[0105] In one feasible implementation, for each target word pair, the average of the feature vectors of the first word and the second word contained therein can be calculated, and this average value can be used as the new visual word after merging. The first word and the second word that did not participate in the pairing remain unchanged. All the new visual words after merging and the visual words that did not participate in the merging together constitute the target visual word sequence.

[0106] For example, for a target word pair consisting of the i-th first word and the j-th second word, the average of their feature vectors is calculated to obtain a new word, which replaces the original i-th and j-th word.

[0107] In another feasible implementation, different weights can be assigned to the first and second lexical units in the target lexical pair, and a weighted average can be calculated based on these weights to obtain the new visual lexical unit after merging. The weights can be determined based on lexical importance, attention score, or other evaluation metrics.

[0108] Optionally, the merging process can process all target word pairs simultaneously, completing the merging operation of all words in one go.

[0109] It should be noted that by dividing the first visual word sequence into two word groups and performing cross-group matching, unordered pairing between arbitrary word groups can be avoided, reducing the complexity of similarity calculation. Compared to calculating the similarity between all word groups, this division method only needs to calculate the similarity between the first word group and the second word group, reducing the computational load by about half.

[0110] It should be noted that the length of the target visual word sequence depends on the number of target word pairs. Assuming the first visual word sequence contains N words and M target word pairs are identified, the target visual word sequence contains NM words. When the first similarity threshold is set low, more word pairs that meet the condition are merged, resulting in a shorter target visual word sequence and a greater compression ratio.

[0111] By employing the above technical solution, dividing the first visual word sequence into two word groups and then performing cross-group matching can identify word pairs with high similarity with low computational complexity. Target word pairs requiring merging are selected using a first similarity threshold, achieving precise control over the merging operation. Merging these target word pairs effectively reduces the number of visual words, resulting in a target visual word sequence that retains important visual information while reducing sequence length.

[0112] Figure 5 The fourth flowchart illustrating an information processing method provided in the embodiments of this application is shown.

[0113] like Figure 5 As shown, based on the above embodiments, the input information also includes prompt text, and Embodiment Four may also include the following operations.

[0114] Operation S510: Based on the similarity between visual words in the first visual word sequence, process the first visual word sequence to obtain the second visual word sequence.

[0115] Operation S520 processes the prompt text to obtain a text word sequence;

[0116] Operation S530 involves filtering the second visual word sequence based on the similarity between the text word sequence and the second visual word sequence to obtain the target visual word sequence.

[0117] In this embodiment, the prompt text refers to text information associated with the target image, which is used to guide the model in understanding and processing the target image.

[0118] For example, prompt text may include, but is not limited to: user-inputted question text, task instruction text, description requirements for the target image, and query statements regarding the image content. For instance, in a visual question-answering scenario, prompt text could be "What objects are in the image?" or "Extract table data from a document."

[0119] Optionally, the prompt text can be preset or entered by the user in real time. In different application scenarios, the content and format of the prompt text are determined according to the specific task requirements.

[0120] In operation S510, the second visual lexical sequence refers to the intermediate visual lexical sequence obtained after the first visual lexical sequence has undergone preliminary compression processing.

[0121] For example, the processing methods for obtaining the second visual word sequence may include, but are not limited to: merging the first visual word sequence, or filtering the first visual word sequence based on visual similarity.

[0122] In one feasible implementation, the merging process described in the foregoing embodiments can be used to divide the first visual word sequence into two word groups, identify target word pairs that meet the similarity conditions, and merge them to obtain the second visual word sequence.

[0123] In operation S520, the text lexical sequence refers to the lexical representation sequence obtained after encoding the prompt text.

[0124] For example, the processing methods may include, but are not limited to: segmenting and encoding the prompt text using a text segmenter, and extracting the feature representation of the prompt text using a text encoder.

[0125] In one feasible implementation, the prompt text can be input into a text segmenter, which divides the prompt text into multiple text units, encodes each text unit to obtain a corresponding text word, and arranges all text words in the order they appear in the prompt text to form a text word sequence. For example, for the prompt text "Please describe the main content in the image," the text segmenter divides it into several words or sub-word units, with each unit corresponding to a text word.

[0126] In operation S530, the filtering process refers to the process of selecting and retaining visual words that are highly relevant to the text word sequence from the second visual word sequence based on cross-modal similarity.

[0127] For example, the filtering process may include, but is not limited to: retaining visual words that are highly similar to the text word sequence, removing visual words that are less similar to the text word sequence, and filtering based on a similarity threshold.

[0128] In one feasible implementation, the similarity between each visual word in the second visual word sequence and each text word in the text word sequence can be calculated. Based on these similarity values, the relevance of each visual word to the task can be determined. Visual words with higher relevance are retained, while visual words with lower relevance are removed. The retained visual words constitute the target visual word sequence.

[0129] In another feasible implementation, for each visual word in the second visual word sequence, its overall similarity with the text word sequence can be calculated, and the visual words can be sorted according to the overall similarity. The visual words with the highest similarity ranking can be selected to form the target visual word sequence.

[0130] Optionally, the filtering process can be based on a second similarity threshold, retaining visual words that have a similarity to the text word sequence greater than the second similarity threshold. The setting of the second similarity threshold affects the final number of retained visual words.

[0131] It should be noted that this embodiment employs a two-stage compression process. The first stage, through operation S510, compresses the first visual word sequence based on intra-modal visual similarity, eliminating redundancy in the visual features themselves. The second stage, through operation S530, filters the second visual word sequence based on cross-modal similarity, retaining task-relevant visual information. The two stages work together to reduce visual word redundancy while ensuring the matching degree between the retained visual words and task requirements.

[0132] By employing the above technical solution, when the input information includes prompt text, the first visual word sequence is processed based on the similarity between visual words to obtain the second visual word sequence, completing the initial compression based on internal visual features. Then, the prompt text is processed to obtain the text word sequence, acquiring task-related semantic information. Further filtering is performed based on the cross-modal similarity between the text word sequence and the second visual word sequence, retaining visual words more relevant to the task to obtain the target visual word sequence. This two-stage processing method combines internal visual redundancy elimination and cross-modal task guidance, reducing the number of visual words while retaining visual information important to the task, thus ensuring the quality of the model's response results while reducing computational complexity.

[0133] Based on the above-described embodiment four, embodiment five may further include the following operations.

[0134] The second visual lexical sequence is transformed using a feedforward network to obtain the third visual lexical sequence; the third visual lexical sequence is a semantically transformed visual lexical sequence; the third visual lexical sequence is filtered based on the text lexical sequence to obtain the target visual lexical sequence.

[0135] In this embodiment, the feedforward network refers to a network structure used to perform nonlinear transformations on visual features, which can be understood as a transformation module that maps visual features to a higher-level semantic space.

[0136] For example, the feedforward network may include, but is not limited to: one or more fully connected layers, a multilayer perceptron containing activation functions, a feedforward subnetwork containing residual connections, etc.

[0137] In one feasible implementation, the feedforward network may include two fully connected layers. The first fully connected layer expands the dimension of the input features, and after passing through an activation function, the second fully connected layer restores the feature dimension to the original dimension.

[0138] It should be noted that the second visual lexical sequence is a sequence obtained after merging visual lexical units based on their similarity. The lexical features in this sequence still primarily represent visual-level features. A feedforward network is used to transform the features of the second visual lexical sequence, mapping the visual features to a higher-level semantic space, resulting in the third visual lexical sequence. The lexical features in the third visual lexical sequence not only contain visual information but also high-level semantic information after semantic abstraction, enabling these features to be compared with the text lexical sequence in the same semantic space.

[0139] In one feasible implementation, the second visual word sequence can be processed using a feedforward subnetwork of a visual encoder in a multimodal model. A visual encoder typically includes multiple transform layers, each containing an attention sublayer and a feedforward subnetwork. After obtaining the second visual word sequence, it is input into the feedforward subnetwork for feature transformation. The feedforward subnetwork performs non-linear mapping on the features of each word, extracting higher-level semantic features and outputting a third visual word sequence.

[0140] Optionally, the feedforward network can be an existing network structure in a pre-trained multimodal model, requiring no additional training. By utilizing the feedforward network in the pre-trained model, feature transformation capabilities trained on large-scale data can be directly obtained.

[0141] It's important to note that the purpose of semantic processing is to improve the accuracy of cross-modal filtering. The lexical features in the second visual lexical sequence primarily focus on visual-level information, such as low- or mid-level visual features like color, texture, and shape. In contrast, the text lexical sequence represents semantic information at the linguistic level. Directly calculating similarity to text based on visual features may not accurately identify the correspondence between visual content and text semantics. By mapping visual features to the semantic space through a feedforward network, a semantically processed third visual lexical sequence is obtained. This allows visual and text lexical units to be compared at the same semantic level, enabling a more accurate assessment of the relevance between visual content and textual cues, thereby improving the accuracy of the filtering process.

[0142] By employing the above technical solution, a feedforward network is used to transform the features of the second visual word sequence, mapping the visual features to a higher-level semantic space to obtain a semantically transformed third visual word sequence. Semanticization ensures that the feature representations of visual words and text words are at the same semantic level, facilitating accurate cross-modal similarity calculation. Filtering based on the semantically transformed third visual word sequence enables more accurate identification of visual content related to the prompt text.

[0143] Figure 6 The fifth flowchart illustrating an information processing method provided in the embodiments of this application is shown.

[0144] like Figure 6 As shown, based on the above embodiment four, embodiment six may also include the following operations.

[0145] Operation S610 calculates the second similarity between each visual word in the second visual word sequence and each text word in the text word sequence; the second similarity represents the semantic similarity between the visual word and the text word.

[0146] Operation S620: Determine the target visual word in the second visual word sequence based on the second similarity, wherein at least one of the multiple second similarities corresponding to the target visual word is greater than the second similarity threshold;

[0147] Operation S630 obtains the target visual word sequence based on the target visual word.

[0148] In operation S610, the second similarity refers to a numerical value that measures the degree of semantic association between visual lexical units and textual lexical units.

[0149] In one feasible implementation, for each visual word in the second visual word sequence, the cosine similarity between it and the feature vector of each text word in the text word sequence can be calculated to obtain multiple second similarity values ​​corresponding to the visual word.

[0150] Assuming the second visual word sequence contains P visual words and the text word sequence contains M text words, then a total of P×M second similarity values ​​need to be calculated. For example, for the i-th visual word in the second visual word sequence, calculate its similarity with the 1st, 2nd, up to the Mth text words in the text word sequence to obtain M second similarity values.

[0151] Optionally, before calculating the second similarity, the feature vectors of visual and textual words can be normalized to ensure that the feature vectors of different words have the same magnitude, thus avoiding the influence of differences in feature vector magnitude on the similarity calculation.

[0152] It should be noted that the second similarity score reflects the strength of the association between visual content and text semantics. The higher the second similarity score, the more semantically related the corresponding visual word and text word are. By calculating the similarity between each visual word and all text words, the overall relevance of the visual word and the prompt text can be comprehensively assessed.

[0153] In operation S620, the target visual word refers to the visual word that is determined to be retained in the second visual word sequence.

[0154] In one feasible implementation, for each visual word in the second visual word sequence, its corresponding multiple second similarity values ​​can be checked to determine whether there is at least one value greater than the second similarity threshold among these second similarity values. If so, the visual word is identified as the target visual word. For example, for the i-th visual word in the second visual word sequence, its corresponding M second similarity values ​​are s i1 s i2 ... s iM If at least one of these M values ​​is greater than the second similarity threshold, then the i-th visual word is the target visual word.

[0155] Optionally, the maximum value among multiple second similarities corresponding to the target visual word element is greater than the second similarity threshold. The maximum value represents the similarity between the visual word element and the most relevant text word element in the text word element sequence. The larger the maximum value, the more highly relevant the visual word element is to at least one semantic component in the prompt text.

[0156] Optionally, the second similarity threshold can be a pre-set fixed value, or it can be dynamically determined based on the features of the second visual word sequence and the text word sequence.

[0157] It should be noted that by controlling that at least one of the multiple second similarities corresponding to the target visual word element is greater than the second similarity threshold, it is ensured that the retained visual word elements have a strong correlation with at least some part of the semantic content in the prompt text. This judgment method avoids filtering out important visual information, because even if a visual word element is only related to a part of the content in the prompt text, as long as the correlation is strong enough, the visual word element will still be retained.

[0158] In operation S630, all words that are identified as target visual words in the second visual word sequence can be extracted and arranged according to their original position order in the second visual word sequence to form the target visual word sequence.

[0159] Optionally, the target visual word sequence maintains the relative positional relationship of the target visual words in the second visual word sequence, which helps to preserve the spatial structural characteristics of visual information.

[0160] Optionally, the length of the target visual word sequence depends on the number of target visual words that meet the conditions in the second visual word sequence, and this number is affected by the second similarity threshold. The higher the second similarity threshold, the fewer the number of visual words identified as targets, and the shorter the length of the target visual word sequence.

[0161] By employing the aforementioned technical solution, the second similarity between each visual word in the second visual word sequence and each text word in the text word sequence is calculated, establishing a complete cross-modal similarity relationship. Target visual words are determined based on the second similarity and the second similarity threshold, filtering out visual information semantically relevant to the prompt text. The determined target visual words are then used to construct a target visual word sequence, achieving task-oriented visual word filtering. This reduces the number of visual words while retaining visual features highly relevant to the task, thus reducing computational complexity while ensuring the accuracy of the model's response results.

[0162] Figure 7 The sixth flowchart illustrating an information processing method provided in an embodiment of this application is shown.

[0163] like Figure 7 As shown, based on the above embodiments, Embodiment Seven may also include the following operations.

[0164] Operate S710 to obtain the calibration dataset; the calibration dataset includes sample images and sample prompt text.

[0165] Operate the S720 to obtain the target compression ratio;

[0166] Operation S730 determines a first similarity threshold and a second similarity threshold based on the calibration dataset and the target compression ratio. The first similarity threshold is used to process the first visual word sequence based on the similarity between visual words in the first visual word sequence.

[0167] In operation of S710, the calibration dataset refers to the set of sample data used to determine compression processing parameters, which can be understood as reference data reflecting the characteristic distribution of the actual application scenario.

[0168] For example, the calibration dataset may include, but is not limited to: typical image and text pairs collected from the target application scenario, sample data with features similar to the actual input information, and representative samples that can cover different image complexities.

[0169] In one feasible implementation, a calibration dataset can be constructed by selecting several sample images and their corresponding sample prompt texts from the target application scenario. The sample images cover possible situations in actual applications in terms of content type, information density, and visual complexity. For example, in a document understanding scenario, the calibration dataset includes document images with different layout densities, table images with different content complexities, and mixed document images containing different amounts of text and graphic elements.

[0170] Optionally, the sample images may include images with high information density and images with low information density. Images with high information density include scenes with rich details, documents containing a lot of text, objects with complex textures, etc. Images with low information density include scenes with large backgrounds, photos of a single object, simple scenes with uniform colors, etc.

[0171] In the operation of S720, the target compression ratio refers to the expected degree of visual word compression, which can be understood as the ratio of the number of visual words retained after compression to the total number of original visual words.

[0172] For example, the target compression ratio can be expressed as a numerical value r, which is calculated by dividing the number of compressed visual words by the total number of visual words.

[0173] In one feasible implementation, the target compression ratio can be set according to the computing power of the deployed device and the real-time requirements of the task. In scenarios with limited computing resources or high real-time requirements, a smaller target compression ratio can be set to achieve a higher degree of compression, while in scenarios with sufficient computing resources or high accuracy requirements, a larger target compression ratio can be set to retain more visual information.

[0174] In operation S730, determining the first similarity threshold and the second similarity threshold refers to finding the combination of threshold parameters that can achieve the target compression ratio by performing a compression process on the calibration dataset and evaluating the compression effect.

[0175] For example, the determination process may include, but is not limited to, setting multiple sets of candidate threshold combinations, evaluating the compression ratio of each set of candidate thresholds on a calibration dataset, and selecting the optimal threshold combination.

[0176] In one feasible implementation, several candidate threshold combinations (ts, te) can be set, where ts represents the first similarity threshold of the candidates, and te represents the second similarity threshold of the candidates. For each candidate threshold combination, a complete compression process is performed on each sample image in the calibration dataset, including converting the sample image into a first visual word sequence, processing the first visual word sequence based on the candidate first similarity threshold ts to obtain a second visual word sequence, and filtering the second visual word sequence based on the candidate second similarity threshold te to obtain the target visual word sequence. The average compression ratio of all samples in the calibration dataset is calculated by dividing the sum of the lengths of the target visual word sequences of all samples by the sum of the lengths of the first visual word sequences of all samples. The difference between the average compression ratio corresponding to each candidate threshold combination and the target compression ratio r is compared, and the candidate threshold combination that makes the average compression ratio closest to the target compression ratio r is selected as the finally determined first and second similarity thresholds.

[0177] In another feasible implementation, an iterative search method can be used to determine the threshold combination. An initial set of thresholds (ts, te) is set, and the corresponding average compression ratio is evaluated on the calibration dataset. The threshold values ​​are adjusted according to the difference between the average compression ratio and the target compression ratio. The evaluation and adjustment process is repeated until the average compression ratio converges to near the target compression ratio.

[0178] It should be noted that the first similarity threshold controls the degree of merging processing based on the similarity between visual units, while the second similarity threshold controls the degree of filtering processing based on cross-modal similarity. These two thresholds together determine the final visual unit compression effect. By simulating the complete compression process on a calibration dataset and statistically analyzing the average compression ratio, a threshold parameter configuration that matches the target compression ratio can be found.

[0179] It should be noted that the above threshold determination method achieves adaptive length visual word compression. Adaptive length means that the number of visual words retained after compression is automatically adjusted according to the information density of the image itself.

[0180] Specifically, for images with high information density, such as detailed scene images, document images containing a large amount of text, and object images with complex textures, the overall similarity between visual lexical units is relatively low. When performing merging processing based on the first similarity threshold, the number of visual lexical unit pairs with similarity exceeding the first similarity threshold is small, resulting in fewer merged lexical units and a larger number of lexical units retained in the second visual lexical unit sequence. When performing filtering processing based on the second similarity threshold, due to the retention of more differentiated visual features, the final target visual lexical unit sequence is longer, thus fully expressing rich visual content.

[0181] Conversely, for images with low information density, such as scene images with large, uniform backgrounds, simple compositions of single objects, and images with minimal color and texture variations, the overall similarity between visual units is relatively high. When performing merging processing based on a first similarity threshold, a large number of visual unit pairs with similarity exceeding the first threshold are present, resulting in a large number of merged units and significantly compressing the length of the second visual unit sequence. The final target visual unit sequence retains fewer visual units, avoiding the retention of redundant information.

[0182] Using the similarity-based compression method described above, the same set of first and second similarity thresholds will produce different compression levels when processing different images, with the compression level adapting to the information density of the image. Images with high information density are compressed less due to their low internal similarity, while images with low information density are compressed more due to their high internal similarity, thus achieving an adaptive compression effect. This adaptability avoids both over-compression of complex images leading to information loss and under-compression of simple images leading to wasted computational resources.

[0183] It should be noted that the first and second similarity thresholds determined on the calibration dataset ensure that the average compression ratio of the calibration dataset is close to the target compression ratio. The actual compression ratio of an individual sample image will fluctuate around the target compression ratio based on its information density characteristics. Samples with high information density will have a higher compression ratio than the target compression ratio, while samples with low information density will have a lower compression ratio than the target compression ratio, but the overall average effect will meet the requirements of the target compression ratio.

[0184] By adopting the above technical solution, data features of actual application scenarios are obtained through a calibration dataset. A target compression ratio is set as the optimization objective. A complete compression process is executed on multiple candidate thresholds on the calibration dataset, and the average compression ratio is calculated. The threshold combination that makes the average compression ratio closest to the target compression ratio is selected as the first similarity threshold and the second similarity threshold. The determined threshold parameters can achieve the expected compression effect in practical applications. Furthermore, since the compression process is based on similarity judgment, the compression degree can be adaptively adjusted according to the information density of the image. Images with high information density retain more visual words to maintain information integrity, while images with low information density retain fewer visual words to improve computational efficiency. This achieves a balance between computational efficiency and information preservation on different types of input images.

[0185] Figure 8 The seventh flowchart illustrating an information processing method provided in an embodiment of this application is shown.

[0186] like Figure 8 As shown, based on the above embodiments, Embodiment 8 may also include the following operations.

[0187] Operation S810 processes the first visual word sequence based on the similarity between visual words in the first visual word sequence to obtain the first intermediate word sequence.

[0188] Operation S820 inputs the first intermediate word sequence into the second transformation layer to obtain the second intermediate word sequence; the second transformation layer is concatenated after the first transformation layer.

[0189] Operation S830 processes the second intermediate word sequence based on the similarity between visual words in the second intermediate word sequence to obtain the target visual word sequence.

[0190] In this embodiment, the encoder includes multiple cascaded transform layers. The first visual lexical sequence is a lexical sequence obtained after processing by the first transform layer. The first intermediate lexical sequence is a lexical sequence obtained after compressing the first visual lexical sequence. The second intermediate lexical sequence is a lexical sequence obtained after inputting the first intermediate lexical sequence into the second transform layer.

[0191] This embodiment employs a layer-by-layer compression approach across multiple transform layers of the encoder. After the first transform layer outputs the first visual word sequence, a first compression process is performed to obtain the first intermediate word sequence. The compressed first intermediate word sequence is then input into the second transform layer for processing, and the second transform layer outputs the second intermediate word sequence. A second compression process is then performed on the second intermediate word sequence to obtain the target visual word sequence. Through this layer-by-layer compression method, the number of visual words is gradually reduced during the encoder's processing.

[0192] For example, the encoder may include 12, 24, or other numbers of transform layers. Compression processing may be performed after each transform layer or at intervals of several layers.

[0193] In one feasible implementation, multiple compression processes can be performed in the first few layers of the encoder to gradually compress the number of visual lexical units to the target number. In subsequent layers, no further compression processes are performed, and the compressed lexical unit sequence is used for processing.

[0194] In another feasible implementation, compression operations can be evenly distributed across the layers of the encoder, so that the number of visual lexical units decreases smoothly throughout the encoding process.

[0195] Optionally, compression processing performed on different layers can use the same similarity threshold or different similarity thresholds. When using different thresholds, the threshold size can be adjusted according to the semantic abstraction level of the features at each layer. Shallow feature similarity calculation focuses more on low-level visual features, while deep feature similarity calculation focuses more on high-level semantic features.

[0196] Optionally, the compression ratio can be the same or different for each compression process. In one configuration, a larger compression ratio can be performed at the shallow layer to quickly reduce the number of tokens, while a smaller compression ratio can be performed at the deep layer to retain more semantic information.

[0197] In this embodiment, by performing compression layer by layer across multiple transform layers, the number of visual terms gradually decreases during the encoding process, thus reducing the number of terms processed by subsequent transform layers. Since the computational complexity of the transform layer is positively correlated with the number of terms, reducing the number of terms layer by layer can significantly reduce the computational load of subsequent transform layers. Compared to performing compression only once after the encoder output, the layer-by-layer compression method can reduce the computational burden from the beginning of the encoding process, thereby more effectively improving the overall processing efficiency.

[0198] In this embodiment, layer-by-layer compression and feature extraction at the transform layers work in tandem. Each transform layer performs feature transformation and semantic extraction on the input lexical sequence, outputting a higher-level feature representation. After the transform layer output, compression is performed, lexical merging or filtering is conducted based on the feature similarity at the current level to eliminate redundant information. The compressed lexical sequence is then input into the next transform layer for further feature extraction, and the newly extracted features are evaluated and compressed again in the next compression process. This alternating execution of feature extraction and compression allows the compression operation to adapt to the redundant distribution of features at different levels, achieving more refined redundancy elimination.

[0199] Figure 9 The eighth flowchart illustrating an information processing method provided in an embodiment of this application is shown.

[0200] like Figure 9 As shown, based on the above embodiments, Embodiment Nine may also include the following operations.

[0201] Operation S910 determines the first weight of the merged word in the second transformation layer in the first intermediate word sequence. The merged word is obtained by merging the first visual word sequence based on the similarity between visual words in the first visual word sequence.

[0202] Operate S920 to replace the first weight with the second weight, where the second weight is greater than the first weight.

[0203] In operation S910, the first weight refers to the weight value assigned to the merged lexical by the second transform layer when calculating attention. The merged lexical is a lexical obtained by merging multiple visual lexicals from the first visual lexical sequence.

[0204] It should be noted that after visual word compression, the number of words in the first intermediate word sequence is less than the number of words in the first visual word sequence. The merged words in the first intermediate word sequence integrate the feature information of multiple original visual words. When calculating attention in the second transform layer, if no adjustment is made, merged words and unmerged words will be assigned similar weights, resulting in the actual information represented by the merged words not being reflected in the attention calculation.

[0205] In operation S920, the first weight is replaced with the second weight, so that the merged word unit gets a higher weight in the attention calculation.

[0206] For example, the second weight can be determined based on the number of merged units corresponding to the merged unit, where the number of merged units refers to how many original visual units the merged unit is derived from.

[0207] In one feasible implementation, a scaling variable can be introduced to adjust the attention weights. Specifically, when calculating the attention weights in the second transform layer, the following calculation method is used:

[0208]

[0209] Where A is the attention weight matrix, Q and K are the query matrix and key matrix obtained by linear transformation of the input first intermediate word sequence, respectively, d is the feature dimension of K, and s is the scaling variable vector. The length of the scaling variable vector s is equal to the number of words in the first intermediate word sequence, and each element s in the vector... i This indicates how many original visual lexical units were merged to form the lexical unit at position i in the first intermediate lexical unit sequence. For lexical units that have not been merged, the corresponding s... i The value is 1; for a merged lexical obtained by merging multiple lexical units, the corresponding s i The value equals the number of tokens merged.

[0210] By adding a logs term before the softmax operation in attention weight calculation, the attention score at the corresponding position of the merged word is improved before softmax normalization. Due to the properties of the softmax function, the input score increases by logs. i This will cause the weights at the corresponding positions to be magnified by approximately s after normalization. i Times. Therefore, by s i The weight obtained by merging visual words into a single word group in the attention calculation is approximately s times the original weight. i This multiplies the weight of the merged word elements, thus matching the amount of information they represent.

[0211] Optionally, when performing compression layer by layer in multiple transform layers of the encoder, each transform layer needs to determine the corresponding scaling variable based on the compressed word sequence of the previous layer, and use this scaling variable to adjust when calculating the attention weights. As compression progresses layer by layer, the same merged word may continue to merge with other words, and the corresponding number of merges will accumulate and increase.

[0212] By amplifying the weight of merged lexical units by a factor proportional to the number of merges, the influence of merged lexical units in attention calculation is matched to the amount of information they represent. This adjustment maintains semantic consistency in attention calculation before and after compression, ensuring that compression does not alter the relative importance of different visual features. For example, if two visual lexical units receive attention weights of 0.2 and 0.3 respectively before compression, after merging them, the merged lexical unit should receive an attention weight of approximately 0.5, thus maintaining consistency in information contribution before and after merging.

[0213] It should be noted that the weight adjustment operation is performed during the attention calculation of the second transform layer and does not require modification to other parts of the transform layer. Adaptive weight scaling can be achieved by adding a scaling term to the attention score calculation or by adjusting it during attention weight normalization. This adjustment method has low computational overhead, involving only vector addition and element-level scaling operations, and does not significantly increase computational complexity.

[0214] By employing the above technical solution, the first weight corresponding to the merged word in the first intermediate word sequence in the second transform layer is determined, and this first weight is replaced with a larger second weight, ensuring that the merged word receives a weight in attention calculation that matches the amount of information it represents. By introducing a scaling variable based on the number of merges, the weight of the merged word is amplified during attention weight calculation, with the amplification factor proportional to the number of merges. This weight adjustment method resolves the problem of uneven attention distribution caused by compression processing, maintaining semantic consistency in attention calculation before and after compression.

[0215] Figure 10 The illustration shows a flowchart of a multimodal information processing method provided in an embodiment of this application.

[0216] like Figure 10As shown, the input image (Image) undergoes self-attention processing through the ViT transform layer to obtain visual tokens, which correspond to the first visual token sequence. Inter-modality matching is performed based on the similarity between visual tokens in the first visual token sequence. Visual tokens with high similarity are merged to obtain merged visual tokens, corresponding to the second visual token sequence. The second visual token sequence is then input into a feedforward network (MLP) for feature transformation to obtain semantically encoded visual tokens. Text tokens are then combined with the semantically encoded visual tokens to filter out those highly relevant to the text prompts, resulting in filtered tokens, corresponding to the target visual token sequence. The target visual token sequence and the text token sequence are then input together into a subsequent multimodal processing module for understanding or generation.

[0217] It should be noted that, Figure 10 The process described corresponds to the processing in ViT transform layer N. In an encoder containing multiple transform layers, the above compression process can be repeated in multiple transform layers to reduce the number of visual words layer by layer. After the self-attention processing of each transform layer, the visual word sequence of the current layer is obtained. This sequence is then merged and filtered to obtain a compressed word sequence, which serves as the input for the next layer, thus achieving layer-by-layer compression.

[0218] The above processing flow involves a two-stage compression process on the visual word sequence obtained from image conversion. The first stage merges visual words based on their similarity to eliminate redundant visual information. The second stage filters visual words based on their cross-modal similarity with the text, retaining visual information relevant to the user's query. This two-stage compression significantly reduces the number of visual words and improves the processing efficiency of the multimodal model.

[0219] Please refer to Figure 11 , Figure 11 A block diagram of a processing system provided in an embodiment of this application is shown schematically.

[0220] like Figure 11 As shown in the embodiments, this application also discloses a processing system, including:

[0221] The data acquisition module is used to acquire input information, including the target image.

[0222] The image processing module is used to process the target image to obtain the first visual word sequence;

[0223] The data processing module is used to process the first visual word sequence based on the similarity between visual words in the first visual word sequence to obtain the target visual word sequence; the number of visual words in the target visual word sequence is less than the number of visual words in the first visual word sequence;

[0224] The result generation module is used to generate response results based on the target visual word sequence using the target model.

[0225] According to an embodiment of this application, the data processing module is further configured to merge the first visual word sequence based on the similarity between visual words in the first visual word sequence to obtain a target visual word sequence; and to perform word filtering processing on the first visual word sequence based on the similarity between visual words in the first visual word sequence to obtain a target visual word sequence.

[0226] According to an embodiment of this application, the data processing module is further configured to divide the first visual word sequence into a first word group and a second word group; determine target word pairs, wherein the target word pairs include at least one first word in the first word group and at least one second word in the second word group, wherein the first similarity between the first word and the second word is greater than a first similarity threshold; and merge the target word pairs to obtain a target visual word sequence.

[0227] According to an embodiment of this application, the data processing module is further configured to process the first visual word sequence based on the similarity between visual words in the first visual word sequence to obtain a second visual word sequence; process the prompt text to obtain a text word sequence; and filter the second visual word sequence based on the similarity between the text word sequence and the second visual word sequence to obtain a target visual word sequence.

[0228] According to an embodiment of this application, the data processing module is further configured to calculate a second similarity between each visual word in the second visual word sequence and each text word in the text word sequence; the second similarity characterizes the semantic similarity between the visual word and the text word; based on the second similarity, a target visual word is determined in the second visual word sequence, wherein at least one of the multiple second similarities corresponding to the target visual word is greater than the second similarity threshold; and a target visual word sequence is obtained based on the target visual word.

[0229] According to an embodiment of this application, the data processing module is further configured to obtain a calibration dataset; the calibration dataset includes sample images and sample prompt text; obtain a target compression ratio; and determine a first similarity threshold and a second similarity threshold based on the calibration dataset and the target compression ratio. The first similarity threshold is used to process the first visual word sequence based on the similarity between visual words in the first visual word sequence.

[0230] According to an embodiment of this application, the data processing module is further configured to process the first visual word sequence based on the similarity between visual words in the first visual word sequence to obtain a first intermediate word sequence; input the first intermediate word sequence into a second transformation layer to obtain a second intermediate word sequence; the second transformation layer is concatenated after the first transformation layer; and the second intermediate word sequence is processed based on the similarity between visual words in the second intermediate word sequence to obtain a target visual word sequence.

[0231] According to an embodiment of this application, the data processing module is further configured to determine the first weight corresponding to the merged word in the first intermediate word sequence in the second transformation layer. The merged word is obtained by: merging the first visual word sequence based on the similarity between visual words in the first visual word sequence; and replacing the first weight with a second weight, wherein the second weight is greater than the first weight.

[0232] According to an embodiment of this application, the data processing module is further configured to perform feature transformation on the second visual word sequence using a feedforward network to obtain a third visual word sequence; the third visual word sequence is a semantically encoded visual word sequence; and the third visual word sequence is filtered based on the text word sequence to obtain a target visual word sequence.

[0233] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this application is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this application, and all such substitutions and modifications should fall within the scope of this application.

Claims

1. An information processing method, comprising: Obtain input information, including the target image; The target image is processed to obtain a first visual word sequence; Based on the similarity between visual words in the first visual word sequence, the first visual word sequence is processed to obtain a target visual word sequence; the number of visual words in the target visual word sequence is less than the number of visual words in the first visual word sequence. The target model generates the response result of the input information based on the target visual lexical sequence.

2. The method according to claim 1, wherein processing the first visual word sequence based on the similarity between visual words in the first visual word sequence to obtain the target visual word sequence includes at least one of the following: Based on the similarity between visual words in the first visual word sequence, the first visual word sequence is merged to obtain the target visual word sequence. Based on the similarity between visual words in the first visual word sequence, word filtering is performed on the first visual word sequence to obtain the target visual word sequence.

3. The method according to claim 2, wherein merging the first visual word sequence based on the similarity between visual words in the first visual word sequence to obtain the target visual word sequence includes: The first visual word sequence is divided into a first word group and a second word group; Determine a target word pair, wherein the target word pair includes at least one first word in the first word group and at least one second word in the second word group, and the first similarity between the first word and the second word is greater than a first similarity threshold; The target word pairs are merged to obtain the target visual word sequence.

4. The method according to claim 1, wherein the input information further includes prompt text; The step of processing the first visual word sequence based on the similarity between visual words in the first visual word sequence to obtain the target visual word sequence includes: Based on the similarity between visual words in the first visual word sequence, the first visual word sequence is processed to obtain the second visual word sequence. The prompt text is processed to obtain a text word sequence; Based on the similarity between the text lexical sequence and the second visual lexical sequence, the second visual lexical sequence is filtered to obtain the target visual lexical sequence.

5. The method according to claim 4, wherein the second visual word sequence is filtered based on the similarity between the text word sequence and the second visual word sequence to obtain the target visual word sequence, comprising: Calculate the second similarity between each visual word in the second visual word sequence and each text word in the text word sequence; The second similarity represents the semantic similarity between visual lexical units and textual lexical units; Based on the second similarity, a target visual word is determined in the second visual word sequence, wherein at least one of the multiple second similarities corresponding to the target visual word is greater than the second similarity threshold; A target visual word sequence is obtained based on the target visual word.

6. The method according to claim 5, further comprising: Obtain the calibration dataset; The calibration dataset includes sample images and sample prompt text; Obtain the target compression ratio; A first similarity threshold and a second similarity threshold are determined based on the calibration dataset and the target compression ratio. The first similarity threshold is used to process the first visual word sequence based on the similarity between visual words in the first visual word sequence.

7. The method according to claim 1, wherein the first visual word sequence is obtained through a first transform layer, the first transform layer being one of multiple transform layers cascaded in the encoder, and the step of processing the first visual word sequence based on the similarity between visual words in the first visual word sequence to obtain a target visual word sequence includes: The first visual word sequence is processed based on the similarity between visual words in the first visual word sequence to obtain the first intermediate word sequence. The first intermediate word sequence is input into the second transformation layer to obtain the second intermediate word sequence; the second transformation layer is concatenated after the first transformation layer; Based on the similarity between visual lexical units in the second intermediate lexical unit sequence, the second intermediate lexical unit sequence is processed to obtain the target visual lexical unit sequence.

8. The method according to claim 7, further comprising: Determine the first weight of the merged word in the first intermediate word sequence in the second transformation layer. The merged word is obtained by merging the first visual word sequence based on the similarity between visual words in the first visual word sequence. The first weight is replaced with a second weight, where the second weight is greater than the first weight.

9. The method according to claim 4, wherein the step of filtering the second visual lexical sequence based on the text lexical sequence to obtain the target visual lexical sequence includes: The second visual word sequence is transformed using a feedforward network to obtain the third visual word sequence; The third visual lexical sequence is a semantically encoded visual lexical sequence; The target visual word sequence is obtained by filtering the third visual word sequence based on the text word sequence.

10. An information processing apparatus, comprising: The data acquisition module is used to acquire input information, including the target image. The image processing module is used to process the target image to obtain a first visual word sequence; The data processing module is used to process the first visual word sequence based on the similarity between visual words in the first visual word sequence to obtain a target visual word sequence; the number of visual words in the target visual word sequence is less than the number of visual words in the first visual word sequence. The result generation module is used to generate a response result of the input information based on the target visual word sequence using the target model.

Images