Image processing method and related device

Abstract

The application discloses an image processing method and related equipment. The method can perform feature extraction processing on a to-be-processed image to obtain an image feature sequence; perform processing on an image feature block in the image feature sequence through a network layer to obtain a target image feature sequence; perform classification processing on the processed image feature block in the target image feature sequence to obtain a to-be-deleted feature sequence and a to-be-kept feature sequence, wherein the to-be-deleted feature sequence includes at least one to-be-deleted image feature block, and the to-be-kept feature sequence includes at least one to-be-kept image feature block; aggregate the features of the to-be-deleted image feature block into the to-be-kept feature sequence to obtain an aggregated feature sequence; and obtain an image processing task result according to the aggregated feature sequence. The application can reduce the calculation overhead, improve the content reasoning efficiency, and ensure the accuracy of the content reasoning.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to an image processing method and related equipment. Background Technology

[0002] With the continuous advancement of computer technology, multimodal large language models (MLLMs) with powerful reasoning capabilities have emerged. Specifically, based on the reasoning capabilities of MLLMs, computer devices can perform image understanding and visual question answering by calling upon the massive model parameters within the MLLM, performing reasoning analysis on image content, and obtaining corresponding content reasoning results.

[0003] While multimodal large language models excel in tasks such as image understanding and visual question answering, excessive visual tagging input can lead to excessive computational overhead, thus impacting inference efficiency. In some scenarios, using high-resolution images or video input further increases the number of visual tags, resulting in a significant decrease in content inference efficiency. Summary of the Invention

[0004] This application provides an image processing method and related equipment. The related equipment may include an image processing device, an electronic device, a computer-readable storage medium, and a computer program product, which can reduce computational overhead and improve content reasoning efficiency; at the same time, it ensures the accuracy of content reasoning.

[0005] This application provides an image processing method, including:

[0006] A determination is made of the image to be processed; and feature extraction processing is performed on the image to be processed to obtain an image feature sequence, wherein the image feature sequence includes multiple image feature blocks of the image to be processed;

[0007] The image feature blocks in the image feature sequence are processed by a network layer to obtain a target image feature sequence, which includes multiple processed image feature blocks.

[0008] The processed image feature blocks in the target image feature sequence are classified to obtain a feature sequence to be deleted and a feature sequence to be retained. The feature sequence to be deleted includes at least one image feature block to be deleted, and the feature sequence to be retained includes at least one image feature block to be retained.

[0009] The features of the image feature blocks to be deleted are aggregated into the feature sequence to be retained to obtain the aggregated feature sequence, which includes the aggregated image feature blocks corresponding to each image feature block to be retained.

[0010] Based on the aggregated feature sequence, the image processing task result of the image to be processed is obtained.

[0011] Accordingly, embodiments of this application provide an image processing apparatus, including:

[0012] A feature extraction unit is used to determine the image to be processed and to perform feature extraction processing on the image to be processed to obtain an image feature sequence, wherein the image feature sequence includes multiple image feature blocks of the image to be processed.

[0013] The processing unit is configured to process the image feature blocks in the image feature sequence through a network layer to obtain a target image feature sequence, wherein the target image feature sequence includes multiple processed image feature blocks;

[0014] A classification unit is used to classify the processed image feature blocks in the target image feature sequence to obtain a feature sequence to be deleted and a feature sequence to be retained. The feature sequence to be deleted includes at least one image feature block to be deleted, and the feature sequence to be retained includes at least one image feature block to be retained.

[0015] The feature aggregation unit is used to aggregate the features of the image feature blocks to be deleted into the feature sequence to be retained, so as to obtain the aggregated feature sequence. The aggregated feature sequence includes the aggregated image feature blocks corresponding to each image feature block to be retained.

[0016] The result determination unit is used to obtain the image processing task result of the image to be processed based on the aggregated feature sequence.

[0017] Optionally, in some embodiments of this application, the image processing apparatus may further include a text feature extraction unit and a sequence fusion unit, as follows:

[0018] The text feature extraction unit is used to obtain task instruction text for the image to be processed; and to extract features from the task instruction text to obtain a text feature sequence, wherein the text feature sequence includes text feature blocks corresponding to multiple text units in the task instruction text;

[0019] A sequence fusion unit is used to fuse the image feature sequence and the text feature sequence to obtain a feature sequence;

[0020] Specifically, the processing unit can be used to perform attention interaction processing on each feature unit in the feature sequence through a network layer to obtain a processed feature sequence. The processed feature sequence includes a target image feature sequence and a target text feature sequence. The target text feature sequence includes processed text feature blocks corresponding to the plurality of text units. The feature unit is an image feature block or a text feature block.

[0021] The classification unit can be used to classify the processed image feature blocks in the target image feature sequence based on the processed text feature blocks, so as to obtain the feature sequence to be deleted and the feature sequence to be retained.

[0022] Optionally, in some embodiments of this application, the feature aggregation unit may be used to aggregate the features of the image feature block to be deleted into each image feature block to be retained in the feature sequence to be retained, so as to obtain the aggregated feature sequence.

[0023] Optionally, in some embodiments of this application, the feature aggregation unit may include a matrix construction subunit and a feature aggregation subunit, as follows:

[0024] The matrix construction subunit is used to construct a feature relationship matrix based on the semantic similarity between the image feature blocks to be deleted and the image feature blocks to be retained. The elements in the feature relationship matrix represent the association information between the image feature blocks to be deleted and the image feature blocks to be retained at the corresponding positions.

[0025] The feature aggregation subunit is used to aggregate the image feature blocks to be deleted into each image feature block to be retained based on the feature relationship matrix, so as to obtain the aggregated feature sequence.

[0026] Optionally, in some embodiments of this application, the feature aggregation subunit may be specifically used to fuse the feature relationship matrix with the feature sequence to be deleted to obtain a fused feature sequence; and to fuse the fused feature sequence with each image feature block to be retained in the feature sequence to be retained to obtain an aggregated feature sequence.

[0027] Optionally, in some embodiments of this application, the matrix construction subunit can be specifically used to construct a feature relationship graph based on the semantic similarity between every two processed image feature blocks in the target image feature sequence. The feature relationship graph includes multiple nodes and edges between nodes. Each node represents a processed image feature block, and the edge represents the association information between the two nodes connected to it. Based on the association information between the nodes corresponding to the image feature blocks to be deleted and the image feature blocks to be retained in the feature relationship graph, a feature relationship matrix corresponding to the image feature blocks to be deleted and the image feature blocks to be retained is constructed.

[0028] Optionally, in some embodiments of this application, the step "constructing a feature relationship matrix corresponding to the image feature blocks to be deleted and the image feature blocks to be retained based on the association information between the nodes corresponding to the image feature blocks to be deleted and the image feature blocks to be retained in the feature relationship graph" may include:

[0029] Based on the connection relationships between nodes in the feature relationship graph, the degree of each node in the feature relationship graph is determined.

[0030] Based on the degree, the adjacency matrix corresponding to the feature relationship graph is normalized to obtain a processed adjacency matrix. The elements in the processed adjacency matrix represent the association information between two image feature blocks at corresponding positions.

[0031] The elements corresponding to the image feature blocks to be deleted and the image feature blocks to be retained in the processed adjacency matrix are filtered to obtain the feature relationship matrix corresponding to the image feature blocks to be deleted and the image feature blocks to be retained.

[0032] Optionally, in some embodiments of this application, the processing unit may be specifically used to perform multi-layer attention interaction processing on each feature unit in the feature sequence through multiple network layers to obtain the processed feature sequence output by each layer of attention interaction processing, wherein the output of the attention interaction processing of the previous network layer is used as the input of the attention interaction processing of the next network layer.

[0033] The classification unit may include a determining subunit and a classification subunit, as follows:

[0034] The determining subunit is used to determine, from the processed feature sequences output by the attention interaction processing of the plurality of network layers, at least one target processed feature sequence output by the attention interaction processing of the target network layer.

[0035] The classification subunit is used to classify the processed image feature blocks in the target image feature sequence based on the processed text feature blocks in the at least one target processed feature sequence, to obtain the feature sequence to be deleted and the feature sequence to be retained.

[0036] Optionally, in some embodiments of this application, the classification subunit may be specifically used to determine the attention information between the processed text feature blocks in each target processed feature sequence and the processed image feature blocks for each processed image feature block in the target image feature sequence; to fuse the attention information between the processed text feature blocks in all target processed feature sequences and the processed image feature blocks to obtain the target attention information of the processed image feature blocks; and to classify the processed image feature blocks in the target image feature sequence based on the target attention information to obtain the feature sequence to be deleted and the feature sequence to be retained.

[0037] Optionally, in some embodiments of this application, the target processed feature sequence output by the attention interaction processing of the target network layer includes the processed sub-feature sequences output by the target network layer in multiple subspaces, and the processed sub-feature sequences include processed text feature blocks in the corresponding subspaces;

[0038] The step "fusing the attention information between the processed text feature blocks and the processed image feature blocks in all target-processed feature sequences to obtain the target attention information of the processed image feature blocks" may include:

[0039] For each target network layer, the attention information of the processed text feature blocks output by the target network layer in each subspace and the processed image feature blocks are fused to obtain the attention information of the processed text feature blocks and the processed image feature blocks of the target network layer.

[0040] The attention information of the processed text feature blocks of all target network layers is fused with the attention information of the processed image feature blocks to obtain the target attention information of the processed image feature blocks.

[0041] Optionally, in some embodiments of this application, the processing unit may be specifically used to perform attention interaction processing on each feature unit in the feature sequence through the network layer in the attention module in the current processing module group of the processing model to obtain the processed feature sequence. The processing model includes multiple processing module groups, and each processing module group includes an attention module, a feature selection module, and a feature aggregation module.

[0042] The classification unit can be specifically used to classify the processed image feature blocks in the target image feature sequence based on the processed text feature blocks through the feature selection module in the current processing module group, so as to obtain the feature sequence to be deleted and the feature sequence to be retained;

[0043] Specifically, the feature aggregation unit can be used to aggregate the features of the image feature block to be deleted into the feature sequence to be retained through the feature aggregation module in the current processing module group, so as to obtain the aggregated feature sequence;

[0044] Specifically, the result determination unit can be used to take the fused feature sequence of the aggregated feature sequence and the target text feature sequence as the input of the next processing module group of the current processing module group, so as to continue the content reasoning processing and obtain the image processing task result of the image to be processed.

[0045] Optionally, in some embodiments of this application, the image processing apparatus may further include a model acquisition unit, a grouping unit, and a module adding unit, as follows:

[0046] The model acquisition unit is used to acquire the original processing model, which includes multiple attention modules;

[0047] A grouping unit is used to group multiple attention modules in the original processing model to obtain multiple processing module groups, each processing module group including at least one attention module.

[0048] The module adding unit is used to add feature selection modules and feature aggregation modules to each processing module group to obtain the processing model.

[0049] Optionally, in some embodiments of this application, the grouping unit may be specifically used to determine the number of groups for the original processing model based on computational requirement indication information; and to group multiple attention modules in the original processing model according to the number of groups to obtain multiple processing module groups.

[0050] Optionally, in some embodiments of this application, the result determination unit may specifically include a fusion subunit, a module group determination subunit, and an inference subunit, as follows:

[0051] The fusion subunit is used to fuse the aggregated feature sequence with the target text feature sequence to obtain the fused feature sequence.

[0052] The module group determines the sub-unit, which is used to take the next processing module group of the current processing module group as the new current processing module group and the fused feature sequence as the new feature sequence;

[0053] The inference subunit is used to return to the attention module in the current processing module group that executes the processing model, and to perform attention interaction processing on each feature unit in the feature sequence to obtain the processed feature sequence, so as to obtain the image processing task result of the image to be processed.

[0054] Optionally, in some embodiments of this application, the inference subunit may specifically be used to return to the execution of the attention module in the current processing module group of the processing model, performing attention interaction processing on each feature unit in the feature sequence, until the aggregated feature sequence and target text feature sequence output by the last processing module group in the processing model are obtained; based on the aggregated feature sequence and target text feature sequence output by the last processing module group in the processing model, the target inference result unit for this inference is predicted; based on the aggregated feature sequence and target text feature sequence output by each processing module group of the processing model, the content feature sequence corresponding to each processing module group of the processing model is determined respectively; through the attention module in the processing module group of the processing model, based on the content feature sequence corresponding to the processing module group, attention processing is performed on the target inference result unit to predict the subsequent inference result unit of the target inference result unit; based on the target inference result unit and the subsequent inference result unit, the image processing task result of the image to be processed is determined.

[0055] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0056] An electronic device provided in this application includes a processor and a memory. The memory stores multiple instructions, and the processor loads the instructions to execute the steps in the image processing method provided in this application.

[0057] This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps in the image processing method provided in this application.

[0058] Furthermore, embodiments of this application also provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the image processing method provided in embodiments of this application.

[0059] This application provides an image processing method and related equipment, which can determine an image to be processed; perform feature extraction processing on the image to be processed to obtain an image feature sequence, the image feature sequence including multiple image feature blocks of the image to be processed; process the image feature blocks in the image feature sequence through a network layer to obtain a target image feature sequence, the target image feature sequence including multiple processed image feature blocks; classify the processed image feature blocks in the target image feature sequence to obtain a feature sequence to be deleted and a feature sequence to be retained, the feature sequence to be deleted including at least one image feature block to be deleted, the feature sequence to be retained including at least one image feature block to be retained; aggregate the features of the image feature blocks to be deleted into the feature sequence to be retained to obtain an aggregated feature sequence, the aggregated feature sequence including the aggregated image feature blocks corresponding to each image feature block to be retained; and obtain the image processing task result of the image to be processed based on the aggregated feature sequence.

[0060] This application can reduce computational overhead and improve content reasoning efficiency by deleting some image feature blocks; at the same time, it can also aggregate the deleted image feature blocks into the retained image feature blocks to avoid the loss of visual information caused by deleting image feature blocks and ensure the accuracy of content reasoning. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1a This is a schematic diagram of a scene illustrating the image processing method provided in an embodiment of this application;

[0063] Figure 1b This is a flowchart of the image processing method provided in the embodiments of this application;

[0064] Figure 1c This is a model architecture diagram of the image processing method provided in the embodiments of this application;

[0065] Figure 1d These are illustrative diagrams illustrating the image processing method provided in the embodiments of this application;

[0066] Figure 1e This is another illustrative diagram of the image processing method provided in the embodiments of this application;

[0067] Figure 1f This is another illustrative diagram of the image processing method provided in the embodiments of this application;

[0068] Figure 1g This is another illustrative diagram of the image processing method provided in the embodiments of this application;

[0069] Figure 1h This is another illustrative diagram of the image processing method provided in the embodiments of this application;

[0070] Figure 1i This is another illustrative diagram of the image processing method provided in the embodiments of this application;

[0071] Figure 2 This is another flowchart of the image processing method provided in the embodiments of this application;

[0072] Figure 3 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application;

[0073] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0074] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0075] This application provides an image processing method and related equipment. The related equipment may include an image processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Specifically, the image processing apparatus may be integrated into an electronic device, which may be a terminal or a server, etc.

[0076] It is understood that the image processing method of this embodiment can be executed on a terminal, on a server, or jointly by a terminal and a server. The above examples should not be construed as limiting this application.

[0077] like Figure 1a As shown, an image processing method is executed jointly by a terminal and a server, as an example. The image processing system provided in this application includes a terminal 10 and a server 11, etc.; the terminal 10 and the server 11 are connected via a network, such as a wired or wireless network, etc., wherein the image processing device can be integrated into the server.

[0078] Server 11 can be used to: determine an image to be processed; perform feature extraction processing on the image to be processed to obtain an image feature sequence, the image feature sequence including multiple image feature blocks of the image to be processed; process the image feature blocks in the image feature sequence through a network layer to obtain a target image feature sequence, the target image feature sequence including multiple processed image feature blocks; classify the processed image feature blocks in the target image feature sequence to obtain a feature sequence to be deleted and a feature sequence to be retained, the feature sequence to be deleted including at least one image feature block to be deleted, the feature sequence to be retained including at least one image feature block to be retained; aggregate the features of the image feature blocks to be deleted into the feature sequence to be retained to obtain an aggregated feature sequence, the aggregated feature sequence including the aggregated image feature blocks corresponding to each image feature block to be retained; and obtain the image processing task result of the image to be processed based on the aggregated feature sequence. Server 11 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0079] Terminal 10 can be used to: send an image to be processed to server 11, so that server 11 can perform content inference on the image; terminal 10 can also receive the image processing task results of the image to be processed sent by server 11. Terminal 10 can include mobile phones, vehicle terminals, aircraft, tablet computers, laptops, or personal computers (PCs), etc. A client can also be set on terminal 10, which can be an application client or a browser client, etc.

[0080] The image processing and other steps in the aforementioned server 11 can also be performed by the terminal 10.

[0081] The image processing method provided in this application relates to natural language processing and computer vision technologies in the field of artificial intelligence.

[0082] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0083] This embodiment will be described from the perspective of an image processing device, which can be integrated into an electronic device, such as a server or a terminal.

[0084] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0085] like Figure 1b As shown, the specific process of this image processing method can be as follows:

[0086] 101. Determine the image to be processed; and perform feature extraction processing on the image to be processed to obtain an image feature sequence, wherein the image feature sequence includes multiple image feature blocks of the image to be processed.

[0087] Among them, the image to be processed is the image whose content needs to be reasoned. The reasoning here can be image understanding tasks such as image classification and visual question answering. Specifically, it can be achieved by analyzing and interpreting the semantic information in the image through computer vision technology, thereby realizing the accurate identification and understanding of the image content.

[0088] In this embodiment, the feature extraction process of the image to be processed can be specifically performed by a visual encoder to extract the features of the image to be processed, obtain image coded features, and then convert the extracted image coded features into multiple visual tags.

[0089] In some embodiments, the image coding features can be reduced in dimensionality using a projector to obtain processed image coding features. These processed features are then subjected to feature segmentation to obtain multiple visual labels. Specifically, the projector can be a neural network module that maps high-dimensional features extracted from an image to a low-dimensional space; this process can also be called feature projection.

[0090] In this embodiment, the image feature sequence is specifically a visual token sequence, which includes multiple visual tokens, each of which is an image feature block. Visual tokens specifically refer to the image feature representation of each small block after the image is divided into multiple small blocks.

[0091] Specifically, the image-coded features are converted into multiple visual tags. This can be achieved by decomposing or segmenting the image-coded features into smaller units through feature partitioning, resulting in multiple feature blocks. Each feature block contains a portion of the information from the image-coded features, and each feature block is equivalent to a visual tag. There are various feature partitioning methods, and this embodiment does not limit them.

[0092] For example, feature partitioning can be performed directly based on rules of fixed size. For instance, if the feature size of an image encoding feature is 16*16*C (where C is the number of feature channels), it can be divided into 16*16=256 feature blocks, with each feature block having a size of 1*1*C.

[0093] For example, feature division can also be based on feature type or importance. Specifically, the importance of each channel feature in the image coding feature can be evaluated first. For channel features with higher importance, the division granularity can be finer; for channel features with relatively lower importance, the division granularity can be coarser; that is, channel features with higher importance can be divided into more feature blocks. Understandably, the measurement of feature importance can be determined according to the actual situation.

[0094] For example, feature partitioning can be performed based on statistical characteristics. Specifically, these statistical characteristics can include mean, variance, etc. For instance, the global mean and variance of each channel feature map of an image encoding feature can be calculated. For channel feature maps with smaller means and variances, the granularity of feature partitioning can be coarser because these features exhibit relatively small spatial variations. For channel feature maps with larger means and variances, the granularity of feature partitioning can be finer, dividing these channel feature maps into more feature blocks. Understandably, the magnitude of the mean and variance can be determined based on the specific circumstances.

[0095] 102. The image feature blocks in the image feature sequence are processed by the network layer to obtain the target image feature sequence, which includes multiple processed image feature blocks.

[0096] Optionally, in this embodiment, before the step "processing the image feature blocks in the image feature sequence through a network layer to obtain the target image feature sequence", the following may be included:

[0097] Obtain the task instruction text for the image to be processed; and perform feature extraction on the task instruction text to obtain a text feature sequence, wherein the text feature sequence includes text feature blocks corresponding to multiple text units in the task instruction text;

[0098] The image feature sequence and the text feature sequence are fused to obtain a feature sequence;

[0099] The step "processing the image feature blocks in the image feature sequence through network layers to obtain the target image feature sequence" may include:

[0100] The network layer performs attention interaction processing on each feature unit in the feature sequence to obtain a processed feature sequence. The processed feature sequence includes a target image feature sequence and a target text feature sequence. The target text feature sequence includes processed text feature blocks corresponding to the plurality of text units. The feature unit is an image feature block or a text feature block.

[0101] Specifically, the task instruction text can be used to provide guidance information to the model for performing image content extraction. Specifically, the task instruction text provides the model with a task description or contextual clues for image content analysis and image classification within a pre-defined business scenario. This can include, but is not limited to, task summaries, problem descriptions, or mapping relationships between inputs and expected outputs. Through the task instruction text, the accuracy and relevance of the model in solving relevant tasks under pre-defined business scenarios can be improved, thereby enhancing the reliability of the output results. For example, in a specific content review business scenario, the task instruction text could be "Describe this image and analyze whether it contains specific content."

[0102] Specifically, feature extraction of the task instruction text can be achieved by splitting the task instruction text into segments using a tokenizer, mapping them to corresponding indices in a vocabulary, and then obtaining the text embedding features corresponding to each segment through the indices, thereby obtaining a text feature sequence. The text feature sequence includes text feature blocks corresponding to multiple text units in the task instruction text, with each text unit corresponding to a segment, and the text feature block corresponding to the text unit being the text token.

[0103] In this embodiment, the image feature sequence and the text feature sequence are fused. Specifically, the image feature sequence and the text feature sequence are concatenated. The concatenation order is not limited. The image feature sequence can be concatenated after the text feature sequence, or the text feature sequence can be concatenated after the image feature sequence.

[0104] Specifically, the attention mechanism can be used to directly capture the dependency between any two positions in the feature sequence, thereby efficiently capturing long-distance dependencies and modeling the interaction between elements in the sequence.

[0105] The attention interaction processing for each feature unit in the feature sequence can be performed as follows: Each feature unit is linearly transformed to generate three vectors: query, key, and value. The query and key vectors are then multiplied by a dot product and normalized using a scaling factor to obtain an attention score. The attention score is then normalized using a softmax activation function to obtain the attention weight for each position. Finally, the value vectors are weighted and summed according to the attention weights to obtain the context representation for each position, i.e., the interactive attention feature. The processed image feature block and the processed text feature block are the resulting interactive attention features.

[0106] 103. Classify the processed image feature blocks in the target image feature sequence to obtain a feature sequence to be deleted and a feature sequence to be retained. The feature sequence to be deleted includes at least one image feature block to be deleted, and the feature sequence to be retained includes at least one image feature block to be retained.

[0107] Optionally, in this embodiment, the step "classifying the processed image feature blocks in the target image feature sequence to obtain the feature sequence to be deleted and the feature sequence to be retained" may include:

[0108] Based on the processed text feature blocks, the processed image feature blocks in the target image feature sequence are classified to obtain the feature sequence to be deleted and the feature sequence to be retained.

[0109] In this embodiment, the attention score of text tags to visual tags can be used to segment visual tags. The processed text feature blocks are also text tags, and the processed image feature blocks in the target image feature sequence are also visual tags. The attention score can serve as an indicator of the importance of visual tags, and based on this indicator, visual tags to be deleted and visual tags to be retained can be determined.

[0110] Specifically, the target text feature sequence includes multiple processed text feature blocks. The processed image feature blocks in the target image feature sequence can be classified based on the attention score of the last processed text feature block in the target text feature sequence to the processed image feature blocks.

[0111] Optionally, in this embodiment, the step "performing attention interaction processing on each feature unit in the feature sequence through a network layer to obtain a processed feature sequence" may include:

[0112] The feature sequence is processed by multi-layer attention interaction by multiple network layers to obtain the processed feature sequence output by each layer of attention interaction. The output of the attention interaction of the previous network layer is used as the input of the attention interaction of the next network layer.

[0113] The step "based on the processed text feature blocks, classify the processed image feature blocks in the target image feature sequence to obtain the feature sequence to be deleted and the feature sequence to be retained" may include:

[0114] From the processed feature sequences output by the attention interaction processing of the multiple network layers, determine the target processed feature sequence output by the attention interaction processing of at least one target network layer;

[0115] Based on the processed text feature blocks in the at least one target processed feature sequence, the processed image feature blocks in the target image feature sequence are classified to obtain a feature sequence to be deleted and a feature sequence to be retained.

[0116] Among them, the network layer can be the attention layer.

[0117] Specifically, this can be achieved by using multiple consecutive attention modules to perform attention interaction processing on the feature units in the feature sequence. Each attention module's processing corresponds to one layer of attention interaction processing. The feature sequence is obtained by concatenating the text feature sequence and the image feature sequence. Through each layer of attention interaction processing, the tokens in the text feature sequence and the image feature sequence can be updated. The processed feature sequence output by each layer includes the corresponding processed text feature sequence and the corresponding processed image feature sequence.

[0118] Specifically, the target network layer is the layer selected to evaluate the importance of feature blocks in the processed image; the target network layer can be any layer. The processed feature sequence output by the attention interaction processing of the target network layer is the target processed feature sequence.

[0119] For example, before classifying the processed image feature blocks, there are S attention layers that perform attention interaction processing on the feature units in the feature sequence. In this embodiment, the attention scores of multiple attention layers can be used to evaluate the importance of the processed image feature blocks in the target image feature sequence, and then perform classification. For example, the last L layers can be selected as the target network layers, that is, the target network layers are layers S-L+1 to S. Based on the attention scores of the processed text feature blocks and processed image feature blocks in the processed feature sequence output by these L layers, the importance of the processed image feature blocks in the target image feature sequence is evaluated. It is understood that in some embodiments, the attention scores of all layers can be selected to evaluate the importance of the image feature blocks.

[0120] Specifically, the target image feature sequence here is the image feature sequence output from the last layer in the multi-layer attention interaction processing.

[0121] Optionally, in this embodiment, the step "based on the processed text feature blocks in the at least one target processed feature sequence, classify the processed image feature blocks in the target image feature sequence to obtain a feature sequence to be deleted and a feature sequence to be retained" may include:

[0122] For each processed image feature block in the target image feature sequence, determine the attention information between the processed text feature block in each target processed feature sequence and the processed image feature block;

[0123] The attention information between the processed text feature blocks and the processed image feature blocks in all target processed feature sequences is fused to obtain the target attention information of the processed image feature blocks.

[0124] Based on the target attention information, the processed image feature blocks in the target image feature sequence are classified to obtain the feature sequence to be deleted and the feature sequence to be retained.

[0125] Here, the attention information specifically refers to the attention score. In this embodiment, the attention information of each target network layer on the visual markers can be fused to obtain the target attention information of the visual markers. There are various fusion methods, such as averaging or weighted averaging, and this embodiment does not limit the method.

[0126] Specifically, the processed text feature block in the target processed feature sequence can be a processed text feature block at a certain position in the target processed feature sequence. In a specific embodiment, the importance of the processed image feature block can be evaluated using the attention information between the last processed text feature block in the target processed feature sequence obtained from each target network layer and the processed image feature block in the target image feature sequence.

[0127] In some embodiments, processed image feature blocks with target attention information greater than a preset value can be identified as image feature blocks to be retained, while processed image feature blocks with target attention information less than the preset value can be identified as image feature blocks to be deleted. In other embodiments, processed image feature blocks in the target image feature sequence can be sorted according to the magnitude of the target attention information, such as sorting from largest to smallest, and the top n processed image feature blocks in the sorted result can be identified as image feature blocks to be retained, while the other processed image feature blocks are identified as image feature blocks to be deleted.

[0128] Optionally, in this embodiment, the target processed feature sequence output by the attention interaction processing of the target network layer includes the processed sub-feature sequences output by the target network layer in multiple subspaces, and the processed sub-feature sequences include the processed text feature blocks of the corresponding subspaces;

[0129] The step "fusing the attention information between the processed text feature blocks and the processed image feature blocks in all target-processed feature sequences to obtain the target attention information of the processed image feature blocks" may include:

[0130] For each target network layer, the attention information of the processed text feature blocks output by the target network layer in each subspace and the processed image feature blocks are fused to obtain the attention information of the processed text feature blocks and the processed image feature blocks of the target network layer.

[0131] The attention information of the processed text feature blocks of all target network layers is fused with the attention information of the processed image feature blocks to obtain the target attention information of the processed image feature blocks.

[0132] The two attention information fusion methods can be weighted operations, etc., and this embodiment does not limit them.

[0133] Optionally, the attention mechanism used in this embodiment can be a multi-head attention mechanism. A multi-head attention mechanism segments the input features into multiple "heads" and processes each head independently, i.e., runs multiple independent attention mechanisms in parallel, to improve the model's expressive and learning capabilities. It can capture various dependencies in the input sequence from different subspaces, thus overcoming the limitations of self-attention mechanisms when handling complex tasks. Specifically, each "head" independently processes the Query, Key, and Value matrices. After scaling dot product attention operations, the outputs of all heads are concatenated and fused through a linear layer to ultimately generate a more comprehensive representation.

[0134] In this embodiment, each subspace corresponds to a "header". It can be understood that the processed sub-feature sequence output by each subspace includes the processed text feature sequence and the processed image feature sequence.

[0135] In specific scenarios, the average attention scores of the last L attention layers can be used to guide the division of image feature blocks to be deleted and image feature blocks to be retained. Specifically, considering that the focal region of the image is determined based on the task instruction text, the importance of each visual marker can be evaluated based on the attention of the text to the visual marker. Since the newly generated text markers are predicted based on the feature information of the last marker (i.e., the last text marker) in the current model input sequence, this embodiment uses the attention score of the last text marker to the visual marker as an indicator of the importance of the visual marker, denoted as I, as shown in equation (1):

[0136]

[0137] in, Let I represent the attention score of the last text tag to the visual tag in the j-th head of the multi-head attention mechanism at layer i, and let I represent the target attention information in the above embodiment. This embodiment can average the attention scores of all heads H to ensure that information from all subspaces is utilized.

[0138] After obtaining I, the target image feature sequence x can be determined based on the metric I. vis Divided into feature sequences to be retained and the feature sequence to be deleted As shown in equations (2) and (3):

[0139]

[0140]

[0141] Here, topk(·) represents the visual marker with the highest importance index I among the selected proportions p. The visual markers to be deleted. It is the total visual marker x vis and visual markers to be retained The difference between them.

[0142] For example, p can be 0.5, if the total visual marker x vis If there are 400, then the visual markers to be retained Then there are 200.

[0143] 104. Aggregate the features of the image feature blocks to be deleted into the feature sequence to be retained to obtain an aggregated feature sequence, wherein the aggregated feature sequence includes the aggregated image feature blocks corresponding to each image feature block to be retained.

[0144] Feature aggregation can be used to delete feature sequences while ensuring the accuracy of image processing.

[0145] The number of image feature blocks contained in the feature sequence to be retained and the aggregated feature sequence is the same. Specifically, the feature dimension of the aggregated image feature block and the corresponding image feature block to be retained is the same.

[0146] Optionally, in this embodiment, the step "aggregating the features of the image feature blocks to be deleted into the feature sequence to be retained, to obtain the aggregated feature sequence" may include:

[0147] The features of the image feature blocks to be deleted are aggregated into the individual image feature blocks to be retained in the feature sequence to be retained, resulting in the aggregated feature sequence.

[0148] Optionally, in this embodiment, the step "aggregating the features of the image feature blocks to be deleted into the feature sequence to be retained, to obtain the aggregated feature sequence" may include:

[0149] Based on the semantic similarity between the image feature blocks to be deleted and the image feature blocks to be retained, a feature relationship matrix is ​​constructed. The elements in the feature relationship matrix represent the association information between the image feature blocks to be deleted and the image feature blocks to be retained at the corresponding positions.

[0150] Based on the feature relationship matrix, the image feature blocks to be deleted are aggregated into each image feature block to be retained, resulting in an aggregated feature sequence.

[0151] Specifically, the row and column indices of the feature relation matrix correspond to the visual tags to be retained and the visual tags to be deleted, respectively. In other words, each row of the feature relation matrix represents the correlation between a visual tag to be retained and all visual tags to be deleted. Thus, through the feature relation matrix, information about each visual tag to be deleted can be propagated to multiple different visual tags to be retained. It is understood that in some embodiments, the row index may also correspond to the visual tag to be deleted, and the column index may also correspond to the visual tag to be retained.

[0152] The association information of the element representation can be determined based on the semantic similarity between the corresponding image feature blocks to be deleted and the image feature blocks to be retained. Specifically, the semantic similarity between the image feature blocks to be deleted and the image feature blocks to be retained can be calculated, for example, by using cosine similarity. After obtaining the cosine similarity, a cosine similarity of not less than 0 can be used as the association information between the corresponding image feature blocks to be deleted and the image feature blocks to be retained, while for cosine similarities less than 0, the association information between the corresponding image feature blocks to be deleted and the image feature blocks to be retained can be set to 0.

[0153] Optionally, in this embodiment, the step "aggregating the image feature blocks to be deleted into various image feature blocks to be retained based on the feature relationship matrix, to obtain an aggregated feature sequence" may include:

[0154] The feature relationship matrix is ​​fused with the feature sequence to be deleted to obtain the fused feature sequence;

[0155] The fused feature sequence is fused with each image feature block to be retained in the feature sequence to be retained to obtain the aggregated feature sequence.

[0156] The fusion of the feature relation matrix and the feature sequence to be deleted can be performed by multiplication, etc. The fusion of the fused feature sequence with each image feature block to be retained in the feature sequence to be retained can be performed by addition, etc.

[0157] Optionally, in this embodiment, the step "constructing a feature relationship matrix based on the semantic similarity between the image feature blocks to be deleted and the image feature blocks to be retained" may include:

[0158] Based on the semantic similarity between every two processed image feature blocks in the target image feature sequence, a feature relationship graph is constructed. The feature relationship graph includes multiple nodes and edges between nodes. Each node represents a processed image feature block, and the edge represents the association information between the two nodes connected to it.

[0159] Based on the association information between the nodes corresponding to the image feature blocks to be deleted and the image feature blocks to be retained in the feature relationship graph, a feature relationship matrix corresponding to the image feature blocks to be deleted and the image feature blocks to be retained is constructed.

[0160] In the feature graph, the weight of an edge is determined based on the semantic similarity between the two nodes connected by the edge. Specifically, the weight of any edge with a cosine similarity less than 0 can be reset to 0 to avoid aggregating information from nodes with different semantics.

[0161] Optionally, in this embodiment, the step "constructing a feature relationship matrix corresponding to the image feature blocks to be deleted and the image feature blocks to be retained based on the association information between the nodes corresponding to the image feature blocks to be deleted and the image feature blocks to be retained in the feature relationship graph" may include:

[0162] Based on the connection relationships between nodes in the feature relationship graph, the degree of each node in the feature relationship graph is determined.

[0163] Based on the degree, the adjacency matrix corresponding to the feature relationship graph is normalized to obtain a processed adjacency matrix. The elements in the processed adjacency matrix represent the association information between two image feature blocks at corresponding positions.

[0164] The elements corresponding to the image feature blocks to be deleted and the image feature blocks to be retained in the processed adjacency matrix are filtered to obtain the feature relationship matrix corresponding to the image feature blocks to be deleted and the image feature blocks to be retained.

[0165] In this context, the degree of a node refers to the number of edges connected to that node. For an undirected graph, the degree is the number of directly connected edges. For example, if a node is connected to 3 edges, then its degree is 3.

[0166] Specifically, based on the degree values, a pairwise angle matrix can be constructed, where the values ​​on the diagonal represent the degree of the i-th node. Then, the pairwise angle matrix and the adjacency matrix corresponding to the feature graph are fused to normalize the adjacency matrix, thus obtaining the processed adjacency matrix.

[0167] The adjacency matrix is ​​another way to represent a feature graph. For a graph G = (V, E), where V is the set of nodes and E is the set of edges, the adjacency matrix A is a matrix of |V| × |V|, where |V| is the number of nodes. The elements A of the matrix... ij This represents the weight of the edge between node i and node j. If there is no edge between node i and node j, then A... ij It can also be set to 0.

[0168] In this embodiment, the elements in the adjacency matrix of the original feature graph represent the connection strength between two corresponding nodes. Normalization based on the node's degree allows the processed adjacency matrix to better reflect the node's structural role in the graph. For example, even if two nodes have the same connection strength, their representations in the normalized adjacency matrix will differ if they are connected to different numbers of other nodes. Furthermore, this approach avoids dominance by highly connected nodes and increases the visibility of smaller connections (edges connecting to nodes with lower degrees).

[0169] Optionally, in this embodiment, the step "constructing a feature relationship matrix corresponding to the image feature blocks to be deleted and the image feature blocks to be retained based on the association information between the nodes corresponding to the image feature blocks to be deleted and the image feature blocks to be retained in the feature relationship graph" may include:

[0170] Based on the nodes corresponding to the image feature blocks to be deleted and the nodes corresponding to the image feature blocks to be retained in the feature relationship graph, the feature relationship graph is deconstructed to obtain the relationship subgraphs corresponding to the image feature blocks to be deleted and the image feature blocks to be retained.

[0171] Based on the association information between nodes in the relational subgraph, construct the feature relation matrix corresponding to the relational subgraph.

[0172] Specifically, based on the corresponding nodes and connections of the image feature blocks to be deleted and the image feature blocks to be retained in the feature relationship graph, a relationship subgraph for the image feature blocks to be deleted and the image feature blocks to be retained can be extracted and constructed, and then the adjacency matrix of the relationship subgraph can be determined, and its adjacency matrix can be used as the feature relationship matrix.

[0173] 105. Based on the aggregated feature sequence, obtain the image processing task result of the image to be processed.

[0174] Optionally, in this embodiment, the step "obtaining the image processing task result of the image to be processed based on the aggregated feature sequence" may include:

[0175] The aggregated feature sequence is fused with the target text feature sequence to obtain the fused feature sequence;

[0176] The fused feature sequence is used as a new feature sequence, and the step of performing attention interaction processing on each feature unit in the feature sequence to obtain the processed feature sequence is returned to obtain the target fused feature sequence that meets the preset conditions.

[0177] Content reasoning is performed based on the target fusion feature sequence to obtain the image processing task result of the image to be processed.

[0178] The fusion process of the aggregated feature sequence and the target text feature sequence can specifically be a concatenation process, and the concatenation order is not limited.

[0179] Specifically, the fused feature sequence of the aggregated feature sequence output by the last feature aggregation process and the corresponding target text feature sequence is the target fused feature sequence that meets the preset conditions. The inference result can be predicted based on the target fused feature sequence.

[0180] In this embodiment, attention interaction processing is performed on the feature sequence composed of text feature blocks and image feature blocks. Then, the processed image feature blocks are classified to determine the image feature blocks to be deleted and the image feature blocks to be retained. The features of the image feature blocks to be deleted are aggregated into the image feature blocks to be retained. After completing the first round of image feature block deletion and aggregation, a new round of attention interaction processing is performed on the new feature sequence composed of the aggregated feature sequence and the processed text feature sequence. Then, the processed image feature blocks obtained from the new round of attention interaction processing are deleted and feature aggregation operations are performed. By performing multiple rounds of attention interaction processing, image feature block deletion classification and feature aggregation on the feature sequence composed of text feature blocks and image feature blocks, visual markers (i.e., image feature blocks) that are less related to the semantics of the task instruction text can be gradually discarded. Moreover, the discarded visual marker information can be integrated into the retained visual markers and the integrated information can be fully fused, enhancing the stability of the information extraction process.

[0181] Optionally, in this embodiment, the step "performing attention interaction processing on each feature unit in the feature sequence through a network layer to obtain a processed feature sequence" may include:

[0182] By performing attention interaction processing on each feature unit in the feature sequence through the network layer in the attention module of the current processing module group of the processing model, a processed feature sequence is obtained. The processing model includes multiple processing module groups, and each processing module group includes an attention module, a feature selection module, and a feature aggregation module.

[0183] The step "based on the processed text feature blocks, classify the processed image feature blocks in the target image feature sequence to obtain the feature sequence to be deleted and the feature sequence to be retained" may include:

[0184] The feature selection module in the current processing module group classifies the processed image feature blocks in the target image feature sequence based on the processed text feature blocks to obtain the feature sequence to be deleted and the feature sequence to be retained.

[0185] The step "aggregating the features of the image feature blocks to be deleted into the feature sequence to be retained, to obtain the aggregated feature sequence" may include:

[0186] The feature aggregation module in the current processing module group aggregates the features of the image feature blocks to be deleted into the feature sequence to be retained, thereby obtaining the aggregated feature sequence.

[0187] The step "obtaining the image processing task result of the image to be processed based on the aggregated feature sequence" may include:

[0188] The fused feature sequence of the aggregated feature sequence and the target text feature sequence is used as the input of the next processing module group of the current processing module group to continue content reasoning processing and obtain the image processing task result of the image to be processed.

[0189] Specifically, the processing model can be a content reasoning model. For example, a content reasoning model can perform relevant question-and-answer processing based on the image to be processed.

[0190] In this embodiment, the processing model may include multiple processing module groups. Each processing module group can perform attention interaction processing, image feature block deletion classification, and feature aggregation on the feature sequence composed of text feature blocks and image feature blocks. After processing by multiple processing module groups, visual markers (i.e. image feature blocks) that are less related to the semantics of the task instruction text can be gradually discarded. Furthermore, the discarded visual marker information can be integrated into the retained visual markers, and the integrated information can be fully fused to enhance the stability of the information extraction process.

[0191] Specifically, the content reasoning model can be a Multimodal Large Language Model (MLLM).

[0192] Optionally, in this embodiment, the step "using the fused feature sequence of the aggregated feature sequence and the target text feature sequence as the input of the next processing module group of the current processing module group to continue content reasoning processing and obtain the image processing task result of the image to be processed" may include:

[0193] The aggregated feature sequence is fused with the target text feature sequence to obtain the fused feature sequence;

[0194] The next processing module group of the current processing module group is taken as the new current processing module group, and the fused feature sequence is taken as the new feature sequence.

[0195] The process involves returning to the attention module in the current processing module group of the processing model, performing attention interaction processing on each feature unit in the feature sequence to obtain the processed feature sequence, thereby obtaining the image processing task result of the image to be processed.

[0196] The fusion process of the aggregated feature sequence and the target text feature sequence can specifically be a concatenation process, and the concatenation order is not limited.

[0197] Optionally, in this embodiment, the step "returning to execute the attention module in the current processing module group of the processing model, performing attention interaction processing on each feature unit in the feature sequence to obtain the processed feature sequence, so as to obtain the image processing task result of the image to be processed" may include:

[0198] Return to the step of performing attention interaction processing on each feature unit in the feature sequence in the current processing module group of the processing model, until the aggregated feature sequence and the target text feature sequence output by the last processing module group in the processing model are obtained;

[0199] Based on the aggregated feature sequence and target text feature sequence output by the last processing module group in the processing model, predict the target inference result unit for this inference.

[0200] Based on the aggregated feature sequence and the target text feature sequence output by each processing module group of the processing model, the content feature sequence corresponding to each processing module group of the processing model is determined respectively.

[0201] By using the attention module in the processing module group of the processing model, attention processing is performed on the target inference result unit based on the content feature sequence corresponding to the processing module group, so as to predict the subsequent inference result units of the target inference result unit.

[0202] Based on the target reasoning result unit and the subsequent reasoning result unit, the image processing task result of the image to be processed is determined.

[0203] It should be noted that the image processing task results are obtained through step-by-step reasoning. That is, it requires multiple rounds of reasoning to obtain the complete image processing task results. Each round of reasoning can only obtain a portion of the reasoning result units.

[0204] In this embodiment, visual marker deletion and aggregation can be performed only in the first round of inference. Subsequent inference processes directly use the aggregated feature sequences from the first round. Specifically, the aggregated feature sequences and target text feature sequences output by each processing module group in the first round of inference can be cached and used in subsequent rounds of inference, including the second and third rounds. Subsequent rounds of inference do not involve the image selection module and image aggregation module; that is, subsequent rounds of inference only use the attention module of the content inference model.

[0205] Specifically, the first round of reasoning refers to the reasoning process of predicting the first reasoning result unit, which is also the target reasoning result unit. The second round of reasoning is the process of predicting the second reasoning result unit, and the third round of reasoning is the process of predicting the third reasoning result unit.

[0206] Specifically, for each processing module group, the aggregated feature sequence output by the processing module group during the first round of inference and the target text feature sequence can be concatenated to obtain the content feature sequence of that processing module group.

[0207] It should be noted that the reasoning process of the subsequent reasoning result unit is the same as the aggregated feature sequence used in the previous reasoning process.

[0208] After obtaining the target inference result unit and subsequent inference result units, these inference result units can be decoded by a word segmenter to obtain the final answer text, which is the content inference result, i.e., the image processing task result.

[0209] Optionally, in this embodiment, the step "by using the attention module in the processing module group of the content reasoning model to perform attention processing on the target reasoning result unit based on the content feature sequence corresponding to the processing module group, so as to predict subsequent reasoning result units" may include:

[0210] The target reasoning result unit is used as reasoning reference information;

[0211] The attention module in the first processing module group of the content reasoning model performs attention processing on the reasoning reference information based on the content feature sequence corresponding to the first processing module group to obtain the processed information.

[0212] The processed information is input into the next processing module group of the first processing module group, and the next processing module group is used as the current processing module group;

[0213] The attention module in the current processing module group performs attention processing on the processed information based on the content feature sequence corresponding to the current processing module group to obtain attention processing information.

[0214] The attention processing information is used as the new processed information, and the next processing module group of the current processing module group is used as the new current processing module group.

[0215] Return to the step of performing attention processing on the processed information based on the content feature sequence corresponding to the current processing module group through the attention module in the current processing module group, until the target attention processing information output by the last processing module group is obtained;

[0216] Based on the target attention processing information, predict the subsequent inference result unit.

[0217] In subsequent reasoning processes, the previously predicted reasoning result units can be used as reasoning reference information.

[0218] Specifically, after predicting the first inference result unit, the inference result unit can be concatenated with the content feature sequence corresponding to the first processing module group previously cached, and then sent back to the first processing module group of the LLM decoder to predict the next inference result unit.

[0219] Specifically, the content feature sequence corresponding to each processing module group can include the aggregated feature sequence obtained by the feature selection module and feature aggregation module in the process of predicting the first inference result unit, as well as the target text feature sequence output by the last attention module in the corresponding processing module group.

[0220] Optionally, in this embodiment, the step "predicting subsequent inference result units based on the target attention processing information" may include:

[0221] Based on the target attention processing information, predict the next round of reasoning result units;

[0222] Based on the target reasoning result unit and the new round of reasoning result units, update the reasoning reference information;

[0223] Return to the attention module in the first processing module group of the content reasoning model, and perform attention processing on the reasoning reference information based on the content feature sequence corresponding to the first processing module group to obtain the subsequent reasoning result unit.

[0224] In this process, the target reasoning result unit and the new round of reasoning result unit can be spliced ​​together to obtain new reasoning reference information.

[0225] Specifically, when a new inference result unit is predicted by each processing module group of the content inference model, this inference result unit will be sent back to the first processing module group of the content inference model together with the previously predicted inference result units. Specifically, this inference result unit can be concatenated with the previously predicted inference result units to obtain a sequence (specifically, inference reference information). Then, this sequence will be processed by group1, group2, group3, ..., group N in the content inference model to generate the next prediction tag, that is, to predict the next inference result unit.

[0226] Wherein, group1 represents the first processing module group in the content reasoning model, group2 represents the second processing module group in the content reasoning model, group3 represents the third processing module group in the content reasoning model, and group N represents the Nth processing module group in the content reasoning model.

[0227] It should be noted that the prediction process of each inference result unit requires processing by the attention module of each processing module group. However, the feature selection module and feature aggregation module in the processing module group are only used in the prediction process of the first inference result unit. Subsequent inference result units do not need to use the feature selection module and feature aggregation module, and can directly use the aggregated feature sequence obtained from the prediction process of the first inference result unit.

[0228] For example, if the image to be processed is a picture of a cat, and the task instruction text is "describe this cat," the content inference model described above can extract features from the cat image to obtain visual tags, and convert the task instruction text into text tags. Then, the visual tags and text tags are concatenated together to form a comprehensive input feature sequence. The decoder of the content inference model receives this input feature sequence and predicts the first tag, such as "this one." Then, the newly generated tag "this one" is concatenated with the content feature sequence corresponding to the first processing module group to form a new sequence, which is then fed into the first processing module group to obtain the corresponding attention processing information. This attention processing information is then concatenated with the content feature sequence corresponding to the second processing module group, and the resulting sequence is fed into the second processing module group to obtain the attention processing information output by the second processing module group. This is then concatenated with the content feature sequence corresponding to the third processing module group, and so on, to predict the second tag, such as "cat," based on the output of the last processing module group. This process repeats continuously. Each time, a new tag is generated and added to the sequence of already predicted tags. Then, the next tag is predicted again by the various processing modules of the decoder. For example, at the third time step, the decoder might predict "very," and at the fourth time step, it might predict "cute." This process continues until a complete description, such as "This cat is very cute," is generated. The generation process ends when the decoder generates a terminating tag, such as a period.

[0229] Specifically, after generating the first token, the decoder adds this newly generated token to the input generated sequence, that is, it concatenates it with the previously predicted token to form a new sequence. This new sequence is then fed back into the decoder, which uses this expanded sequence to predict the next token. This process is iterative, generating a new token each time and adding it to the sequence, then predicting the next token again, until a complete output sequence is generated or a certain termination condition is reached, such as the generation of a specific end token.

[0230] Optionally, in this embodiment, the image processing method may further include:

[0231] Obtain the original processing model, which includes multiple attention modules;

[0232] The original processing model is grouped into multiple processing module groups, each of which includes at least one attention module.

[0233] Add a feature selection module and a feature aggregation module to each processing module group to obtain the processing model.

[0234] Specifically, the original processing model can be the original content inference model, whose decoder part can be composed of multiple stacked Transformer layers. Each Transformer layer is an attention module.

[0235] Within each processing module group, the attention modules are consecutive. A feature selection module can be connected after multiple consecutive attention modules; that is, the output of the last attention module in the processing module group becomes the input of the feature selection module. A feature aggregation module can be connected after the feature selection module. The feature aggregation module of each processing module group is connected to the first attention module of the next processing module group.

[0236] Optionally, in this embodiment, the step "grouping the multiple attention modules in the original processing model to obtain multiple processing module groups" may include:

[0237] Based on the computational requirements, determine the number of groups for the original processing model;

[0238] Based on the number of groups, the multiple attention modules in the original processing model are grouped to obtain multiple processing module groups.

[0239] The more groups there are, the more visual tags are deleted. Since the number of attention modules in a content reasoning model is generally fixed, the number of attention modules in each processing module group is related to the number of groups. More groups mean fewer attention modules per group; conversely, fewer groups mean more attention modules per group. Therefore, adjusting the number of attention modules (s) in each processing module group can affect the computational load.

[0240] This application proposes a novel method based on grouped visual tag selection and aggregation to address the problem of low inference efficiency caused by excessive visual tags in large-scale multimodal language models. Specifically, this application first proposes a graph-based visual tag aggregation module, which treats each visual tag as a node and constructs a graph based on the semantic similarity between visual tags. Then, based on this graph, the information of visual tags to be deleted is aggregated into the visual tags to be retained, resulting in a more compact visual tag representation that can retain visual information to the greatest extent while compressing visual tags. In addition, this application proposes a grouped tag selection strategy, which divides visual tags into tags to be retained and tags to be deleted. This division process is guided by text attention in the last few layers of each group. The grouped tag selection strategy can aggregate visual information step by step, enhancing the stability of the visual information extraction process. Extensive experiments demonstrate that this application can achieve a more superior balance between model performance and inference speed.

[0241] In a specific scenario, refer to Figure 1c This is a model architecture diagram of the content reasoning model provided in this application. Specifically, the content reasoning model is an MLLM model, which mainly consists of three components: a visual encoder, a projector, and an LLM decoder. The LLM decoder includes multiple processing module groups. Specifically, the LLM decoder includes N processing module groups, which can be denoted as group1, group2, group3, ..., group N. Each processing module group can include multiple layers, where each layer can be a transformer layer. For example, group 1 includes two layers, and group 2 includes S layers.

[0242] The visual encoder extracts features from the input image, which are then converted into visual tags by a mapper. For text input, the text is first split by a tokenizer and mapped to indices corresponding to the vocabulary. The text embedding features for each token can be obtained through these indices. The LLM decoder consists of multiple transformer layers, such as... Figure 1c At the bottom. Each transformer layer mainly consists of causal self-attention and feedforward networks, used to realize information interaction and semantic extraction between input tokens. The input image and text instruction are converted into visual and text tokens respectively, then concatenated and fed into the LLM decoder to obtain the currently predicted token. This token is then concatenated with the previous token and fed back into the LLM decoder to predict the next token. Finally, the newly generated token sequence can be decoded by a tokenizer to obtain the final answer text.

[0243] like Figure 1c As shown in the overall framework diagram, the method provided in this application can be applied to LLM decoders. Specifically, the transformer layer in the LLM decoder can be divided into multiple groups, and then visual tokens are aggregated at the end of each group to reduce the number of visual tokens, thereby accelerating the model's inference process. In particular, this scheme only performs information aggregation and pruning of visual tokens in each group during the prediction of the first text token. Subsequent text tokens are predicted based on the key visual tokens retained after pruning, ensuring the efficiency of inference.

[0244] like Figure 1cAs shown, Visual Tag Aggregation (VTA) is applied at the end of each group. VTA treats each visual tag as a node and constructs a graph based on their semantic similarity. The network then aggregates information from deleted nodes into the retained nodes based on this graph, resulting in a more compact visual tag representation. Specifically, given a sequence of visual tags x... vis ∈R n×d , where n is the length of the visual tag sequence and d is the size of the feature dimension. VTA constructs a graph based on the cosine similarity between visual tags to represent the weights of edges between nodes. The adjacency matrix of this graph is defined as shown in equation (4):

[0245]

[0246] To prevent nodes from propagating their own information back to themselves, the weights of edges between a node and itself can be reset to 0. Furthermore, the weights of any edges with a cosine similarity less than 0 can also be reset to 0 to avoid aggregating information from nodes with different semantics. Next, the adjacency matrix G can be symmetrically normalized, as shown in equations (5) and (6):

[0247]

[0248]

[0249] Among them, D i，i Let D represent the degree of node i. D is an angle matrix, and the values ​​on its diagonal represent the degree of the i-th node.

[0250] After the graph is constructed, the visual markers x can be added. vis Divide into two sets: the set of visual markers to be retained. (i.e., the feature sequence to be retained in the above embodiments) and the set of visual markers to be removed. (i.e., the feature sequence to be deleted in the above embodiment). Where, n k and n r Let represent the lengths of the visual marker sequences to be retained and those to be removed, respectively. Next, using graph summarization techniques, the information of the visual markers to be removed is aggregated into the semantically related visual markers to be retained, as shown in equation (7). The aggregated feature sequence can then be obtained through the calculation of equation (7):

[0251]

[0252] Among them, here, Represents a symmetric normalized adjacency matrix The subgraph. In the matrix In the matrix, the row index and column index correspond to the visual tags to be retained and the visual tags to be deleted, respectively. That is, the feature relationship matrix in the above embodiments. In other words, Each row in the table represents the correlation between a tag to be retained and all tags to be deleted; that is, information about each tag to be deleted can be propagated to multiple different tags. The hyperparameter α controls the extent of information propagation.

[0253] This embodiment applies visual tag aggregation across each group of transformer layers. Each time tag aggregation is performed, a new graph is built based on the currently updated visual tags. This dynamic graph construction process ensures that information from the updated visual tags can be utilized when aggregating information.

[0254] Specifically, this embodiment can group the layers of the LLM decoder and perform visual tag aggregation at the end of each group. This strategy ensures that the LLM has enough layers to extract visual information from the updated representation each time the visual tag representation is updated, enhancing the stability of the information extraction process. The first group consists of the first two layers to facilitate the extraction of global information from all visual tags in these layers. Subsequently, starting from the second layer, each consecutive S layers are considered a separate group.

[0255] The following is a detailed description of the tag selection strategy for one of the groups. For and In our partitioning method, we utilize the attention information of the entire group to enhance the partitioning. and The effectiveness can be assessed by averaging the attention scores of the last L layers in each group. and The division. Meanwhile, considering that MLLM determines the focal region of the image based on the question prompts, we consider evaluating the importance of each visual marker by focusing on the attention of the text to the visual marker. Since the newly generated text markers are predicted based on the features of the last marker (i.e., the last text marker) in the current model input sequence, the attention score of the last text marker to the visual marker is used as an indicator of the visual marker's importance, denoted as I. The calculation of I can be shown in equation (1) in the above embodiment. Then, based on the metric I, x... vis Divided into and As shown in equations (2) and (3) above. By adjusting the number or proportion of visual markers to be retained and the number of layers S in each group, this application can meet various floating-point operation requirements. This flexibility allows the model to adapt to different application needs and computing environments, thereby achieving more efficient resource utilization and better performance. The specific hyperparameter p remains consistent across different groupings.

[0256] Specifically, if computational resources are limited, the number of visual tags to be retained can be reduced, thereby decreasing the computational load of the model. If higher model performance is required, the number of visual tags to be retained can be increased, thereby improving the model's expressive power; however, this may increase the computational load.

[0257] In summary, this grouping and tag selection strategy, on the one hand, can accurately identify key visual tags through textual attention guidance. On the other hand, it employs a progressive approach to gradually discard visual tags that are less relevant to the semantics of the instructions. Unlike clipping at each layer or only at the second layer, this approach integrates the information from discarded visual tags into the retained ones and fully fuses the integrated information, enhancing the stability of the information extraction process.

[0258] refer to Figure 1d The performance of the image processing method provided in this application is compared with that of some related technologies on four commonly used datasets: TextVQA, POPE, VQAv2, and MME.

[0259] Among them, VISA is the image processing method provided in this application, while LLaVA-1.5-7B, FastV, ToMe, and SparseVLM are some related technologies currently available. Figure 1d It is evident that the image processing method provided in this application is significantly superior to these related technologies. Even under high visual label compression rates or for some tasks that require detailed image information, it can still guarantee model performance. Other related technologies, on the other hand, struggle to retain sufficient visual information under high visual label compression rates, leading to a significant decrease in model performance, or perform poorly in tasks that require understanding detailed image information (such as TextVQA), such as ToMe.

[0260] The image processing method provided in this application can be specifically called VISA (group-wise Visual Token Selection and Aggregation), which is a method based on group-wise visual token selection and aggregation. It mainly includes a visual token aggregation module and a group-wise token selection strategy. The visual token aggregation algorithm proposed in this application aggregates the information of visual tokens to be deleted into visual tokens to be retained through graph summarization. This algorithm makes the visual token representation more compact while preserving visual information to the greatest extent. The group-wise token selection strategy divides visual tokens into tokens to be retained and tokens to be deleted, and the division is guided by text attention in the last few layers of each group. This strategy ensures more stable and accurate visual information extraction and token selection.

[0261] Visual token aggregation (VTA) involves aggregating information from one set of visual tokens into another set. Graph summarization transforms a graph representation into a more compact one.

[0262] This application has significant practical value, as it can be plugged and played into various pre-trained multimodal large language models (including image, video, and high-resolution scenes), improving inference speed and throughput with almost no loss in model performance. Faster model response enhances the user experience. Simultaneously, by reducing the model's floating-point computation and increasing inference speed, the inference cost is also reduced. Existing multimodal large language models possess strong generalization capabilities and can achieve superior recognition results in various image understanding tasks such as image recognition, tag generation, and title generation, with appropriate fine-tuning on business data. This accelerated solution significantly reduces actual deployment inference costs, increasing the likelihood of practical implementation.

[0263] Specifically, the performance of this application and other methods on 10 datasets was evaluated on LLaVA-1.5-7B. Figure 1e As shown in the performance comparison results on LLaVA-1.5-7B, this application achieved the highest overall performance across all compression ratio settings. Other methods experienced significant performance degradation on some datasets with higher compression ratios due to the loss of visual information. For example, FastV's performance dropped by 31.1% on POPE and ToMe's by 36.0% on MM-Vet when only 64 visual labels were retained. In contrast, the method in this application maintained high performance across various benchmarks by effectively preserving visual information. On average, it retained 93.8% of the original model's performance across 10 benchmarks.

[0264] In addition, the results of version 13B of LLaVA-1.5 were compared. For example... Figure 1f As shown, the performance comparison results on LLaVA-1.5-13B are presented. The method of this application is consistently superior to the previous method, which proves that the method of this application can also achieve better performance on models with larger parameters.

[0265] Specifically, by increasing the resolution of input images, some studies have achieved performance improvements on many multimodal tasks. However, high-resolution image inputs result in longer sequences of visual labels, which further reduces the inference speed of the model. Applying this application to LLaVA-NeXT-7B, it can process up to 2880 visual labels. Figure 1gAs shown, the performance comparison results on LLaVA-NeXT-7B are presented. Compared with other methods, this application also performs well in high-resolution scenes, which further proves the effectiveness of this application.

[0266] Additionally, video is another modality that can be converted into a large number of visually labeled inputs. In Video-LLaVA, each video input is extracted with 8 frames, which are ultimately converted into 2048 visually labeled inputs, exhibiting a high degree of redundancy. For example... Figure 1h As shown, the performance comparison results on Video-LLaVA show that even with only 128 visual labels retained, the method proposed in this application can maintain 89.7% of the model performance, surpassing previous methods.

[0267] To demonstrate the inference acceleration effect of this application, the actual inference time of LLaVA-1.5 was tested on an RTX 3090 GPU. The benchmark was MMBeench. Since MMBeench requires the model to answer only one option for each example, it eliminates the impact of the output sequence length on the inference time. Figure 1i As shown, the inference acceleration effect is demonstrated. This application can increase the throughput by 43.6% on LLaVA-1.5-7B and by 80.8% on LLaVA-1.5-13B with almost no performance loss, thereby significantly improving the inference speed of the model.

[0268] As can be seen from the above, this embodiment can determine the image to be processed; perform feature extraction processing on the image to be processed to obtain an image feature sequence, the image feature sequence including multiple image feature blocks of the image to be processed; process the image feature blocks in the image feature sequence through a network layer to obtain a target image feature sequence, the target image feature sequence including multiple processed image feature blocks; classify the processed image feature blocks in the target image feature sequence to obtain a feature sequence to be deleted and a feature sequence to be retained, the feature sequence to be deleted including at least one image feature block to be deleted, the feature sequence to be retained including at least one image feature block to be retained; aggregate the features of the image feature blocks to be deleted into the feature sequence to be retained to obtain an aggregated feature sequence, the aggregated feature sequence including the aggregated image feature blocks corresponding to each image feature block to be retained; and obtain the image processing task result of the image to be processed based on the aggregated feature sequence.

[0269] This application can reduce computational overhead and improve content reasoning efficiency by deleting some image feature blocks; at the same time, it can also aggregate the deleted image feature blocks into the retained image feature blocks to avoid the loss of visual information caused by deleting image feature blocks and ensure the accuracy of content reasoning.

[0270] Based on the method described in the preceding embodiments, the following will provide a more detailed description using the example of the image processing device being specifically integrated into a server.

[0271] This application provides an image processing method, such as... Figure 2 As shown, the specific process of this image processing method can be as follows:

[0272] 201. The server determines the image to be processed and the task instruction text for the image to be processed.

[0273] Among them, the images to be processed are images whose content needs to be reasoned about. The reasoning here can be image understanding tasks such as image classification and visual question answering.

[0274] Specifically, the task instruction text can be used to provide guidance information for the model to perform image content extraction.

[0275] 202. The server performs feature extraction processing on the image to be processed to obtain an image feature sequence, the image feature sequence including multiple image feature blocks of the image to be processed.

[0276] In this embodiment, the feature extraction process of the image to be processed can be specifically performed by a visual encoder to extract the features of the image to be processed, obtain image coded features, and then convert the extracted image coded features into multiple visual tags.

[0277] In this embodiment, the image feature sequence is specifically a visual token sequence, which includes multiple visual tokens, each of which is an image feature block. Visual tokens specifically refer to the image feature representation of each small block after the image is divided into multiple small blocks.

[0278] 203. The server performs feature extraction on the task instruction text to obtain a text feature sequence, wherein the text feature sequence includes text feature blocks corresponding to multiple text units in the task instruction text.

[0279] Specifically, feature extraction of the task instruction text can be achieved by splitting the task instruction text into segments using a tokenizer, mapping them to corresponding indices in a vocabulary, and then obtaining the text embedding features corresponding to each segment through the indices, thereby obtaining a text feature sequence. The text feature sequence includes text feature blocks corresponding to multiple text units in the task instruction text, with each text unit corresponding to a segment, and the text feature block corresponding to the text unit being the text token.

[0280] 204. The server fuses the image feature sequence and the text feature sequence to obtain a feature sequence.

[0281] In this embodiment, the image feature sequence and the text feature sequence are fused. Specifically, the image feature sequence and the text feature sequence are concatenated. The concatenation order is not limited. The image feature sequence can be concatenated after the text feature sequence, or the text feature sequence can be concatenated after the image feature sequence.

[0282] 205. The server performs attention interaction processing on each feature unit in the feature sequence to obtain a processed feature sequence. The processed feature sequence includes a target image feature sequence and a target text feature sequence. The target image feature sequence includes multiple processed image feature blocks. The target text feature sequence includes processed text feature blocks corresponding to the multiple text units. The feature unit is an image feature block or a text feature block.

[0283] Specifically, the attention mechanism can be used to directly capture the dependency between any two positions in the feature sequence, thereby efficiently capturing long-distance dependencies and modeling the interaction between elements in the sequence.

[0284] 206. Based on the processed text feature blocks, the server performs classification processing on the processed image feature blocks in the target image feature sequence to obtain a feature sequence to be deleted and a feature sequence to be retained. The feature sequence to be deleted includes at least one image feature block to be deleted, and the feature sequence to be retained includes at least one image feature block to be retained.

[0285] In this embodiment, the attention score of text tags to visual tags can be used to segment visual tags. The processed text feature blocks are also text tags, and the processed image feature blocks in the target image feature sequence are also visual tags. The attention score can serve as an indicator of the importance of visual tags, and based on this indicator, visual tags to be deleted and visual tags to be retained can be determined.

[0286] Specifically, the target text feature sequence includes multiple processed text feature blocks. The processed image feature blocks in the target image feature sequence can be classified based on the attention score of the last processed text feature block in the target text feature sequence to the processed image feature blocks.

[0287] Optionally, in this embodiment, the step "performing attention interaction processing on each feature unit in the feature sequence to obtain the processed feature sequence" may include:

[0288] Multi-layer attention interaction processing is performed on each feature unit in the feature sequence to obtain the processed feature sequence output by each layer of attention interaction processing, wherein the output of the previous layer of attention interaction processing is used as the input of the next layer of attention interaction processing.

[0289] The step "based on the processed text feature blocks, classify the processed image feature blocks in the target image feature sequence to obtain the feature sequence to be deleted and the feature sequence to be retained" may include:

[0290] Determine at least one target post-processed feature sequence from the attention interaction processing output of the target network layer from the post-processed feature sequences of each layer.

[0291] Based on the processed text feature blocks in the at least one target processed feature sequence, the processed image feature blocks in the target image feature sequence are classified to obtain a feature sequence to be deleted and a feature sequence to be retained.

[0292] Specifically, the target image feature sequence here is the image feature sequence output from the last layer in the multi-layer attention interaction processing.

[0293] Optionally, in this embodiment, the step "based on the processed text feature blocks in the at least one target processed feature sequence, classify the processed image feature blocks in the target image feature sequence to obtain a feature sequence to be deleted and a feature sequence to be retained" may include:

[0294] For each processed image feature block in the target image feature sequence, determine the attention information between the processed text feature block in each target processed feature sequence and the processed image feature block;

[0295] The attention information between the processed text feature blocks and the processed image feature blocks in each target processed feature sequence is fused to obtain the target attention information of the processed image feature blocks.

[0296] Based on the target attention information, the processed image feature blocks in the target image feature sequence are classified to obtain the feature sequence to be deleted and the feature sequence to be retained.

[0297] Specifically, the processed text feature block in the target processed feature sequence can be a processed text feature block at a certain position in the target processed feature sequence. In a specific embodiment, the importance of the processed image feature block can be evaluated using the attention information between the last processed text feature block in the target processed feature sequence obtained from each target network layer and the processed image feature block in the target image feature sequence.

[0298] 207. The server aggregates the features of the image feature blocks to be deleted into each image feature block to be retained in the feature sequence to be retained, and obtains an aggregated feature sequence, wherein the aggregated feature sequence includes the aggregated image feature blocks corresponding to each image feature block to be retained.

[0299] Feature aggregation can be used to delete feature sequences while ensuring the accuracy of image content inference.

[0300] The number of image feature blocks contained in the feature sequence to be retained and the aggregated feature sequence is the same. Specifically, the feature dimension of the aggregated image feature block and the corresponding image feature block to be retained is the same.

[0301] Optionally, in this embodiment, the step "aggregating the features of the image feature blocks to be deleted into the respective image feature blocks to be retained in the feature sequence to be retained, to obtain the aggregated feature sequence" may include:

[0302] Based on the semantic similarity between the image feature blocks to be deleted and the image feature blocks to be retained, a feature relationship matrix is ​​constructed. The elements in the feature relationship matrix represent the association information between the image feature blocks to be deleted and the image feature blocks to be retained at the corresponding positions.

[0303] Based on the feature relationship matrix, the image feature blocks to be deleted are aggregated into each image feature block to be retained, resulting in an aggregated feature sequence.

[0304] Specifically, the row index and column index of the feature relation matrix correspond to the visual tags to be retained and the visual tags to be deleted, respectively. In other words, each row of the feature relation matrix represents the correlation between a visual tag to be retained and all visual tags to be deleted. In this way, through the feature relation matrix, the information of each visual tag to be deleted can be propagated to multiple different visual tags to be retained.

[0305] Optionally, in this embodiment, the step "aggregating the image feature blocks to be deleted into various image feature blocks to be retained based on the feature relationship matrix, to obtain an aggregated feature sequence" may include:

[0306] The feature relationship matrix is ​​fused with the feature sequence to be deleted to obtain the fused feature sequence;

[0307] The fused feature sequence is fused with each image feature block to be retained in the feature sequence to be retained to obtain the aggregated feature sequence.

[0308] The fusion of the feature relation matrix and the feature sequence to be deleted can be performed by multiplication, etc. The fusion of the fused feature sequence with each image feature block to be retained in the feature sequence to be retained can be performed by addition, etc.

[0309] Optionally, in this embodiment, the step "constructing a feature relationship matrix based on the semantic similarity between the image feature blocks to be deleted and the image feature blocks to be retained" may include:

[0310] Based on the semantic similarity between every two processed image feature blocks in the target image feature sequence, a feature relationship graph is constructed. The feature relationship graph includes multiple nodes and edges between nodes. Each node represents a processed image feature block, and the edge represents the association information between the two nodes connected to it.

[0311] Based on the association information between the nodes corresponding to the image feature blocks to be deleted and the image feature blocks to be retained in the feature relationship graph, a feature relationship matrix corresponding to the image feature blocks to be deleted and the image feature blocks to be retained is constructed.

[0312] Optionally, in this embodiment, the step "constructing a feature relationship matrix corresponding to the image feature blocks to be deleted and the image feature blocks to be retained based on the association information between the nodes corresponding to the image feature blocks to be deleted and the image feature blocks to be retained in the feature relationship graph" may include:

[0313] Based on the connection relationships between nodes in the feature relationship graph, the degree of each node in the feature relationship graph is determined.

[0314] Based on the degree, the adjacency matrix corresponding to the feature relationship graph is normalized to obtain a processed adjacency matrix. The elements in the processed adjacency matrix represent the association information between two image feature blocks at corresponding positions.

[0315] The elements corresponding to the image feature blocks to be deleted and the image feature blocks to be retained in the processed adjacency matrix are filtered to obtain the feature relationship matrix corresponding to the image feature blocks to be deleted and the image feature blocks to be retained.

[0316] In this embodiment, the elements in the adjacency matrix of the original feature graph represent the connection strength between two corresponding nodes. Normalization based on the node's degree allows the processed adjacency matrix to better reflect the node's structural role in the graph. For example, even if two nodes have the same connection strength, their representations in the normalized adjacency matrix will differ if they are connected to different numbers of other nodes. Furthermore, this approach avoids dominance by highly connected nodes and increases the visibility of smaller connections (edges connecting to nodes with lower degrees).

[0317] 208. The server performs content reasoning processing based on the aggregated feature sequence to obtain and output the content reasoning result of the image to be processed.

[0318] Optionally, in this embodiment, the step "performing attention interaction processing on each feature unit in the feature sequence to obtain the processed feature sequence" may include:

[0319] The attention module in the current processing module group of the content reasoning model performs attention interaction processing on each feature unit in the feature sequence to obtain the processed feature sequence. The content reasoning model includes multiple processing module groups, and each processing module group includes an attention module, a feature selection module, and a feature aggregation module.

[0320] The step "based on the processed text feature blocks, classify the processed image feature blocks in the target image feature sequence to obtain the feature sequence to be deleted and the feature sequence to be retained" may include:

[0321] The feature selection module in the current processing module group classifies the processed image feature blocks in the target image feature sequence based on the processed text feature blocks to obtain the feature sequence to be deleted and the feature sequence to be retained.

[0322] The step "aggregating the features of the image feature blocks to be deleted into the individual image feature blocks to be retained in the feature sequence to be retained, to obtain the aggregated feature sequence" may include:

[0323] The feature aggregation module in the current processing module group aggregates the features of the image feature blocks to be deleted into each image feature block to be retained in the feature sequence to be retained, thereby obtaining the aggregated feature sequence.

[0324] The step "perform content inference processing based on the aggregated feature sequence to obtain and output the content inference result of the image to be processed" may include:

[0325] The fused feature sequence of the aggregated feature sequence and the target text feature sequence is used as the input of the next processing module group of the current processing module group to continue the content reasoning process, and the content reasoning result of the image to be processed is obtained and output.

[0326] The content reasoning model can include multiple processing module groups. Each processing module group can perform attention interaction processing, image feature block deletion classification, and feature aggregation on the feature sequence composed of text feature blocks and image feature blocks. After processing by multiple processing module groups, visual markers (i.e. image feature blocks) that are less related to the semantics of the task instruction text can be gradually discarded. Furthermore, the discarded visual marker information can be integrated into the retained visual markers, and the integrated information can be fully fused to enhance the stability of the information extraction process.

[0327] As can be seen from the above, this embodiment can determine the image to be processed and the task instruction text for the image to be processed through a server; perform feature extraction processing on the image to be processed to obtain an image feature sequence, the image feature sequence including multiple image feature blocks of the image to be processed; perform feature extraction on the task instruction text to obtain a text feature sequence, the text feature sequence including text feature blocks corresponding to multiple text units in the task instruction text; perform fusion processing on the image feature sequence and the text feature sequence to obtain a feature sequence; perform attention interaction processing on each feature unit in the feature sequence to obtain a processed feature sequence, the processed feature sequence including a target image feature sequence and a target text feature sequence, the target image feature sequence including multiple processed image feature blocks, the target text ... The feature sequence includes processed text feature blocks corresponding to the plurality of text units, wherein the feature units are image feature blocks or text feature blocks; based on the processed text feature blocks, the processed image feature blocks in the target image feature sequence are classified to obtain a feature sequence to be deleted and a feature sequence to be retained, wherein the feature sequence to be deleted includes at least one image feature block to be deleted, and the feature sequence to be retained includes at least one image feature block to be retained; the features of the image feature blocks to be deleted are aggregated into each image feature block to be retained in the feature sequence to be retained to obtain an aggregated feature sequence, wherein the aggregated feature sequence includes the aggregated image feature blocks corresponding to each image feature block to be retained; content reasoning processing is performed based on the aggregated feature sequence to obtain and output the content reasoning result of the image to be processed. This application can reduce computational overhead and improve content reasoning efficiency by deleting some image feature blocks; at the same time, it can also aggregate deleted image feature blocks into retained image feature blocks to avoid visual information loss caused by deleting image feature blocks and ensure the accuracy of content reasoning.

[0328] To better implement the above methods, embodiments of this application also provide an image processing apparatus, such as... Figure 3 As shown, the image processing device may include a feature extraction unit 301, a processing unit 302, a classification unit 303, a feature aggregation unit 304, and a result determination unit 305, as follows:

[0329] (1) Feature extraction unit 301;

[0330] A feature extraction unit is used to determine the image to be processed and to perform feature extraction processing on the image to be processed to obtain an image feature sequence, wherein the image feature sequence includes multiple image feature blocks of the image to be processed.

[0331] Optionally, in some embodiments of this application, the image processing apparatus may further include a text feature extraction unit and a sequence fusion unit, as follows:

[0332] The text feature extraction unit is used to obtain task instruction text for the image to be processed; and to extract features from the task instruction text to obtain a text feature sequence, wherein the text feature sequence includes text feature blocks corresponding to multiple text units in the task instruction text;

[0333] The sequence fusion unit is used to fuse the image feature sequence and the text feature sequence to obtain a feature sequence.

[0334] (2) Processing unit 302;

[0335] The processing unit is used to process the image feature blocks in the image feature sequence through a network layer to obtain a target image feature sequence, wherein the target image feature sequence includes multiple processed image feature blocks.

[0336] Optionally, in some embodiments of this application, the processing unit may be specifically used to perform attention interaction processing on each feature unit in the feature sequence through a network layer to obtain a processed feature sequence. The processed feature sequence includes a target image feature sequence and a target text feature sequence. The target text feature sequence includes processed text feature blocks corresponding to the plurality of text units. The feature unit is an image feature block or a text feature block.

[0337] (3) Classification unit 303;

[0338] A classification unit is used to classify the processed image feature blocks in the target image feature sequence to obtain a feature sequence to be deleted and a feature sequence to be retained. The feature sequence to be deleted includes at least one image feature block to be deleted, and the feature sequence to be retained includes at least one image feature block to be retained.

[0339] Optionally, in some embodiments of this application, the classification unit may be specifically used to classify the processed image feature blocks in the target image feature sequence based on the processed text feature blocks, to obtain the feature sequence to be deleted and the feature sequence to be retained.

[0340] Optionally, in some embodiments of this application, the processing unit may be specifically used to perform multi-layer attention interaction processing on each feature unit in the feature sequence through multiple network layers to obtain the processed feature sequence output by each layer of attention interaction processing, wherein the output of the attention interaction processing of the previous network layer is used as the input of the attention interaction processing of the next network layer.

[0341] The classification unit may include a determining subunit and a classification subunit, as follows:

[0342] The determining subunit is used to determine, from the processed feature sequences output by the attention interaction processing of the plurality of network layers, at least one target processed feature sequence output by the attention interaction processing of the target network layer.

[0343] The classification subunit is used to classify the processed image feature blocks in the target image feature sequence based on the processed text feature blocks in the at least one target processed feature sequence, to obtain the feature sequence to be deleted and the feature sequence to be retained.

[0344] Optionally, in some embodiments of this application, the classification subunit may be specifically used to determine the attention information between the processed text feature blocks in each target processed feature sequence and the processed image feature blocks for each processed image feature block in the target image feature sequence; to fuse the attention information between the processed text feature blocks in all target processed feature sequences and the processed image feature blocks to obtain the target attention information of the processed image feature blocks; and to classify the processed image feature blocks in the target image feature sequence based on the target attention information to obtain the feature sequence to be deleted and the feature sequence to be retained.

[0345] Optionally, in some embodiments of this application, the target processed feature sequence output by the attention interaction processing of the target network layer includes the processed sub-feature sequences output by the target network layer in multiple subspaces, and the processed sub-feature sequences include processed text feature blocks in the corresponding subspaces;

[0346] The step "fusing the attention information between the processed text feature blocks and the processed image feature blocks in all target-processed feature sequences to obtain the target attention information of the processed image feature blocks" may include:

[0347] For each target network layer, the attention information of the processed text feature blocks output by the target network layer in each subspace and the processed image feature blocks are fused to obtain the attention information of the processed text feature blocks and the processed image feature blocks of the target network layer.

[0348] The attention information of the processed text feature blocks of each target network layer is fused with the attention information of the processed image feature blocks to obtain the target attention information of the processed image feature blocks.

[0349] (4) Feature aggregation unit 304;

[0350] The feature aggregation unit is used to aggregate the features of the image feature blocks to be deleted into the feature sequence to be retained, so as to obtain the aggregated feature sequence, which includes the aggregated image feature blocks corresponding to each image feature block to be retained.

[0351] Optionally, in some embodiments of this application, the feature aggregation unit may be used to aggregate the features of the image feature block to be deleted into each image feature block to be retained in the feature sequence to be retained, so as to obtain the aggregated feature sequence.

[0352] Optionally, in some embodiments of this application, the feature aggregation unit may include a matrix construction subunit and a feature aggregation subunit, as follows:

[0353] The matrix construction subunit is used to construct a feature relationship matrix based on the semantic similarity between the image feature blocks to be deleted and the image feature blocks to be retained. The elements in the feature relationship matrix represent the association information between the image feature blocks to be deleted and the image feature blocks to be retained at the corresponding positions.

[0354] The feature aggregation subunit is used to aggregate the image feature blocks to be deleted into each image feature block to be retained based on the feature relationship matrix, so as to obtain the aggregated feature sequence.

[0355] Optionally, in some embodiments of this application, the feature aggregation subunit may be specifically used to fuse the feature relationship matrix with the feature sequence to be deleted to obtain a fused feature sequence; and to fuse the fused feature sequence with each image feature block to be retained in the feature sequence to be retained to obtain an aggregated feature sequence.

[0356] Optionally, in some embodiments of this application, the matrix construction subunit can be specifically used to construct a feature relationship graph based on the semantic similarity between every two processed image feature blocks in the target image feature sequence. The feature relationship graph includes multiple nodes and edges between nodes. Each node represents a processed image feature block, and the edge represents the association information between the two nodes connected to it. Based on the association information between the nodes corresponding to the image feature blocks to be deleted and the image feature blocks to be retained in the feature relationship graph, a feature relationship matrix corresponding to the image feature blocks to be deleted and the image feature blocks to be retained is constructed.

[0357] Optionally, in some embodiments of this application, the step "constructing a feature relationship matrix corresponding to the image feature blocks to be deleted and the image feature blocks to be retained based on the association information between the nodes corresponding to the image feature blocks to be deleted and the image feature blocks to be retained in the feature relationship graph" may include:

[0358] Based on the connection relationships between nodes in the feature relationship graph, the degree of each node in the feature relationship graph is determined.

[0359] Based on the degree, the adjacency matrix corresponding to the feature relationship graph is normalized to obtain a processed adjacency matrix. The elements in the processed adjacency matrix represent the association information between two image feature blocks at corresponding positions.

[0360] The elements corresponding to the image feature blocks to be deleted and the image feature blocks to be retained in the processed adjacency matrix are filtered to obtain the feature relationship matrix corresponding to the image feature blocks to be deleted and the image feature blocks to be retained.

[0361] (5) Result determination unit 305;

[0362] The result determination unit is used to obtain the image processing task result of the image to be processed based on the aggregated feature sequence.

[0363] Optionally, in some embodiments of this application, the processing unit may be specifically used to perform attention interaction processing on each feature unit in the feature sequence through the network layer in the attention module in the current processing module group of the processing model to obtain the processed feature sequence. The processing model includes multiple processing module groups, and each processing module group includes an attention module, a feature selection module, and a feature aggregation module.

[0364] The classification unit can be specifically used to classify the processed image feature blocks in the target image feature sequence based on the processed text feature blocks through the feature selection module in the current processing module group, so as to obtain the feature sequence to be deleted and the feature sequence to be retained;

[0365] Specifically, the feature aggregation unit can be used to aggregate the features of the image feature block to be deleted into the feature sequence to be retained through the feature aggregation module in the current processing module group, so as to obtain the aggregated feature sequence;

[0366] Specifically, the result determination unit can be used to take the fused feature sequence of the aggregated feature sequence and the target text feature sequence as the input of the next processing module group of the current processing module group, so as to continue the content reasoning processing and obtain the image processing task result of the image to be processed.

[0367] Optionally, in some embodiments of this application, the image processing apparatus may further include a model acquisition unit, a grouping unit, and a module adding unit, as follows:

[0368] The model acquisition unit is used to acquire the original processing model, which includes multiple attention modules;

[0369] A grouping unit is used to group multiple attention modules in the original processing model to obtain multiple processing module groups, each processing module group including at least one attention module.

[0370] The module adding unit is used to add feature selection modules and feature aggregation modules to each processing module group to obtain the processing model.

[0371] Optionally, in some embodiments of this application, the grouping unit may be specifically used to determine the number of groups for the original processing model based on computational requirement indication information; and to group multiple attention modules in the original processing model according to the number of groups to obtain multiple processing module groups.

[0372] Optionally, in some embodiments of this application, the result determination unit may specifically include a fusion subunit, a module group determination subunit, and an inference subunit, as follows:

[0373] The fusion subunit is used to fuse the aggregated feature sequence with the target text feature sequence to obtain the fused feature sequence.

[0374] The module group determines the sub-unit, which is used to take the next processing module group of the current processing module group as the new current processing module group and the fused feature sequence as the new feature sequence;

[0375] The inference subunit is used to return to the attention module in the current processing module group that executes the processing model, and to perform attention interaction processing on each feature unit in the feature sequence to obtain the processed feature sequence, so as to obtain the image processing task result of the image to be processed.

[0376] Optionally, in some embodiments of this application, the inference subunit may specifically be used to return to the execution of the attention module in the current processing module group of the processing model, performing attention interaction processing on each feature unit in the feature sequence, until the aggregated feature sequence and target text feature sequence output by the last processing module group in the processing model are obtained; based on the aggregated feature sequence and target text feature sequence output by the last processing module group in the processing model, the target inference result unit for this inference is predicted; based on the aggregated feature sequence and target text feature sequence output by each processing module group of the processing model, the content feature sequence corresponding to each processing module group of the processing model is determined respectively; through the attention module in the processing module group of the processing model, based on the content feature sequence corresponding to the processing module group, attention processing is performed on the target inference result unit to predict the subsequent inference result unit of the target inference result unit; based on the target inference result unit and the subsequent inference result unit, the image processing task result of the image to be processed is determined.

[0377] As can be seen from the above, in this embodiment, the image to be processed can be determined by the feature extraction unit 301; and the image to be processed is subjected to feature extraction processing to obtain an image feature sequence, which includes multiple image feature blocks of the image to be processed; the processing unit 302 processes the image feature blocks in the image feature sequence through a network layer to obtain a target image feature sequence, which includes multiple processed image feature blocks; the classification unit 303 classifies the processed image feature blocks in the target image feature sequence to obtain a feature sequence to be deleted and a feature sequence to be retained, whereby the feature sequence to be deleted includes at least one image feature block to be deleted and the feature sequence to be retained includes at least one image feature block to be retained; the feature aggregation unit 304 aggregates the features of the image feature blocks to be deleted into the feature sequence to be retained to obtain an aggregated feature sequence, which includes the aggregated image feature blocks corresponding to each image feature block to be retained; and the result determination unit 305 obtains the image processing task result of the image to be processed based on the aggregated feature sequence.

[0378] This application can reduce computational overhead and improve content reasoning efficiency by deleting some image feature blocks; at the same time, it can also aggregate the deleted image feature blocks into the retained image feature blocks to avoid the loss of visual information caused by deleting image feature blocks and ensure the accuracy of content reasoning.

[0379] This application also provides an electronic device, such as... Figure 4 The diagram shows a structural schematic of an electronic device involved in an embodiment of this application. This electronic device can be a terminal or a server, specifically:

[0380] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0381] The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 402, and calls data stored in the memory 402, to perform various functions and process data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0382] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0383] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0384] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0385] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0386] The process involves: identifying an image to be processed; extracting features from the image to obtain an image feature sequence, which includes multiple image feature blocks of the image to be processed; processing the image feature blocks in the image feature sequence through a network layer to obtain a target image feature sequence, which includes multiple processed image feature blocks; classifying the processed image feature blocks in the target image feature sequence to obtain a feature sequence to be deleted and a feature sequence to be retained, whereby the feature sequence to be deleted includes at least one image feature block to be deleted and the feature sequence to be retained includes at least one image feature block to be retained; aggregating the features of the image feature blocks to be deleted into the feature sequence to be retained to obtain an aggregated feature sequence, which includes the aggregated image feature blocks corresponding to each image feature block to be retained; and obtaining the image processing result of the image to be processed based on the aggregated feature sequence.

[0387] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0388] As can be seen from the above, this embodiment can determine the image to be processed; perform feature extraction processing on the image to be processed to obtain an image feature sequence, the image feature sequence including multiple image feature blocks of the image to be processed; process the image feature blocks in the image feature sequence through a network layer to obtain a target image feature sequence, the target image feature sequence including multiple processed image feature blocks; classify the processed image feature blocks in the target image feature sequence to obtain a feature sequence to be deleted and a feature sequence to be retained, the feature sequence to be deleted including at least one image feature block to be deleted, the feature sequence to be retained including at least one image feature block to be retained; aggregate the features of the image feature blocks to be deleted into the feature sequence to be retained to obtain an aggregated feature sequence, the aggregated feature sequence including the aggregated image feature blocks corresponding to each image feature block to be retained; and obtain the image processing task result of the image to be processed based on the aggregated feature sequence.

[0389] This application can reduce computational overhead and improve content reasoning efficiency by deleting some image feature blocks; at the same time, it can also aggregate the deleted image feature blocks into the retained image feature blocks to avoid the loss of visual information caused by deleting image feature blocks and ensure the accuracy of content reasoning.

[0390] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0391] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the image processing methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0392] The process involves: identifying an image to be processed; extracting features from the image to obtain an image feature sequence, which includes multiple image feature blocks of the image to be processed; processing the image feature blocks in the image feature sequence through a network layer to obtain a target image feature sequence, which includes multiple processed image feature blocks; classifying the processed image feature blocks in the target image feature sequence to obtain a feature sequence to be deleted and a feature sequence to be retained, whereby the feature sequence to be deleted includes at least one image feature block to be deleted and the feature sequence to be retained includes at least one image feature block to be retained; aggregating the features of the image feature blocks to be deleted into the feature sequence to be retained to obtain an aggregated feature sequence, which includes the aggregated image feature blocks corresponding to each image feature block to be retained; and obtaining the image processing result of the image to be processed based on the aggregated feature sequence.

[0393] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0394] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0395] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the image processing methods provided in the embodiments of this application, the beneficial effects that any of the image processing methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0396] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations of the image processing described above.

[0397] The above provides a detailed description of an image processing method and related equipment provided by the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, characterized in that, include: A determination is made of the image to be processed; and feature extraction processing is performed on the image to be processed to obtain an image feature sequence, wherein the image feature sequence includes multiple image feature blocks of the image to be processed; The image feature blocks in the image feature sequence are processed by a network layer to obtain a target image feature sequence, which includes multiple processed image feature blocks. The processed image feature blocks in the target image feature sequence are classified to obtain a feature sequence to be deleted and a feature sequence to be retained. The feature sequence to be deleted includes at least one image feature block to be deleted, and the feature sequence to be retained includes at least one image feature block to be retained. The features of the image feature blocks to be deleted are aggregated into the feature sequence to be retained to obtain the aggregated feature sequence, which includes the aggregated image feature blocks corresponding to each image feature block to be retained. Based on the aggregated feature sequence, the image processing task result of the image to be processed is obtained.

2. The method according to claim 1, characterized in that, Before processing the image feature blocks in the image feature sequence through the network layer to obtain the target image feature sequence, the method further includes: Obtain the task instruction text for the image to be processed; and perform feature extraction on the task instruction text to obtain a text feature sequence, wherein the text feature sequence includes text feature blocks corresponding to multiple text units in the task instruction text; The image feature sequence and the text feature sequence are fused to obtain a feature sequence; The process of processing image feature blocks in the image feature sequence through a network layer to obtain the target image feature sequence includes: The network layer performs attention interaction processing on each feature unit in the feature sequence to obtain a processed feature sequence. The processed feature sequence includes a target image feature sequence and a target text feature sequence. The target text feature sequence includes processed text feature blocks corresponding to the plurality of text units. The feature unit is an image feature block or a text feature block. The step of classifying the processed image feature blocks in the target image feature sequence to obtain the feature sequence to be deleted and the feature sequence to be retained includes: Based on the processed text feature blocks, the processed image feature blocks in the target image feature sequence are classified to obtain the feature sequence to be deleted and the feature sequence to be retained.

3. The method according to claim 1, characterized in that, The step of aggregating the features of the image feature blocks to be deleted into the feature sequence to be retained, to obtain the aggregated feature sequence, includes: The features of the image feature blocks to be deleted are aggregated into the individual image feature blocks to be retained in the feature sequence to be retained, resulting in the aggregated feature sequence.

4. The method according to claim 1, characterized in that, The step of aggregating the features of the image feature blocks to be deleted into the feature sequence to be retained, to obtain the aggregated feature sequence, includes: Based on the semantic similarity between the image feature blocks to be deleted and the image feature blocks to be retained, a feature relationship matrix is ​​constructed. The elements in the feature relationship matrix represent the association information between the image feature blocks to be deleted and the image feature blocks to be retained at the corresponding positions. Based on the feature relationship matrix, the image feature blocks to be deleted are aggregated into each image feature block to be retained, resulting in an aggregated feature sequence.

5. The method according to claim 4, characterized in that, The step of aggregating the image feature blocks to be deleted into various image feature blocks to be retained based on the feature relationship matrix to obtain an aggregated feature sequence includes: The feature relationship matrix is ​​fused with the feature sequence to be deleted to obtain the fused feature sequence; The fused feature sequence is fused with each image feature block to be retained in the feature sequence to be retained to obtain the aggregated feature sequence.

6. The method according to claim 4, characterized in that, The feature relationship matrix is ​​constructed based on the semantic similarity between the feature blocks of the image to be deleted and the feature blocks of the image to be retained, including: Based on the semantic similarity between every two processed image feature blocks in the target image feature sequence, a feature relationship graph is constructed. The feature relationship graph includes multiple nodes and edges between nodes. Each node represents a processed image feature block, and the edge represents the association information between the two nodes connected to it. Based on the association information between the nodes corresponding to the image feature blocks to be deleted and the image feature blocks to be retained in the feature relationship graph, a feature relationship matrix corresponding to the image feature blocks to be deleted and the image feature blocks to be retained is constructed.

7. The method according to claim 6, characterized in that, The step of constructing a feature relationship matrix corresponding to the image feature blocks to be deleted and the image feature blocks to be retained based on the association information between the nodes corresponding to the image feature blocks to be deleted and the image feature blocks to be retained in the feature relationship graph includes: Based on the connection relationships between nodes in the feature relationship graph, the degree of each node in the feature relationship graph is determined. Based on the degree, the adjacency matrix corresponding to the feature relationship graph is normalized to obtain a processed adjacency matrix. The elements in the processed adjacency matrix represent the association information between two image feature blocks at corresponding positions. The elements corresponding to the image feature blocks to be deleted and the image feature blocks to be retained in the processed adjacency matrix are filtered to obtain the feature relationship matrix corresponding to the image feature blocks to be deleted and the image feature blocks to be retained.

8. The method according to claim 2, characterized in that, The process of performing attention interaction processing on each feature unit in the feature sequence through a network layer to obtain the processed feature sequence includes: The feature sequence is processed by multi-layer attention interaction by multiple network layers to obtain the processed feature sequence output by each layer of attention interaction. The output of the attention interaction of the previous network layer is used as the input of the attention interaction of the next network layer. The step of classifying the processed image feature blocks in the target image feature sequence based on the processed text feature blocks to obtain a feature sequence to be deleted and a feature sequence to be retained includes: From the processed feature sequences output by the attention interaction processing of the multiple network layers, determine the target processed feature sequence output by the attention interaction processing of at least one target network layer; Based on the processed text feature blocks in the at least one target processed feature sequence, the processed image feature blocks in the target image feature sequence are classified to obtain a feature sequence to be deleted and a feature sequence to be retained.

9. The method according to claim 8, characterized in that, The step of classifying the processed image feature blocks in the target image feature sequence based on the processed text feature blocks in the at least one target processed feature sequence to obtain a feature sequence to be deleted and a feature sequence to be retained includes: For each processed image feature block in the target image feature sequence, determine the attention information between the processed text feature block in each target processed feature sequence and the processed image feature block; The attention information between the processed text feature blocks and the processed image feature blocks in all target processed feature sequences is fused to obtain the target attention information of the processed image feature blocks. Based on the target attention information, the processed image feature blocks in the target image feature sequence are classified to obtain the feature sequence to be deleted and the feature sequence to be retained.

10. The method according to claim 9, characterized in that, The target processed feature sequence output by the attention interaction processing of the target network layer includes the processed sub-feature sequences output by the target network layer in multiple subspaces, and the processed sub-feature sequences include the processed text feature blocks of the corresponding subspaces. The process of fusing the attention information between the processed text feature blocks and the processed image feature blocks in all processed feature sequences to obtain the target attention information of the processed image feature blocks includes: For each target network layer, the attention information of the processed text feature blocks output by the target network layer in each subspace and the processed image feature blocks are fused to obtain the attention information of the processed text feature blocks and the processed image feature blocks of the target network layer. The attention information of the processed text feature blocks of all target network layers is fused with the attention information of the processed image feature blocks to obtain the target attention information of the processed image feature blocks.

11. The method according to claim 2, characterized in that, The process of performing attention interaction processing on each feature unit in the feature sequence through a network layer to obtain the processed feature sequence includes: By performing attention interaction processing on each feature unit in the feature sequence through the network layer in the attention module of the current processing module group of the processing model, a processed feature sequence is obtained. The processing model includes multiple processing module groups, and each processing module group includes an attention module, a feature selection module, and a feature aggregation module. The step of classifying the processed image feature blocks in the target image feature sequence based on the processed text feature blocks to obtain a feature sequence to be deleted and a feature sequence to be retained includes: The feature selection module in the current processing module group classifies the processed image feature blocks in the target image feature sequence based on the processed text feature blocks to obtain the feature sequence to be deleted and the feature sequence to be retained. The step of aggregating the features of the image feature blocks to be deleted into the feature sequence to be retained, to obtain the aggregated feature sequence, includes: The feature aggregation module in the current processing module group aggregates the features of the image feature blocks to be deleted into the feature sequence to be retained, thereby obtaining the aggregated feature sequence. The step of obtaining the image processing task result of the image to be processed based on the aggregated feature sequence includes: The fused feature sequence of the aggregated feature sequence and the target text feature sequence is used as the input of the next processing module group of the current processing module group to continue content reasoning processing and obtain the image processing task result of the image to be processed.

12. The method according to claim 11, characterized in that, The method further includes: Obtain the original processing model, which includes multiple attention modules; The original processing model is grouped into multiple processing module groups, each of which includes at least one attention module. Add a feature selection module and a feature aggregation module to each processing module group to obtain the processing model.

13. The method according to claim 12, characterized in that, The process of grouping multiple attention modules in the original processing model to obtain multiple processing module groups includes: Based on the computational requirements, determine the number of groups for the original processing model; Based on the number of groups, the multiple attention modules in the original processing model are grouped to obtain multiple processing module groups.

14. The method according to claim 11, characterized in that, The step of using the fused feature sequence of the aggregated feature sequence and the target text feature sequence as input to the next processing module group of the current processing module group to continue content reasoning processing and obtain the image processing task result of the image to be processed includes: The aggregated feature sequence is fused with the target text feature sequence to obtain the fused feature sequence; The next processing module group of the current processing module group is taken as the new current processing module group, and the fused feature sequence is taken as the new feature sequence. The process involves returning to the attention module in the current processing module group of the processing model, performing attention interaction processing on each feature unit in the feature sequence to obtain the processed feature sequence, thereby obtaining the image processing task result of the image to be processed.

15. The method according to claim 14, characterized in that, The step of returning to execute the attention module in the current processing module group of the processing model, performing attention interaction processing on each feature unit in the feature sequence to obtain the processed feature sequence, in order to obtain the image processing task result of the image to be processed, includes: Return to the step of performing attention interaction processing on each feature unit in the feature sequence in the current processing module group of the processing model, until the aggregated feature sequence and the target text feature sequence output by the last processing module group in the processing model are obtained; Based on the aggregated feature sequence and target text feature sequence output by the last processing module group in the processing model, predict the target inference result unit for this inference. Based on the aggregated feature sequence and the target text feature sequence output by each processing module group of the processing model, the content feature sequence corresponding to each processing module group of the processing model is determined respectively. By using the attention module in the processing module group of the processing model, attention processing is performed on the target inference result unit based on the content feature sequence corresponding to the processing module group, so as to predict the subsequent inference result units of the target inference result unit. Based on the target reasoning result unit and the subsequent reasoning result unit, the image processing task result of the image to be processed is determined.

16. An image processing apparatus, characterized in that, include: A feature extraction unit is used to determine the image to be processed and to perform feature extraction processing on the image to be processed to obtain an image feature sequence, wherein the image feature sequence includes multiple image feature blocks of the image to be processed. The processing unit is configured to process the image feature blocks in the image feature sequence through a network layer to obtain a target image feature sequence, wherein the target image feature sequence includes multiple processed image feature blocks; A classification unit is used to classify the processed image feature blocks in the target image feature sequence to obtain a feature sequence to be deleted and a feature sequence to be retained. The feature sequence to be deleted includes at least one image feature block to be deleted, and the feature sequence to be retained includes at least one image feature block to be retained. The feature aggregation unit is used to aggregate the features of the image feature blocks to be deleted into the feature sequence to be retained, thereby obtaining an aggregated feature sequence, wherein the aggregated feature sequence includes the aggregated image feature blocks corresponding to each image feature block to be retained. The result determination unit is used to obtain the image processing task result of the image to be processed based on the aggregated feature sequence.

17. An electronic device, characterized in that, It includes a memory and a processor; the memory stores an application program, and the processor runs the application program within the memory to perform the operations in the image processing method according to any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the image processing method according to any one of claims 1 to 15.

19. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the image processing method according to any one of claims 1 to 15.

Images