Image recognition method and device and storage medium

Abstract

The embodiment of the invention provides an image recognition method and device and a storage medium, which can be applied to a high-resolution medical image recognition task, and the method comprises the steps: obtaining N intermediate image feature sequences of a to-be-recognized image through N Transform modules in a large-scale pre-training model, enabling the N intermediate image feature sequences to correspond to N fusion modules of an input side fusion network model, inputting a prompt vector to the input side fusion network model, helping the fusion modules to understand and process the intermediate image feature sequences through the prompt vector, and outputting a target image feature sequence of the to-be-recognized image through the N fusion modules. And finally obtaining an identification result of the to-be-identified image based on the target image feature sequence. According to the scheme, the parameters of the large-scale pre-training model can be frozen in the training stage, and only the side fusion network model and the prompt vector are subjected to parameter updating, so that the resource consumption in the training stage is reduced, and the capability of the model in the aspect of processing high-resolution images in the training process is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular, to an image recognition method, device, and storage medium. Background Art

[0002] Large-scale pre-trained models (LPMs) are pre-trained on a large amount of data to learn common language, vision or other feature representations, and then fine-tuned on specific tasks to achieve excellent performance. Parameter-efficient fine-tuning (PEFT) is a common fine-tuning strategy. Its core idea is to intelligently select the subset of parameters that need to be updated in the model, or introduce a small number of new parameters related to the task, thereby significantly reducing the resource consumption of model training without sacrificing performance.

[0003] However, there are still some problems when applying large-scale pre-trained models to image recognition tasks. This is mainly reflected in the fact that when processing high-resolution image recognition (such as recognizing high-resolution medical images with complex detail information), the PEFT method still fails to effectively solve the problem of high resource consumption in model training, which limits the performance of large-scale pre-training in high-resolution image recognition tasks. In other words, the prediction accuracy of large-scale pre-trained models when applied to high-resolution image recognition tasks needs to be improved. Summary of the invention

[0004] The embodiments of the present application provide an image recognition method, device and storage medium, aiming to improve the accuracy of recognition results for high-resolution images.

[0005] In a first aspect, an embodiment of the present application provides an image recognition method, the method comprising the following steps: Inputting the image to be recognized into a large-scale pre-trained model, obtaining intermediate image feature sequences respectively output by N Transformer modules in the large-scale pre-trained model, wherein N is an integer greater than 1, and the N Transformer modules are connected in sequence; Inputting a preset prompt vector and the intermediate graph feature sequence into a trained side fusion network model to obtain a target graph feature sequence, wherein the side fusion network model includes N fusion modules connected in sequence, the N fusion modules correspond to the N Transformer modules one by one, the intermediate graph feature sequence output by each Transformer module is configured to be input into the corresponding fusion module, and the preset prompt vector is configured to be input into the first fusion module of the N fusion modules; The target image feature sequence is input into a trained classification network model to obtain a recognition result of the image to be recognized.

[0006] In the scheme of the embodiment of the present application, N intermediate graph feature sequences of the image to be identified are first obtained through N Transformer modules in the large-scale pre-trained model, and then the N intermediate graph feature sequences are corresponding to the N fusion modules of the input-side fusion network model, and the prompt vector is input to the input-side fusion network model to help the fusion module understand and process the intermediate graph feature sequence through the prompt vector, and then the target graph feature sequence of the image to be identified is output through the N fusion modules, and finally the recognition result of the image to be identified is obtained based on the target graph feature sequence. The system architecture adopted by this scheme can freeze the parameters of the large-scale pre-trained model during the training stage, and only update the parameters of the side fusion network model and the prompt vector, thereby reducing the resource consumption in the training stage, helping to improve the model's ability to process high-resolution images during the training process, and thus improving the recognition accuracy of high-resolution images in the high-application stage.

[0007] In a possible implementation method, each of the fusion modules is configured to perform cross-attention calculation based on the input query vector, key vector, and value vector, and output an updated intermediate graph feature sequence; The first fusion module among the N fusion modules uses the prompt vector as the query vector, and the other fusion modules use the updated intermediate graph feature sequence output by the previous fusion module as the query vector; Each of the fusion modules uses the intermediate graph feature sequence output by the corresponding Transformer module as the key vector and the value vector; The target graph feature sequence is the updated intermediate graph feature sequence output by the last fusion module among the N fusion modules.

[0008] In a possible implementation method, the cross-attention calculation includes the following steps: Normalizing the query vector and the key vector to obtain an attention map; The attention map is multiplied by the value vector to obtain the updated intermediate map feature sequence.

[0009] In a possible implementation method, the large-scale pre-trained model further includes an image segmentation module and an encoding module. The step of inputting the image to be recognized into the trained large-scale pre-trained model to obtain the intermediate image feature sequences respectively output by N Transformer modules in the large-scale pre-trained model includes: performing the following steps through the large-scale pre-trained model: Dividing the image to be identified into tags by the image division module; Encode each of the tags respectively by the encoding module to obtain an embedded representation sequence, wherein the embedded representation sequence includes a plurality of embedded representations corresponding to the tags; Each of the Transformer modules performs self-attention calculation based on the input sequence, and outputs a corresponding updated embedding representation sequence, wherein the input sequence of the first Transformer module among the N Transformer modules is the embedding representation sequence, and the input sequences of the other Transformer modules are the updated embedding representation sequence output by the previous Transformer module; According to the updated embedding representation sequence output by each of the Transformer modules, the intermediate graph feature sequence corresponding to the output of each of the Transformer modules is determined.

[0010] In a possible implementation method, determining the intermediate graph feature sequence outputted by each Transformer module according to the updated embedding representation sequence outputted by each Transformer module includes: Determine an average attention score obtained by each of the markers in the self-attention calculation of the Transformer module; determining important markers based on the average attention score; The intermediate graph feature sequence is determined based on the updated embedding representations corresponding to all the important tags.

[0011] In a possible implementation method, determining the important tag according to the average attention score includes: The K tags with the highest average attention scores are used as the important tags, where K is a preset value or is determined according to a preset ratio.

[0012] In a possible implementation method, the prompt vector and the side fusion network model are trained by the following steps: Acquire training data, the training data comprising a plurality of sample images and identification labels corresponding to the sample images; Freezing the parameters of the large-scale pre-trained model, inputting the sample image into the large-scale pre-trained model, and obtaining the sample intermediate image feature sequences respectively output by N Transformer modules in the large-scale pre-trained model; Inputting the learnable hint vector and the sample intermediate graph feature sequence into the original side fusion network model to obtain the sample target graph feature sequence; Freezing the parameters of the classification network model, inputting the sample target image feature sequence into the classification network model, and obtaining the recognition result of the sample image; Determining a predicted loss value according to the recognition results and recognition labels of each of the sample images; updating the learnable hint vector and the parameters of the side fusion network model according to the predicted loss value; Continue to train the learnable prompt vector and the side fusion network model based on the training data until the prediction loss value converges; After the prediction loss value converges, the current learnable prompt vector is used as the prompt vector, and the current original side fusion network model is used as the side fusion network model.

[0013] In a second aspect, an embodiment of the present application provides an image recognition device, the device comprising: An intermediate image feature sequence determination unit is used to input the image to be recognized into a trained large-scale pre-training model to obtain the intermediate image feature sequences respectively output by N Transformer modules in the large-scale pre-training model, wherein N is an integer greater than 1, and the N Transformer modules are connected in sequence; A target graph feature sequence determination unit is used to input a preset prompt vector and the intermediate graph feature sequence into a trained side fusion network model to obtain a target graph feature sequence, wherein the side fusion network model includes N fusion modules connected in sequence, the N fusion modules correspond to the N Transformer modules one by one, the intermediate graph feature sequence output by each Transformer module is configured to be input into the corresponding fusion module, and the preset prompt vector is configured to be input into the first fusion module among the N fusion modules; The recognition result determination unit is used to input the target image feature sequence into the trained classification network model to obtain the recognition result of the image to be recognized.

[0014] In a third aspect, an embodiment of the present application provides a computer device, including: Memory, used to store programs; A processor, configured to execute the program stored in the memory. When the processor executes the program stored in the memory, the processor is configured to execute: a method as described in any one of claims 1 to 6.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute the method described in the first aspect above.

[0016] The solutions provided in the second to fourth aspects are used to implement or cooperate with the method provided in the first aspect, and thus can achieve the same or corresponding beneficial effects as the first aspect, and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a system architecture provided for an embodiment of the present application; Figure 2 A flowchart of an image recognition method provided by one embodiment of the present application; Figure 3 A schematic diagram of a method for determining an intermediate graph feature sequence based on an important marker selection mechanism provided in an embodiment of the present application; Figure 4 A schematic diagram of a method for cross-attention calculation provided for one embodiment of the present application; Figure 5 A schematic diagram of an image recognition device provided by an embodiment of the present application; Figure 6 A schematic diagram of a computer device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical methods and advantages of this application more clear, the following is a further detailed description of this application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain this application and are not used to limit this application.

[0019] It should be noted that the meaning of "multiple" (or multiple) involved in the description of the embodiments of the present application is more than two, "greater than", "less than", "exceed", etc. are understood as not including the number itself, and "above", "below", "within", etc. are understood as including the number itself. If there is a description of "first", "second", etc., it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0020] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the situation where A exists alone, A and B exist at the same time, and B exists alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b and c can mean: a exists alone, b exists alone, c exists alone, a and b exist at the same time, a and c exist at the same time, b and c exist at the same time, or a, b and c exist at the same time, wherein a, b, c can be single or multiple.

[0021] It should be noted that the information interaction, execution process, etc. between the above-mentioned modules / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0022] With the booming development of deep learning technology, transfer learning has become one of the key technologies to promote the progress of artificial intelligence. In the field of computer vision, through transfer learning, researchers can use large-scale pre-trained models (LPMs) on large-scale datasets to quickly adapt to and solve a variety of downstream tasks. These models have mastered rich feature extraction skills through pre-training on large-scale datasets such as ImageNet, providing strong support for solving specific problems.

[0023] Based on transfer learning, fine-tuning technology allows the model to adapt to new task requirements by adjusting some parameters while maintaining pre-training knowledge. This method significantly improves training efficiency while maintaining the generalization ability of the model. However, the fine-tuning process is often accompanied by the update of all parameters of the model, which leads to a large consumption of computing resources and memory resources, especially when processing large-scale data sets.

[0024] To address this challenge, the parameter-efficient transfer learning (PETL) strategy was proposed. The core of the PETL strategy is to intelligently select the subset of parameters that need to be updated in the model, or introduce a small number of new parameters related to the task, thereby significantly reducing the resource consumption of model training without sacrificing performance. This method not only reduces the computational burden, but also makes it possible to deploy high-performance models in resource-constrained environments.

[0025] Currently, the PETL method is widely used in the field of natural image processing and has demonstrated its potential in improving efficiency and performance. However, for fields that require high-resolution image processing, such as medical image analysis, the application of PETL is relatively small. Medical images such as X-rays, computed tomography (CT) images, and magnetic resonance imaging (MRI) usually have high resolution and complex detail information. Diagnostic clues are often distributed in different parts of the image, requiring high-resolution input to obtain fine-grained information. This means that the processing of high-resolution medical images requires not only powerful computing power, but also very high memory requirements, which limits the application of PETL.

[0026] Existing parameter efficient fine-tuning (PEFT) methods face multiple challenges when applied to high-resolution medical image classification tasks. The use of high-resolution images significantly increases the GPU memory requirements and computational burden during training. Although methods such as Prompt Tuning, Adapter, BitFit, and LoRA have shown promising results in reducing the number of parameters that need to be updated by the model, they still do not effectively address the memory consumption issue when dealing with high-resolution inputs.

[0027] For example, hint tuning methods help models adapt to new tasks by adding additional hint vectors to the input data, but this method usually does not work well when processing medical images, which often contain a lot of detailed information that needs to be captured at high resolution. Adapter methods introduce additional adaptation modules in certain layers of the model, which reduces the overall need for parameter updates, but still faces the problem of high memory usage when processing high-resolution images. On the other hand, BitFit and LoRA reduce training costs by only updating a small number of parameters or using low-rank matrix factorization, but these methods still have difficulty significantly reducing memory usage when processing large-sized inputs.

[0028] In the high-resolution medical image classification task, there have been a variety of transfer learning techniques that attempt to improve memory and computational efficiency, but each of these methods has its own limitations. The following are several existing methods related to the present invention: Full model fine-tuning: Full model fine-tuning is the most direct method, which adjusts the parameters of the entire pre-trained model to adapt it to the new task requirements. Although this method usually achieves the best performance, it has very high computational and memory overhead when processing high-resolution medical images. Especially under high-resolution input, each layer of the model needs to be updated, resulting in huge GPU memory requirements during training, which may reach hundreds of GB, and extremely high computational costs.

[0029] Linear Probing: Linear Probing only adds a new classification head to the last layer of the pre-trained model and only adjusts the parameters of this part. Although this method greatly reduces the number of parameters that need to be updated, thereby reducing the computational cost, its classification performance is usually not as good as fine-tuning the full model because it cannot fully utilize the deep features of the pre-trained model.

[0030] Prompt Tuning: Prompt Tuning helps pre-trained models adapt to new tasks by adding special prompts to the input data. Although it has achieved some success in the field of natural language processing, in medical image classification tasks, the details and complexity of high-resolution images make it difficult for this method to significantly reduce memory requirements because it cannot effectively process a large amount of high-resolution information.

[0031] Adapter: The adapter method inserts small adaptation modules in the middle layer of the pre-trained model. These modules are specially designed for new tasks and trained independently. Although the adapter method reduces the number of parameters that need to be updated, thereby reducing some computational costs, for the processing of high-resolution images, the model still needs to process a large amount of data, resulting in high memory usage.

[0032] Low-Rank Adaptation (LoRA): The LoRA method reduces the training cost by fine-tuning a small part of the model parameters using low-rank matrix decomposition. However, although this method performs well in natural language tasks, in high-resolution medical image tasks, the high complexity of the input data makes the memory consumption still high, and it cannot completely solve the problem caused by high-resolution input.

[0033] Although the above methods have improved parameters and computational efficiency in some aspects, there is still a lot of room for optimization in GPU memory and computational efficiency in high-resolution medical image classification tasks. Therefore, we propose a new Fusion Side Tuning (FST) method to more effectively utilize the features of the pre-trained model, thereby significantly reducing computational and memory overhead while maintaining high performance.

[0034] It can be seen that although the existing parameter efficient transfer learning (PETL) methods have achieved certain success in reducing computational and memory overhead, they still have the following major shortcomings in high-resolution medical image classification tasks: High memory consumption: Although existing PEFT methods (such as Hint Tuning, Adapter, BitFit, LoRA, etc.) have achieved some success in reducing the number of update parameters, they still have serious deficiencies in GPU memory usage when processing high-resolution images. Since large-scale pre-trained models (LPMs) usually still need to process a large number of high-dimensional inputs and generate intermediate features, these methods cannot effectively reduce the memory usage caused by high-resolution inputs.

[0035] High computational cost: Although the full model fine-tuning method can maximize the performance of the model, it requires updating all parameters, resulting in very high computational cost. Other methods, such as linear probe or prompt tuning, although they reduce some of the computational workload, are still computationally inefficient when processing complex high-resolution medical images and require long training times.

[0036] Limited performance: Existing PEFT methods reduce computational overhead to some extent, but their performance is usually lower than full model fine-tuning. Especially in high-resolution medical image classification, these methods cannot fully utilize all the features of the model, resulting in reduced classification accuracy. In particular, for small but important diagnostic clues in medical images, existing methods have difficulty maintaining high performance while reducing memory consumption.

[0037] Not applicable to high-resolution medical images: Most existing PEFT methods are designed for natural image processing and do not consider the specific needs of medical images. Medical images usually require higher resolution to capture subtle diagnostic information, which makes traditional PEFT methods limited in effectiveness in high-resolution scenarios.

[0038] In view of this, the embodiments of the present application provide an image recognition method, device and storage medium, aiming to improve the accuracy of recognition results for high-resolution images. The core idea of ​​the embodiments of the present application is to combine a large-scale pre-trained model (LPM) and a lightweight side fusion network (Side Fusion Network) to reduce memory consumption while maintaining high performance.

[0039] To facilitate understanding of the solutions of the embodiments of the present application, the relevant concepts involved in the embodiments of the present application are introduced below.

[0040] (1) Attention Mechanism: It is a technique in deep learning that imitates the function of human visual attention, enabling the model to focus on the important parts of the input data and ignore unimportant information. This mechanism has been proven to be very effective in a variety of tasks, especially in the fields of natural language processing (NLP) and computer vision (CV).

[0041] The components of the attention mechanism include: Query: the information you want to obtain; Key: A candidate set that may contain the required information; Value: The actual information associated with a key.

[0042] Workflow of the attention mechanism Calculate attention score: Use the query and key to calculate the attention score, usually through a dot product or join operation; Standardization: The scores are standardized by the normalized softmax function so that the sum of all scores is 1; Weighted summation: Use the normalized attention scores to perform weighted summation on the values ​​to get the final output.

[0043] (2) Self-Attention: This is a technique that allows the model to focus on the correlation between different positions within the sequence when processing the input sequence. It allows the model to consider the information of the entire input sequence at the same time, thereby capturing richer contextual information. The self-attention mechanism calculates the similarity (or attention score) between each element in the sequence and other elements, and then performs a weighted summation of the sequence based on these similarities. Its advantages lie in its global vision and parallel computing capabilities, and it performs well in natural language processing tasks (such as machine translation and text classification).

[0044] (3) Cross-Attention: It is a mechanism that establishes associations between two different input sequences and calculates attention weights. Unlike the self-attention mechanism, the cross-attention mechanism focuses on the interaction between two different input sequences. In cross-attention, the elements of one sequence serve as queries, while the elements of the other sequence serve as keys and values, allowing the model to refer to the information of another sequence when processing one sequence. This mechanism is particularly important in tasks such as machine translation and text summarization, because it can help the model better understand the relationship between the source language and the target language and improve the quality of translation or summary.

[0045] (4) Embedding: It is a method of "representing" an object with a numerical vector. The object here can be a word, an item, an image, etc. An object can be represented by a vector because the distance between this vector and other object vectors reflects the similarity of these objects. Furthermore, the distance vector between two vectors can even reflect the relationship between them.

[0046] (5) Transformer module: It is a deep learning model that is widely used in the fields of natural language processing (NLP) and computer vision (CV). Its core idea is to use the self-attention mechanism to capture long-distance dependencies in sequence data.

[0047] The image recognition method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms; the software can be an application that implements the image recognition method, etc., but is not limited to the above forms.

[0048] Embodiments of the present application can be used in numerous general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0049] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on the attribute information or attribute information set of an object (such as a user, etc.) and other data related to the characteristics of the object, the permission or consent of the corresponding object will be obtained first, and the collection, use and processing of these data will comply with relevant laws and standards. In addition, when the embodiment of the present invention needs to obtain the attribute information of an object, it will obtain the separate permission or separate consent of the corresponding object through a pop-up window or jump to a confirmation page, and after clearly obtaining the separate permission or separate consent of the corresponding object, it will obtain the relevant data of the object necessary for the normal operation of the embodiment of the present invention.

[0050] See also Figure 1 , is a schematic diagram of a system architecture provided by an embodiment of the present application, the system architecture includes a large-scale pre-trained model, a lightweight side fusion network model and a classification network model, wherein the large-scale pre-trained model includes N sequentially connected Transformer modules, N is an integer greater than 1, and each Transformer module is configured to output an intermediate graph feature sequence of an image to be identified; the side fusion network model includes N sequentially connected fusion modules, the N fusion modules are in a one-to-one correspondence with the N Transformer modules, the intermediate graph feature sequence output by the Transformer module is input into the corresponding fusion module, and the last fusion module of the side fusion network model outputs a target graph feature sequence of the image to be identified; the classification network model is configured to receive the target graph feature sequence, and output a recognition result of the image to be identified based on the target graph feature sequence.

[0051] It is worth mentioning that the input of the side fusion network model also includes a prompt vector, which is a learnable embedding vector, that is, the parameters in the prompt vector can be updated through training. The prompt vector is used to help the fusion network model understand and process feature representation.

[0052] See also Figure 2 , is a flow chart of an image recognition method provided by an embodiment of the present application. The method is applied to Figure 1 In the system architecture shown in Figure 2 As shown, the method comprises the following steps: Step S110: input the image to be recognized into the large-scale pre-trained model to obtain the intermediate image feature sequences respectively output by N Transformer modules in the large-scale pre-trained model; Step S120, inputting the preset prompt vector and the intermediate graph feature sequence into the trained side fusion network model to obtain the target graph feature sequence, wherein the preset prompt vector is configured to be input into the first fusion module among the N fusion modules; Step S130: input the target image feature sequence into the trained classification network model to obtain the recognition result of the image to be recognized.

[0053] It should be noted that the large-scale pre-trained model in the embodiment of the present application is a neural network model pre-trained on a large-scale data set (such as ImageNet). In the embodiment of the present application, the weights of the large-scale pre-trained model remain frozen and are not updated. The main function of the large-scale pre-trained model is to extract rich fine-grained features from high-resolution input images. These feature representations are crucial in medical image recognition tasks because diagnostic clues in medical images are usually distributed throughout the image and require high resolution to be effectively captured.

[0054] In specific implementation, the high-resolution image to be identified can be first divided into a series of tokens, and then each token is encoded separately to obtain the embedding representation corresponding to each token. The embedding representations of these tokens constitute an embedding representation sequence. The embedding representation sequence is input into the first Transformer module, so that the first Transformer module performs self-attention calculation based on the input embedding representation sequence and outputs the updated embedding representation sequence. For the subsequent second to Nth Transformer modules, the updated embedding representation sequence output by the previous Transformer module is used as the input sequence, and then self-attention calculation is performed based on the input sequence, and the updated embedding representation sequence is output. The intermediate image feature sequence input by the Transformer module to the corresponding fusion module is determined based on the updated embedding representation sequence output by itself, and the intermediate image feature sequence contains rich fine-grained features of the image to be identified.

[0055] It should be noted that in the embodiment of the present application, multiple Transformer modules can be used to obtain the feature representations corresponding to the image to be identified in different subspaces, which is very helpful for understanding the overall information and contextual information of the image to be identified, thereby achieving the effect of enhancing the model's perception of image details.

[0056] It should be noted that the side fusion network model of the embodiment of the present application works in parallel with the large-scale pre-trained model. The structure of the side fusion network model is relatively lightweight compared to the large-scale pre-trained model and is designed for efficient processing of high-resolution images. The side fusion network model effectively utilizes the knowledge of the large-scale pre-trained model, receives the intermediate image feature sequence from the large-scale pre-trained model, and further processes the intermediate image feature sequence to improve image recognition performance.

[0057] It should be noted that the side fusion network model of the embodiment of the present application introduces a hint vector to process the intermediate image feature sequence, and the hint vector is used to help the model understand and process the intermediate image feature sequence, so that the model can further screen out key features that are helpful for image recognition. The hint vector is a learnable embedding vector that contains trainable parameters. These parameters are updated through back propagation during the training process to meet the feature extraction requirements of high-resolution image recognition tasks.

[0058] It should be noted that in some embodiments of the present application, the large-scale pre-training model further includes an image segmentation module and an encoding module, wherein the image segmentation module is used to divide the image to be identified into It should be noted that in some embodiments of the present application, the image to be recognized is input into a trained large-scale pre-training model to obtain the intermediate image feature sequences respectively output by N Transformer modules in the large-scale pre-training model, including performing the following steps S111-S114 through the large-scale pre-training model: Step S111: divide the image to be identified into multiple tags through an image division module.

[0059] It should be noted that the embodiment of the present application divides the image to be identified into a series of marks through the image segmentation module, and these marks can be pixel blocks of the image to be identified. In specific implementation, the image to be identified can be segmented according to a preset pixel block size, such as 2*2 pixels, to obtain multiple pixel blocks, each of which is used as a mark.

[0060] Step S112: Encode each tag separately through the encoding module to obtain an embedded representation sequence, where the embedded representation sequence includes multiple embedded representations corresponding to the tags.

[0061] It should be noted that in the embodiment of the present application, the tag is used as the smallest unit for feature extraction, which is first encoded into an embedded representation and then input into the Transformer module for feature extraction. The embedded representations of all tags of the image to be recognized constitute an embedded representation sequence.

[0062] Step S113: Each Transformer module performs self-attention calculation based on the input sequence, and outputs the corresponding updated embedding representation sequence, wherein the input sequence of the first Transformer module among the N Transformer modules is the embedding representation sequence, and the input sequences of other Transformer modules are the updated embedding representation sequences output by the previous Transformer module.

[0063] It should be noted that in the embodiment of the present application, the embedding representation sequence is processed by the Transformer module to capture the long-distance dependencies in the sequence data and generate a rich feature representation. This feature representation is the above-mentioned updated embedding representation sequence.

[0064] In specific implementation, the embedding representation sequence is input into the first Transformer module among the N Transformer modules. The first Transformer module uses the embedding representation sequence as the input sequence, performs self-attention calculation based on the input sequence, obtains an updated embedding representation sequence, and outputs the updated embedding representation sequence. For other Transformer modules, the updated embedding representation sequence output by the previous Transformer module is used as the input sequence, and self-attention calculation is performed based on the input sequence to obtain an updated embedding representation sequence.

[0065] Specifically, each Transformer module performs self-attention calculation based on the input sequence and outputs the corresponding updated embedding representation sequence, which specifically includes the following steps performed by each Transformer module: linearly transform the input sequence to generate three matrices of query (Q), key (K) and value (V), which are obtained by multiplying the input sequence with three weight matrices (corresponding to Q, K, and V respectively). The weight matrix is ​​a trainable parameter, and its dimension is usually the input dimension (d_model) and the dimension (d_k) of the Q / K / V vector; the dot product operation is used to calculate the similarity between the query matrix Q and the key matrix K to obtain an attention map (the attention map can also be called the attention score matrix), in which each element represents the degree of attention of one element to another element in the input sequence (i.e., the attention score); in order to prevent the dot product result from being too large and causing the gradient vanishing problem, the dot product result will be divided by a scaling factor (usually the square root of the key vector dimension d_k); the attention score is converted into a probability distribution through the Softmax function, so that the sum of the attention weights of each element is 1; finally, the value matrix V is weighted and summed using the attention weights obtained by the Softmax function to obtain the final output sequence.

[0066] Step S114: Determine the intermediate graph feature sequence output by each Transformer module according to the updated embedding representation sequence output by each Transformer module.

[0067] It should be noted that in some embodiments of the present application, the updated embedding representation sequence output by the Transformer module is directly used as the intermediate graph feature sequence. In other embodiments of the present application, important tokens are determined from all tokens based on an important token selection mechanism, and the intermediate graph feature sequence is generated based on the updated embedding representations corresponding to all important tokens.

[0068] It should be noted that in some embodiments of the present application, according to the updated embedding representation sequence output by each Transformer module, determining the intermediate graph feature sequence output by each Transformer module includes the following steps: Step S1141, determine the average attention score obtained by each tag in the self-attention calculation of the Transformer module.

[0069] In specific implementation, the average attention score of each tag can be determined based on the attention map obtained from the self-attention calculation. Each element in the attention map (attention score matrix) represents the degree of attention (i.e., attention score) of one element (tag) in the input sequence to another element. Each row / column of the attention map corresponds to an element (tag) in the input sequence. The average of the attention scores on a row / column of the attention map can be used to obtain the average attention score of the corresponding tag.

[0070] Step S1142, determining important tags according to the average attention score; It should be noted that in some embodiments of the present application, determining the important tags according to the average attention score includes: taking the K tags with the highest average attention scores as important tags, where K is a preset value or is determined according to a preset ratio. For example, taking the 20 tags with the highest average attention scores as important tags, or taking the 20% tags with the highest average attention scores as important tags.

[0071] Step S1143: Determine an intermediate graph feature sequence based on the updated embedding representations corresponding to all important tags.

[0072] See also Figure 3 , is a schematic diagram of a method for determining an intermediate graph feature sequence based on an important marker selection mechanism provided in an embodiment of the present application. Figure 3As shown, each Transformer module performs linear transformation based on the input sequence to generate three vectors: query (Q), key (K) and value (V). The query vector Q and the key vector K obtain an attention map through dot product operation. Each element in the attention map represents the attention score of one element to another element in the input sequence. The value vector V is weighted and summed using the attention score to obtain the final output. This output is the updated embedding representation sequence, including the updated embedding representation corresponding to each tag. In the embodiment of the present application, the average attention score of each tag is determined according to the attention map, and the K tags with the highest scores are taken as important tags. The embedding representations of these important tags can be considered as key feature representations for obtaining accurate image recognition results. Therefore, the embodiment of the present application passes the updated embedding representations corresponding to the important tags to the fusion module for processing, and the updated embedding representations corresponding to other tags are not passed to the fusion module. This can improve the processing efficiency of the fusion module for features while ensuring the accuracy of image recognition.

[0073] Taking medical images as an example, medical images usually contain a lot of detailed information, but only a small number of areas are important for diagnosis. In order to improve computing and memory efficiency, the embodiment of the present application introduces an important marker selection mechanism, which selects the marker with the highest score in the attention map, retains the most important diagnostic information while significantly reducing memory consumption.

[0074] It should be noted that, in the embodiment of the present application, each fusion module is configured to perform cross-attention calculation based on the input query vector, key vector and value vector, and output an updated intermediate graph feature sequence; the first fusion module among the N fusion modules uses the prompt vector as the query vector, and the other fusion modules use the updated intermediate graph feature sequence output by the previous fusion module as the query vector; each fusion module uses the intermediate graph feature sequence output by the corresponding Transformer module as the key vector and value vector; the target graph feature sequence is the updated intermediate graph feature sequence output by the last fusion module among the N fusion modules.

[0075] Specifically, the cross-attention calculation in the embodiment of the present application includes the following steps S121 and S122: Step S121, normalize the query vector and the key vector to obtain an attention map; Step S122: multiply the attention map by the value vector to obtain an updated intermediate map feature sequence.

[0076] See also Figure 4 , is a schematic diagram of cross-attention calculation provided by an embodiment of the present application. Figure 4As shown in the figure, the first fusion module uses the input prompt vector as the query vector (Q), the input intermediate graph feature sequence as the key vector (K) and the value vector (V), and performs normalization calculation (softmax) on the query vector (Q) and the key vector (K) to obtain the attention map; the value vector (V) is weighted summed based on the score of the attention map to obtain the final output (updated intermediate graph feature sequence), and the output of the first fusion module is used as the query vector (Q) of the next fusion module. Subsequent fusion modules use the updated intermediate graph feature sequence output by the previous fusion module as the query vector (Q), and the intermediate graph feature sequence output by the corresponding Transformer module as the key vector (K) and the value vector (V) to perform cross-attention calculations.

[0077] It should be noted that during the training of the side fusion network model in the embodiment of the present application, the back-propagation gradient will not pass through a large-scale pre-training model, thus significantly saving memory overhead in the training phase.

[0078] It should be noted that in some embodiments of the present application, the prompt vector and the side fusion network model are trained and obtained by the following steps: Step S310: acquiring training data, where the training data includes a plurality of sample images and identification labels corresponding to the sample images; Step S320, freezing the parameters of the large-scale pre-training model, inputting the sample image into the large-scale pre-training model, and obtaining the sample intermediate image feature sequences respectively output by the N Transformer modules in the large-scale pre-training model; Step S330, inputting the learnable hint vector and the sample intermediate graph feature sequence into the original side fusion network model to obtain the sample target graph feature sequence; Step S340, freezing the parameters of the classification network model, inputting the sample target image feature sequence into the classification network model, and obtaining the recognition result of the sample image; Step S350: determining a predicted loss value according to the recognition result and recognition label of each sample image; Step S360: updating the learnable hint vector and the parameters of the side fusion network model according to the predicted loss value; Step S370, continue to train the learnable prompt vector and the side fusion network model based on the training data until the prediction loss value converges; Step S380: After the prediction loss value converges, the current learnable prompt vector is used as the prompt vector, and the current original side fusion network model is used as the side fusion network model.

[0079] The embodiment of the present application proposes a fusion side tuning (FST) method, which significantly improves the memory and computing efficiency of high-resolution medical image classification tasks by introducing a lightweight side fusion network (Side Fusion Network) and freezing the weights of a large-scale pre-trained model (LPM). The side fusion network can effectively utilize the intermediate features extracted by the LPM without the need to return the gradient of the entire model, thereby greatly reducing the memory consumption and computing cost during training.

[0080] The fusion module in the FST method combines the intermediate features of LPM with the features in the side fusion network through a cross-attention mechanism. This module uses the key features (Key) and value features (Value) extracted from the frozen pre-trained model and fuses these features with the query features (Q) in the side fusion network, ensuring that the side fusion network can fully utilize the rich knowledge contained in LPM and improve classification performance.

[0081] In order to further reduce memory consumption, the present invention adopts an important token selection mechanism (Important Token Selection) to select the most valuable tokens through the attention map. This mechanism only selects tokens containing important information, reduces the number of features passed to the side fusion network, significantly reduces GPU memory usage, and ensures efficient feature transfer and diagnostic performance.

[0082] The side fusion network used in the embodiment of the present application is a lightweight network specially designed to efficiently process the features extracted from LPM. It processes and fuses the intermediate features from LPM through multiple fusion modules to achieve efficient feature utilization and classification tasks. The lightweight design of the side fusion network significantly reduces memory consumption and computing costs.

[0083] The fusion module of the embodiment of the present application may also include feature compression and expansion operations to make the fused features more compact and effectively reduce redundant information. This feature compression technology ensures that while reducing memory usage, key diagnostic information is not lost, and improves the classification efficiency and accuracy of high-resolution images.

[0084] In general, the embodiment of the present application first obtains N intermediate graph feature sequences of the image to be identified through N Transformer modules in the large-scale pre-trained model, and then corresponds the N intermediate graph feature sequences to the N fusion modules of the input-side fusion network model, and inputs the prompt vector to the input-side fusion network model to help the fusion module understand and process the intermediate graph feature sequence through the prompt vector, and finally outputs the target graph feature sequence of the image to be identified through the N fusion modules, and obtains the recognition result of the image to be identified based on the target graph feature sequence. The system architecture adopted by this solution can freeze the parameters of the large-scale pre-trained model during the training stage, and only update the parameters of the side fusion network model and the prompt vector, thereby reducing the resource consumption during the training stage, helping to improve the performance of the model in processing high-resolution images, and improving the accuracy of high-resolution image recognition results.

[0085] The present application also provides an image recognition device, such as Figure 5 As shown, the device comprises: An intermediate image feature sequence determination unit is used to input the image to be recognized into the trained large-scale pre-training model to obtain the intermediate image feature sequences respectively output by N Transformer modules in the large-scale pre-training model, wherein N is an integer greater than 1, and the N Transformer modules are connected in sequence; A target graph feature sequence determination unit is used to input a preset prompt vector and an intermediate graph feature sequence into a trained side fusion network model to obtain a target graph feature sequence, wherein the side fusion network model includes N fusion modules connected in sequence, the N fusion modules correspond to N Transformer modules one by one, the intermediate graph feature sequence output by each Transformer module is configured as an input to the corresponding fusion module, and the preset prompt vector is configured as an input to the first fusion module among the N fusion modules; The recognition result determination unit is used to input the target image feature sequence into the trained classification network model to obtain the recognition result of the image to be recognized.

[0086] It should be noted that the information interaction, execution process, etc. between the above-mentioned modules / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0087] Figure 6 The computer device 300 provided in the embodiment of the present application is shown. The computer device 300 may be a server or a terminal, and the internal structure of the computer device 300 includes but is not limited to: A memory 310, used for storing programs; The processor 320 is used to execute the program stored in the memory 310. When the processor 320 executes the program stored in the memory 310, the processor 320 is used to execute the above-mentioned image recognition method.

[0088] The processor 320 and the memory 310 may be connected via a bus or in other ways.

[0089] The memory 310 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs and non-transitory computer executable programs, such as the image recognition method described in any embodiment of the present application. The processor 320 implements the above-mentioned image recognition method by running the non-transitory software program and instructions stored in the memory 310.

[0090] The memory 310 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store and execute the above-mentioned image recognition method. In addition, the memory 310 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 310 may optionally include a memory remotely arranged relative to the processor 320, and these remote memories may be connected to the processor 320 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0091] The non-transitory software programs and instructions required to implement the above-mentioned image recognition method are stored in the memory 310, and when executed by one or more processors 320, the image recognition method provided by any embodiment of the present application is executed.

[0092] The embodiment of the present application also provides a computer-readable storage medium storing computer-executable instructions, and the computer-executable instructions are used to execute the above-mentioned image recognition method.

[0093] In one embodiment, the storage medium stores computer executable instructions, which are executed by one or more control processors, for example, by one or more processors 320 in the above-mentioned computer device 300, so that the above-mentioned one or more processors 320 can execute the image recognition method provided by any embodiment of the present application.

[0094] The above described embodiments are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0095] It will be appreciated by those skilled in the art that all or some of the steps and systems in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically include computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0096] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementation. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. An image recognition method, characterized in that: The method comprises the following steps: Inputting the image to be recognized into a large-scale pre-trained model, obtaining intermediate image feature sequences respectively output by N Transformer modules in the large-scale pre-trained model, wherein N is an integer greater than 1, and the N Transformer modules are connected in sequence; Inputting a preset prompt vector and the intermediate graph feature sequence into a trained side fusion network model to obtain a target graph feature sequence, wherein the side fusion network model includes N fusion modules connected in sequence, the N fusion modules correspond to the N Transformer modules one by one, the intermediate graph feature sequence output by each Transformer module is configured to be input into the corresponding fusion module, and the preset prompt vector is configured to be input into the first fusion module of the N fusion modules; The target image feature sequence is input into a trained classification network model to obtain a recognition result of the image to be recognized.

2. The method according to claim 1, characterized in that Each of the fusion modules is configured to perform cross-attention calculation based on the input query vector, key vector and value vector, and output an updated intermediate graph feature sequence; The first fusion module among the N fusion modules uses the prompt vector as the query vector, and the other fusion modules use the updated intermediate graph feature sequence output by the previous fusion module as the query vector; Each of the fusion modules uses the intermediate graph feature sequence output by the corresponding Transformer module as the key vector and the value vector; The target graph feature sequence is the updated intermediate graph feature sequence output by the last fusion module among the N fusion modules.

3. The method according to claim 2, characterized in that The cross-attention calculation includes the following steps: Normalizing the query vector and the key vector to obtain an attention map; The attention map is multiplied by the value vector to obtain the updated intermediate map feature sequence.

4. The method according to claim 1, characterized in that The large-scale pre-trained model further includes an image segmentation module and an encoding module. The inputting of the image to be recognized into the trained large-scale pre-trained model to obtain the intermediate image feature sequences respectively output by N Transformer modules in the large-scale pre-trained model includes: performing the following steps through the large-scale pre-trained model: Dividing the image to be identified into tags by the image division module; Encode each of the tags respectively by the encoding module to obtain an embedded representation sequence, wherein the embedded representation sequence includes a plurality of embedded representations corresponding to the tags; Each of the Transformer modules performs self-attention calculation based on the input sequence, and outputs a corresponding updated embedding representation sequence, wherein the input sequence of the first Transformer module among the N Transformer modules is the embedding representation sequence, and the input sequences of the other Transformer modules are the updated embedding representation sequence output by the previous Transformer module; According to the updated embedding representation sequence output by each of the Transformer modules, the intermediate graph feature sequence corresponding to the output of each of the Transformer modules is determined.

5. The method according to claim 4, characterized in that Determining the intermediate graph feature sequence outputted by each Transformer module according to the updated embedding representation sequence outputted by each Transformer module includes: Determine an average attention score obtained by each of the markers in the self-attention calculation of the Transformer module; determining important markers based on the average attention score; The intermediate graph feature sequence is determined based on the updated embedding representations corresponding to all the important tags.

6. The method according to claim 5, characterized in that Determining the important tag according to the average attention score includes: The K tags with the highest average attention scores are used as the important tags, where K is a preset value or is determined according to a preset ratio.

7. The method according to claim 1, characterized in that The prompt vector and the side fusion network model are trained by the following steps: Acquire training data, the training data comprising a plurality of sample images and identification labels corresponding to the sample images; Freezing the parameters of the large-scale pre-trained model, inputting the sample image into the large-scale pre-trained model, and obtaining the sample intermediate image feature sequences respectively output by N Transformer modules in the large-scale pre-trained model; Inputting the learnable hint vector and the sample intermediate graph feature sequence into the original side fusion network model to obtain the sample target graph feature sequence; Freezing the parameters of the classification network model, inputting the sample target image feature sequence into the classification network model, and obtaining the recognition result of the sample image; Determining a predicted loss value according to the recognition results and recognition labels of each of the sample images; updating the learnable hint vector and the parameters of the side fusion network model according to the predicted loss value; Continue to train the learnable prompt vector and the side fusion network model based on the training data until the prediction loss value converges; After the prediction loss value converges, the current learnable prompt vector is used as the prompt vector, and the current original side fusion network model is used as the side fusion network model.

8. An image recognition device, characterized in that: The device comprises: An intermediate image feature sequence determination unit is used to input the image to be recognized into a trained large-scale pre-training model to obtain the intermediate image feature sequences respectively output by N Transformer modules in the large-scale pre-training model, wherein N is an integer greater than 1, and the N Transformer modules are connected in sequence; A target graph feature sequence determination unit is used to input a preset prompt vector and the intermediate graph feature sequence into a trained side fusion network model to obtain a target graph feature sequence, wherein the side fusion network model includes N fusion modules connected in sequence, the N fusion modules correspond to the N Transformer modules one by one, the intermediate graph feature sequence output by each Transformer module is configured to be input into the corresponding fusion module, and the preset prompt vector is configured to be input into the first fusion module among the N fusion modules; The recognition result determination unit is used to input the target image feature sequence into the trained classification network model to obtain the recognition result of the image to be recognized.

9. A computer device, characterized in that: include: Memory, used to store programs; A processor, configured to execute the program stored in the memory. When the processor executes the program stored in the memory, the processor is configured to execute: a method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Computer executable instructions are stored, and the computer executable instructions are used to execute: the method according to any one of claims 1 to 7.

Images