Method for extracting image features and computing device

Abstract

The invention provides a method for extracting image features. The method comprises the following steps: acquiring a first feature extracted by a ViT visual converter for a first image; acquiring a second feature extracted by the CNN for the first image; and fusing the first feature and the second feature through an attention mechanism to obtain a target feature of the first image. The CNN is relatively high in detail understanding capability, and by fusing the features extracted by the ViT and the CNN, the detail features extracted by the CNN are added on the basis of the features extracted by the ViT, so that the method provided by the invention is relatively good in performance in an actual application scene needing the detail understanding capability. Moreover, according to the method of the specification, image fusion is carried out by using an attention mechanism instead of adopting a fixed fusion weight, and fusion can be carried out in a more targeted manner according to the characteristics of each image, so that the image obtained by fusion better meets the requirements.

Description

Technical Field

[0001] The embodiments in this specification belong to the field of artificial intelligence technology, and in particular relate to a method and computing device for extracting image features. Background Technology

[0002] With the development of multimodal large-scale models, AI systems are demanding increasingly higher levels of visual information understanding capabilities. In practical applications, multimodal large-scale models typically employ a Vision Transformer (ViT) as the visual encoder. While ViT performs well in global semantic understanding of images, its fixed-size patch segmentation strategy and primary pre-training on low-resolution images limit its ability to capture local detail features. In real-world applications requiring detailed understanding, such as document understanding, medical image analysis, and high-resolution image processing, problems arise where AI struggles to accurately identify text content in images, understand complex chart structures, and perform poorly in detecting small objects, impacting user experience and application effectiveness. Summary of the Invention

[0003] The purpose of this specification is to provide a method and computing device for extracting image features, including:

[0004] The first aspect of this specification provides a method for extracting image features, including:

[0005] Obtain the first feature extracted from the first image by the ViT visual converter;

[0006] Obtain the second feature extracted by the CNN convolutional neural network from the first image;

[0007] The first feature and the second feature are fused using an attention mechanism to obtain the target feature of the first image.

[0008] A second aspect of this specification provides a computer-readable storage medium having a computer program stored thereon that, when executed in a computer, causes the computer to perform the methods described in the first aspect of this specification.

[0009] A third aspect of this specification provides a computing device including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method described in the first aspect of this specification.

[0010] This specification provides a fourth aspect of a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first aspect of this specification.

[0011] CNNs excel in detail understanding. By fusing features extracted by ViT and CNNs, and adding detailed features extracted by CNNs to the ViT features, the method provided in this specification performs well in real-world applications requiring detailed understanding. Furthermore, the method in this specification utilizes an attention mechanism for image fusion, rather than using fixed fusion weights. This allows for more targeted fusion based on the characteristics of each image, resulting in a fused image that better meets the requirements. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a method for extracting image features in one embodiment; Figure 2 This is a schematic diagram of another method for extracting image features in one embodiment; Figure 3 This is a block diagram of an apparatus for extracting image features in one embodiment. Detailed Implementation

[0014] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0015] In this specification, the Large Language Model (LLM) may also be referred to simply as the Large Model. A Large Language Model is a natural language processing model based on deep learning techniques, typically with billions to hundreds of billions or even more parameters, possessing powerful language understanding and generation capabilities. Large Language Models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. Large Language Models learn the statistical features and semantic relationships of language through pre-training on large-scale corpora, giving them good generalization capabilities. The core capabilities of Large Language Models include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Its usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, the user guides the Large Language Model to generate specific outputs by designing prompts. Cue words can be task descriptions or instructions in text form, used to stimulate the semantic understanding and generation capabilities of large language models. In fine-tuning mode, large language models are further trained on small-scale datasets in specific domains to optimize their performance on specific tasks. The powerful generalization ability and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0016] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio data). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0017] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025, public link: https: / / doi.org / 10.48550 / arXiv.2303.18223), and will not be repeated here.

[0018] The following section will describe a method for extracting image features as shown in this specification. For example... Figure 1 As shown, the method includes the following steps:

[0019] Step 101: Obtain the first feature extracted by ViT for the first image.

[0020] The first image is the image from which features need to be extracted. The ViT process involves first dividing the image into fixed-size patches. Through linear projection, each patch is processed into a one-dimensional token, and positional encoding information is added to the token so that the position of the corresponding patch in the image can be determined from the token. Then, based on the ViT block, the tokens are processed to finally obtain the first feature.

[0021] In one alternative implementation, the first image may be the original input image. In another alternative implementation, the first image may be enhanced with details to obtain the first image. Specifically, a data augmentation strategy for detail preservation may be preset, such as local magnification, detail sharpening, and text region enhancement. Optionally, a model for recognizing detail enhancement may be trained to determine the regions that need detail enhancement, and these regions may be enhanced using the data augmentation strategy.

[0022] In one optional implementation, ViT may include a stem module and a ViT block. The stem module refers to the structure at the very beginning of the model, immediately following the input image, used to convert the input image into a feature representation that can be processed by the subsequent backbone network. In the embodiments described in this specification, the stem module may be used to segment the first image into fixed-size patches and convert the patches into tokens with location encoding information.

[0023] The ViT block is the backbone network of the ViT model, and it uses a transformer architecture. In the implementation described in this specification, the ViT block can output a first feature based on the token output by the stem module.

[0024] In a ViT block, global features can be extracted using a multi-head self-attention mechanism. ViT blocks typically also include layer normalization, feedforward networks, and residual connections. Layer normalization normalizes features before and after the attention mechanism, and the feedforward network performs feature transformation through a multilayer perceptron. Residual connections refer to the method of adding the input and output in a deep network and using the sum as the input to the next layer; residual connections can alleviate the vanishing gradient problem during deep network training.

[0025] In an alternative implementation, a fully connected layer can be added after the ViT block. This allows for further processing of the features output by the ViT block to obtain the first feature. This can achieve better first feature extraction results.

[0026] In an alternative implementation, a hybrid attention approach can be employed. Specifically, local attention can be introduced into ViT's self-attention mechanism, where attention is calculated only based on local patches. This way, some attention heads focus on the local neighborhood, while others handle global relationships. Finally, the calculation results from the local and global attention heads are combined to obtain the final attention result. The local attention head can limit the attention scope to neighboring patches, while the global attention head can maintain global attention. This allows ViT to achieve better feature extraction performance.

[0027] In another alternative implementation, ViT can be decomposed into multiple composable modules. Based on image characteristics, a suitable module combination method is searched for the current first image, and the ViT combined using this method is then processed. Each module can specifically handle a particular type of visual information.

[0028] Step 103: Obtain the second feature extracted by the CNN from the first image.

[0029] Convolutional Neural Network (CNN) is a deep learning model that is better at extracting local features of images than ViT.

[0030] In an optional implementation, the CNN in this specification may include a stem module and a convolutional processing stage. The stem module of the CNN receives a first image and performs feature extraction. Then, batch normalization and activation function processing are applied to the extracted features. Preliminary downsampling is then performed using pooling operations to obtain initial output features. These initial features become the output of subsequent modules. The convolutional processing stage performs further convolutional feature extraction, ultimately yielding second features.

[0031] Step 105: The first feature and the second feature are fused using an attention mechanism to obtain the target features of the first image.

[0032] Attention mechanisms are computational methods that adaptively assign weights to different parts of the input based on their importance, giving greater weight to key information and aggregating them into an output representation. Through attention mechanisms, the first and second features can be automatically fused according to their relative importance to obtain the target feature.

[0033] In one optional implementation of step 105, the Q matrix of the attention mechanism can be determined based on the first feature; the K and V matrices of the attention mechanism can be determined based on the second feature; and the target features of the first image can be determined based on the QKV matrix.

[0034] Specifically, it can be based on the weight matrix The Q matrix is ​​determined by multiplying the first feature with the second feature and then by multiplying it with the weight matrix. and weight matrix The product of the Q and V matrices is used to determine the K and V matrices. Then, according to the attention calculation formula, the attention weights A are calculated by substituting the previously calculated Q and K matrices. Finally, the target features are obtained by multiplying the A and V matrices.

[0035] In another alternative implementation, the K matrix and V matrix can be determined based on the first feature, the Q matrix can be determined based on the second feature, and then the target feature can be determined based on the QKV matrix.

[0036] In another alternative implementation, weighting can be achieved through channel attention mechanisms, including classic channel attention and channel self-attention. Classic channel attention is represented by SENet (SE), CBAM (Channel), and ECA. For each channel, a channel weight is calculated based on global statistical information, and the channels are weighted and labeled to highlight important channels and suppress unimportant channels. Channel self-attention mechanisms treat each channel as an interconnected object, explicitly calculating the correlation between channels and generating dynamic weighting results accordingly (e.g., using the QKV matrix mentioned earlier), thereby modeling the dependencies between channels and reorganizing information.

[0037] The examples above are merely examples for this manual, and this manual does not limit the implementation of step 105.

[0038] In the method described in this specification, since the first feature is extracted by ViT and the second feature is extracted by CNN, ViT is good at extracting global features, while CNN is good at capturing low-level visual features such as local textures and edges. By fusing the first feature and the second feature, detailed features can be added while retaining the global features extracted by ViT. This can solve the problem of ViT's insufficient ability to understand details.

[0039] Furthermore, in step 105, instead of using fixed weights for fusion, an attention mechanism is employed. This allows for the automatic identification of key components in different images, applying varying fusion weights accordingly. This enables more targeted fusion based on the characteristics of each image, resulting in better fusion performance.

[0040] Furthermore, in step 105, if the feature resolutions of the first feature and the second feature are different, their feature resolutions can be unified to facilitate fusion. Here, feature resolution refers to the resolution of the feature map corresponding to the first feature or the second feature.

[0041] Feature resolution unification can be achieved by adjusting the resolution of the second feature extracted by the CNN or the resolution of the first feature extracted by the ViT. The specific resolution adjustment method can be either upsampling or downsampling.

[0042] Upsampling and downsampling can be implemented using methods from related techniques. For example, upsampling can be achieved using transposed convolution or interpolation, while downsampling can be achieved using strided convolution or pooling. Transposed convolution learns weights only in the inverse process of convolution, achieving learnable upsampling. Stride convolution refers to a convolution with a stride greater than 1, discarding some positions during convolution to achieve downsampling. Pooling refers to taking the maximum or average value within a local window, using a single value to represent a region, achieving size compression. This specification does not limit the implementation methods of upsampling and downsampling.

[0043] When unifying feature resolution by adjusting the resolution of the second feature extracted by the CNN, this can be achieved through the following steps: obtaining an initial second feature extracted by the CNN convolutional neural network for the first image; the resolution of the initial second feature is different from the resolution of the first feature; processing the initial second feature using an upsampling or downsampling method to obtain a second feature; the resolution of the second feature is the same as that of the first feature. This example is not intended to limit the scope of this specification; in an optional implementation, it can also be achieved by adjusting the resolution of the ViT.

[0044] The extracted target features can be used as input for downstream image processing tasks on the first image. For example, the target features can be used for object recognition, image classification, etc. In an optional implementation, the target features can also be used as input for a large language model to generate a response to prompts input by the user.

[0045] Generally, a CNN consists of multiple stages. Since each stage in a CNN downsamples its output features after processing, the feature resolution decreases as the stage number increases. In other words, a CNN has n stages; the stage number and the feature resolution of the corresponding stage's output are negatively correlated. Feature resolution refers to the size of the feature map corresponding to the extracted features. The feature map can be represented by a tensor of [C, H, W], where C represents the number of channels, and H and W represent the height and width, respectively. Higher resolution results in a larger feature map, and consequently, larger H and W. Correspondingly, smaller stage numbers mean that the features extracted by earlier stages are more representative of detailed features.

[0046] Correspondingly, ViT can also include multiple stages. The following sections will explain the processing methods for ViT and CNN when they include multiple stages.

[0047] In one alternative implementation, the second feature can be a feature output by the last stage of the CNN, and the first feature can be a feature output by the last stage of ViT. In other words, Figure 1 The method shown, for the nth stage (the last stage), could be as follows: Step 101 could be: Based on the third feature of the first image output by the (n-1)th stage of the ViT, extract the first feature of the first image through the nth stage of the ViT. Step 103 could be: Based on the fourth feature of the first image output by the (n-1)th stage of the CNN, extract the second feature of the first image through the nth stage of the CNN.

[0048] In another alternative implementation, the same number of stages n can be set for ViT and CNN, and correspondingly, at each stage, the following can be performed: Figure 1 The method shown.

[0049] Optionally, this can be done at each stage. Figure 1 The method shown is fused. The fused features are then fed into the next stage of CNN and ViT.

[0050] Optionally, in stages other than the final stage, the features processed by the CNN can be directly used as input to the next stage of the CNN, while the features obtained by fusing the CNN and ViT can be used as input to the next stage of the ViT. The reason for inputting the fused features into ViT instead of the CNN is that ViT has better feature extraction performance, and ViT is the backbone network of this model. The method in this specification aims to add detailed features while ensuring the global features extracted by ViT; therefore, the entire method mainly revolves around ViT. Here, ViT is the core of the processing, so the fused features are input into ViT instead of the CNN.

[0051] In other words, for stage i in stages 1 to n-1, we can: obtain the fifth feature extracted by ViT for the first image in stage i; obtain the sixth feature extracted by CNN for the first image in stage i; use the sixth feature as input to stage i+1 of CNN; fuse the fifth feature and the sixth feature through an attention mechanism to obtain the seventh feature; use the seventh feature as input to stage i+1 of ViT.

[0052] Optionally, n ViT models with different patch sizes can be set, and the first image can also be decomposed into multiple versions with different resolutions. The different resolution versions of the model are then input into the ViT models with different patch sizes. At each stage, the features extracted by ViT are fused with the features extracted by CNN.

[0053] In this implementation, unlike the previous one, the input to the ViT model at each stage is an image, and the input to the CNN at each stage is the output of the CNN at the previous stage (the input to the CNN at the first stage is the output of the CNN stem module for the original first image).

[0054] This yields n fused features, which are then combined using a feature pyramid. The feature pyramid generates feature maps of various sizes (from large to small) and fuses these different sizes using a specific structure, ensuring that each feature map contains relatively complete semantic information.

[0055] In the above embodiments, alternatively, different attention strategies can be employed at different stages of the attention mechanism. For example, a local attention strategy can be used in the fusion of stages with smaller numbers, while a global attention strategy can be used in stages with larger numbers. This allows the shallower layers to focus more on local details and the deeper layers to focus on global semantics, resulting in fusion features that better meet the requirements.

[0056] The following will use CNN and ViT, comprising four stages, as examples to illustrate a fusion process provided in the embodiments of this specification. A schematic diagram of the method is shown below. Figure 2 As shown. Among them, Figure 2 The red numbers in the image represent the downsampling ratio, which is the ratio of the output features to the downsampled image.

[0057] like Figure 2 As shown, this method involves two main branches. The first branch is the ViT processing branch, which includes four stages and a ViT stem module. Each stage includes a ViT block and a fully connected layer. Details regarding the ViT stem module, ViT block, and fully connected layer are provided above and will not be repeated here.

[0058] The second branch is the CNN processing branch, each of which includes a CNN stem module and four stages of CNN processing. The CNN stem module performs initial convolutional computations. Each stage of the CNN processing module can include several convolutional layers. In stage 1, shallow convolutional feature extraction is performed, and the extracted feature maps maintain a relatively high resolution. Stage 2 further extracts convolutional features based on stage 1, obtaining mid-level features and reducing the resolution. Stage 3 further extracts deep features based on stage 2, further reducing the resolution. Stage 4 further extracts high-level semantic features based on stage 3, obtaining features with the lowest resolution.

[0059] In this branch, Figure 2 The diagram also shows that each stage of the CNN has residual connections. Optionally, in addition to residual connections, CNNs can also have dense connections between stages (not shown in the diagram). Dense connections between multiple stages mean that the feature map of any deeper stage not only receives the output of its previous stage, but also receives the concatenation (or summation) of the output features of all previous stages as input. Through dense connections, cross-stage feature connections can be established, enabling the effective propagation of feature information.

[0060] In one alternative implementation, in dense connections and residual connections, data from multiple stages can be weighted and summed. The weights can be learned during training; by learning these weights, the contribution of features at different stages can be controlled, resulting in features that better meet the requirements.

[0061] After explaining the two branches, we will now explain how to merge them.

[0062] exist Figure 2 In this process, the aforementioned fusion is achieved through a channel attention module, where channel attention is an attention mechanism, also known as channel self-attention. Figure 2 The channel attention module shown may include a global pooling layer, a multilayer perceptron, an activation function, and a fusion submodule. The global pooling layer aggregates spatial information from the input features; the multilayer perceptron learns the non-linear relationships and importance weights between channels (i.e., the feature maps extracted from the two branches); the activation function generates normalized attention weights; and the fusion submodule performs fusion based on these attention weights.

[0063] Both branches perform feature fusion at each stage using the aforementioned channel self-attention mechanism. For example, in stage 1, the data processed by the CNN is downsampled and then fused with the data obtained from ViT stage 1 using the channel self-attention mechanism. The fused data is then input into the ViT model in stage 2. The input to CNN stage 2 is the output of CNN stage 1. The processing methods for other stages are similar and will not be elaborated further.

[0064] Whether upsampling or downsampling is used in each stage is determined by the resolution of the feature map obtained from the CNN and the feature map obtained from ViT. For example Figure 2 In the process, downsampling was used in stage 1, upsampling was used in stages 3 and 4, and the feature maps obtained from the two branches in stage 2 had the same resolution and were not processed.

[0065] In one alternative implementation, the number of channels in the feature maps of the two branches may also be different. Therefore, the number of channels in the feature maps extracted by CNN or ViT can be adjusted to make the number of channels in the two adjusted feature maps the same, facilitating their fusion. In one alternative implementation, the number of channels can be adjusted using a convolutional projection layer.

[0066] By fusing features extracted from both CNN and ViT branches through the aforementioned channel self-attention mechanism, complementary feature enhancement and redundancy suppression are achieved. The trained channel attention module can adaptively generate different weights for different images, thereby obtaining a more suitable fused image for various scenarios.

[0067] The training method for the above model will be explained next.

[0068] First, ViT and CNN can be pre-trained. The pre-training methods for both can be the same as those used in related technologies.

[0069] Secondly, the pre-trained CNN and ViT can be fine-tuned using the training dataset, and the weights corresponding to the attention mechanism can be trained. End-to-end training can be used here.

[0070] In other words, a training dataset consisting of multiple images can be obtained; the images in the training dataset are processed according to a first model including ViT, CNN, and channel self-attention mechanism, and the weights of ViT, CNN, and channel self-attention mechanism are adjusted according to the processing results.

[0071] For a specific training method, in one optional implementation, a contrastive learning approach can be used. For an image in the training dataset, images obtained by applying different data augmentations to that image are considered positive samples, while other images are considered negative samples. The image and its corresponding positive and negative samples are then input into the first model. The goal is to make the features of the positive samples extracted by the first model more similar to the features of the image, while making the features of the negative samples extracted by the first model more different from the features of the image.

[0072] In another alternative implementation, a reconstruction / mask prediction method can be used for training, that is, the image is masked and the model predicts the masked part based on the unmasked part.

[0073] Of course, the above training methods do not represent a limitation on the training method of the first model.

[0074] After training, the first model can be directly used for downstream tasks, such as image classification, object detection, and large language models. In an alternative implementation, to make the first model more suitable for the needs of downstream tasks, training datasets corresponding to each downstream task can be obtained, and then fine-tuned using each training dataset to obtain different first models suitable for different downstream tasks.

[0075] This allows the trained first model to better meet the needs of downstream tasks. For example, in the method described above, the encoding results of ViT and CNN are fused, which may result in the image features output by the first model not having the same semantic meaning as the original ViT output features. To ensure that the large language model can still immediately capture the semantic features of the image, fine-tuning can be performed during training to ensure semantic alignment between the image and text while fusing detailed features.

[0076] Furthermore, different first models can be trained for different downstream tasks, so that the fused features better meet the requirements of the downstream tasks. For example, in tasks that require detailed understanding, when using an attention mechanism to fuse the first and second features, the proportion of the first features extracted by the CNN in the fusion can be increased to enhance the detailed representation.

[0077] In one alternative implementation, the first model trained on the training dataset may include multiple stages, such as... Figure 2The diagram illustrates four stages. For different downstream tasks, the output of one stage can be used as the input. For example, for downstream tasks requiring detailed understanding, the output of a stage with a lower number, such as stage two or three, can be used as the input. Conversely, for downstream tasks requiring a more holistic understanding of the image, the output of a stage with a higher number, such as stage four, can be used as the input.

[0078] Correspondingly, when fine-tuning for different downstream tasks, the output of the stage corresponding to the downstream task is used as the input of the downstream task, and fine-tuning is performed based on this.

[0079] The more layers a CNN has, the better the extracted features reflect the overall characteristics of the image; conversely, fewer layers result in features that better reflect local details. By selecting the output of an appropriate stage as input for a downstream task, features that better meet the needs of that task can be obtained.

[0080] The method provided in this manual can be used to combine the features extracted by CNN and ViT, and take advantage of the fact that CNN is good at extracting low-level features such as edges and textures to make up for the lack of details in ViT.

[0081] Moreover, compared to directly concatenating the features of CNN and ViT, fusion can remove redundant information from both sets of features. Directly concatenating the features of CNN and ViT amplifies redundant information and increases feature dimensionality, thus affecting model efficiency, because the two models extract similar feature information.

[0082] Compared to the fixed-weight approach that weights the features of CNN and ViT, this approach allows for targeted fusion for each different image, eliminating the need for expert experience and improving the generalization ability of the solution.

[0083] This specification also provides an apparatus for extracting image features, such as... Figure 3 As shown, it includes:

[0084] The first extraction module 310 is used to obtain the first feature extracted by the ViT visual converter for the first image;

[0085] The second extraction module 320 is used to obtain the second feature extracted by the CNN convolutional neural network from the first image;

[0086] The fusion module 330 is used to fuse the first feature and the second feature through an attention mechanism to obtain the target feature of the first image.

[0087] In one optional embodiment, the ViT includes n stages, and the CNN includes n stages; the stage number of the CNN and the feature resolution output by the corresponding stage are negatively correlated; the first extraction module 310 is specifically used to extract the first feature of the first image through the nth stage of the ViT based on the third feature of the first image output by the (n-1)th stage of the ViT. The second extraction module 320 is specifically used to extract the second feature of the first image through the nth stage of the CNN based on the fourth feature of the first image output by the (n-1)th stage of the CNN.

[0088] In an optional implementation, for the i-th stage among the 1 to n-1 stages, the method further includes: a processing module 340 (not shown in the figure), used to obtain the fifth feature extracted from the first image by the i-th stage of ViT; obtain the sixth feature extracted from the first image by the i-th stage of CNN; the sixth feature is used as the input of the (i+1)-th stage of CNN; the fifth feature and the sixth feature are fused through an attention mechanism to obtain a seventh feature; the seventh feature is used as the input of the (i+1)-th stage of ViT.

[0089] In one alternative implementation, dense connections exist between different stages of the CNN.

[0090] In an optional implementation, the fusion module 330 is specifically used to determine the Q matrix of the attention mechanism based on the first feature; determine the K matrix and V matrix of the attention mechanism based on the second feature; and determine the target features of the first image based on the QKV matrix.

[0091] In one alternative implementation, the ViT includes a stem module, a ViT block, and a fully connected layer.

[0092] In an optional embodiment, the second extraction module 320 is specifically used to obtain an initial second feature extracted by the CNN convolutional neural network for the first image; the resolution of the initial second feature is different from the resolution of the first feature; the initial second feature is processed by an upsampling or downsampling method to obtain a second feature; the resolution of the second feature is the same as that of the first feature.

[0093] In an optional implementation, a training module 300 (not shown in the figure) is further included, which is used to acquire a training dataset including multiple images; process the images in the training dataset according to a first model including ViT, CNN and channel self-attention mechanism, and adjust the weights of ViT, CNN and channel self-attention mechanism according to the processing results.

[0094] In one alternative implementation, the target features are used as input for a downstream image processing task for the first image.

[0095] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0096] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0097] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a server system. Of course, this specification does not exclude the possibility that, with the future development of computer technology, the computer implementing the functions of the above embodiments may be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0098] While one or more embodiments of this specification provide the operational steps of the methods described in the embodiments or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible order of execution among many steps and does not represent the only possible order. In actual device or end product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes said elements is not excluded. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any particular order.

[0099] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0100] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0101] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0102] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0103] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0104] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0105] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage, graphene storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0106] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0107] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0108] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0109] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of these embodiments. Various modifications and variations can be made to these embodiments by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A method for extracting image features, comprising: Obtain the first feature extracted from the first image by the ViT visual converter; Obtain the second feature extracted by the CNN convolutional neural network from the first image; The first feature and the second feature are fused using an attention mechanism to obtain the target feature of the first image.

2. The method according to claim 1, wherein the ViT comprises n stages, and the CNN comprises n stages; The stage number of a CNN is negatively correlated with the feature resolution of the output of the corresponding stage; The acquisition of the first feature extracted from the first image by the ViT visual converter includes: Based on the third feature of the first image output by the (n-1)th stage of ViT, the first feature of the first image is extracted through the nth stage of ViT; The process of obtaining the second feature extracted by the CNN convolutional neural network for the first image includes: Based on the fourth feature of the first image output by the (n-1)th stage of the CNN, the second feature of the first image is extracted through the nth stage of the CNN.

3. The method according to claim 2, for the i-th stage among the 1 to n-1 stages, further comprising: Obtain the fifth feature extracted from the first image in the i-th stage of ViT; Obtain the sixth feature extracted from the first image by the i-th stage of the CNN; The sixth feature serves as the input to the (i+1)th stage of the CNN; The fifth and sixth features are fused using an attention mechanism to obtain the seventh feature; The seventh feature serves as the input for the (i+1)th stage of ViT.

4. The method according to claim 2, wherein dense connections exist between different stages of the CNN.

5. The method according to claim 1, wherein fusing the first feature and the second feature through an attention mechanism to obtain the target feature of the first image comprises: The Q-matrix of the attention mechanism is determined based on the first feature; The K and V matrices of the attention mechanism are determined based on the second feature; Based on the QKV matrix, the target features of the first image are determined.

6. The method according to claim 1, wherein the ViT comprises a stem module, a ViT block, and a fully connected layer.

7. The method according to claim 1, wherein obtaining the second feature extracted by the CNN convolutional neural network for the first image comprises: Obtain the initial second features extracted by the CNN convolutional neural network for the first image; The resolution of the initial second feature is different from the resolution of the first feature; The initial second feature is processed by upsampling or downsampling methods to obtain a second feature; the second feature has the same resolution as the first feature.

8. The method according to claim 1, further comprising: Obtain a training dataset that includes multiple images; The images in the training dataset are processed using a first model that includes ViT, CNN, and channel self-attention mechanism, and the weights of ViT, CNN, and channel self-attention mechanism are adjusted based on the processing results.

9. The method of claim 1, wherein the target feature is used as input to a downstream image processing task for the first image.

10. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-9.

Images