Image recognition method, device, equipment and storage medium

Abstract

The application relates to the technical field of image recognition, and discloses an image recognition method, device, equipment and storage medium. The method comprises the following steps: acquiring an image to be recognized; performing feature coding on the image to be recognized to obtain a graph block coding representation of an image block; performing pixel correlation degree feature extraction on the image to be recognized based on a multi-head self-attention mechanism through the graph block coding representation to obtain an image attention weight representation and an image feature representation of the image to be recognized; performing feature indexing on the image attention weight representation to obtain an attention weight index representation containing information with the strongest and weakest attention intensity; performing feature indexing on the image feature representation based on the attention weight index representation to obtain an image feature index representation; and performing image recognition based on the image feature index representation to obtain an image recognition result. According to the embodiment of the application, the features of the region with the strongest attention intensity and the region with the weakest attention intensity are considered simultaneously, and therefore the accuracy of fine-grained image recognition can be improved.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to an image recognition method, apparatus, device and storage medium. Background Technology

[0002] Image classification can be broadly categorized into coarse-grained and fine-grained classification. Fine-grained recognition is an important branch of computer vision, aiming to accurately distinguish between different categories in an image that exhibit subtle differences.

[0003] Among related technologies, deep learning methods such as Region Proposal Networks (RPN) can be used to perform fine-grained recognition tasks well.

[0004] However, current image recognition methods focus too much on regions with strong attention intensity in an image, while ignoring regions with weak attention intensity. These regions may contain important classification feature information that was missed by the strong attention regions, leading to a decrease in fine-grained recognition accuracy. Summary of the Invention

[0005] The purpose of this application is to provide an image recognition method, apparatus, device, and storage medium, which aims to improve the accuracy of fine-grained image recognition by simultaneously considering the characteristics of both regions with strong attention intensity and regions with weak attention intensity.

[0006] This application provides an image recognition method, including:

[0007] Acquire the image to be identified; the image to be identified includes multiple image blocks;

[0008] The image to be identified is feature-encoded to obtain the patch-encoded representation of the image patch;

[0009] Based on the multi-head self-attention mechanism, pixel correlation features are extracted from the image to be identified through the patch encoding representation to obtain the image attention weight representation and image feature representation of the image to be identified.

[0010] The image attention weight representation is indexed by features to obtain the attention weight index representation; the attention weight index representation includes the attention weight representation of the patch with the strongest attention intensity and the attention weight representation of the patch with the weakest attention intensity in the image attention weight representation;

[0011] Based on the attention weight index representation, feature indexing is performed on the image feature representation to obtain the image feature index representation;

[0012] Image recognition is performed based on the image feature index representation to obtain the image recognition result.

[0013] In some embodiments, the step of performing feature encoding on the image to be identified to obtain a patch-coded representation of the image patch includes:

[0014] Perform a linear mapping on the image patch to obtain a linear mapping vector;

[0015] The linear mapping vector is embedded with positions to obtain the tile-coded representation.

[0016] In some embodiments, the step of extracting pixel correlation features from the image to be identified based on the multi-head self-attention mechanism through the patch encoding representation to obtain the image attention weight representation and image feature representation of the image to be identified includes:

[0017] The tile attention weight representation of the tile encoding is determined based on a multi-head attention mechanism;

[0018] The attention weight representations of each of the aforementioned image patches are concatenated and compressed to obtain the image attention weight representation.

[0019] The image attention weight representation is subjected to feature mapping processing to obtain the image feature representation.

[0020] In some embodiments, performing feature indexing on the image attention weight representation to obtain an attention weight index representation includes:

[0021] The attention weight representation of the image is multiplied to obtain the attention weight multiplication representation;

[0022] The attention weight index representation is obtained by indexing the attention weight representations of the patch with the strongest and weakest attention intensity in the cumulative attention weight representation.

[0023] In some embodiments, the step of performing feature indexing on the image feature representation based on the attention weight index representation to obtain the image feature index representation includes:

[0024] Based on the attention weight index representation, the patch feature representations in the image feature representation are indexed to obtain a first target feature representation and a second target feature representation; the first target feature representation is composed of the patch feature representation corresponding to the patch attention weight representation with the strongest attention intensity, and the second target feature representation is composed of the patch feature representation corresponding to the patch attention weight representation with the weakest attention intensity.

[0025] The image feature index representation is obtained by concatenating the first target feature representation and the second target feature representation.

[0026] In some embodiments, concatenating the first target feature representation and the second target feature representation to obtain the image feature index representation includes:

[0027] The first target feature representation is weighted and activated using a first learnable parameter to obtain a first intermediate parameter; the second target feature representation is weighted and activated using a second learnable parameter to obtain a second intermediate parameter.

[0028] Based on the first intermediate parameter and the second intermediate parameter, calculate the first weight parameter and the second weight parameter, construct the value range of the first weight parameter and the second weight parameter, and determine the value of the first weight parameter and the second weight parameter.

[0029] The first target feature representation and the second target feature representation are weighted and fitted using the first weight parameter and the second weight parameter to obtain the image feature index representation.

[0030] In some embodiments, the step of performing image recognition based on the image feature index representation to obtain the image recognition result includes:

[0031] The image feature index representation is normalized to obtain an image feature sequence;

[0032] Add a classification header to the image feature sequence to construct an image classification feature representation;

[0033] The image classification feature representation is subjected to multi-label classification to obtain the image recognition result.

[0034] This application also provides an image recognition device, including:

[0035] The first module is used to acquire the image to be identified; the image to be identified includes multiple image blocks.

[0036] The second module is used to perform feature encoding on the image to be identified to obtain the patch encoding representation of the image block;

[0037] The third module is used to extract pixel correlation features from the image to be identified based on the multi-head self-attention mechanism and the patch encoding representation, so as to obtain the image attention weight representation and image feature representation of the image to be identified.

[0038] The fourth module is used to perform feature indexing on the image attention weight representation to obtain an attention weight index representation; the attention weight index representation includes the attention weight representation of the patch with the strongest attention intensity and the attention weight representation of the patch with the weakest attention intensity in the image attention weight representation;

[0039] The fifth module is used to perform feature indexing on the image feature representation based on the attention weight index representation to obtain the image feature index representation;

[0040] The sixth module is used to perform image recognition based on the image feature index representation to obtain the image recognition result.

[0041] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the image recognition method described above.

[0042] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image recognition method described above.

[0043] The beneficial effects of this application are as follows: When recognizing target objects in an image to be recognized, based on the strongest and weakest attention weight representations of the image attention weight representations of the image to be recognized, feature indexing is performed on the image feature representations obtained by extracting pixel correlation features from the image to be recognized through feature encoding and multi-head self-attention mechanism. This results in the indexed image feature representation containing the patch feature representations of the strongest and weakest attention patches. Image recognition is then performed based on the image feature index representation, fully considering the feature information of both the strongest and weakest attention regions in the image to be recognized. The resulting image recognition result has good fine-grained image recognition accuracy. Attached Figure Description

[0044] Figure 1 This is an application environment diagram of the image recognition method provided in the embodiments of this application.

[0045] Figure 2 This is a flowchart of the image recognition method provided in the embodiments of this application.

[0046] Figure 3 This is a flowchart of the specific method of step S203 provided in the embodiments of this application.

[0047] Figure 4 This is a flowchart of the specific method for step S204 provided in the embodiments of this application.

[0048] Figure 5 This is a flowchart of the specific method of step S206 provided in the embodiments of this application.

[0049] Figure 6This is a schematic diagram of the structure of the image recognition device provided in the embodiments of this application.

[0050] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application.

[0051] Figure 8 This is a schematic diagram of the structure of the image recognition model provided in the embodiments of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0053] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and drawings are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0055] The image recognition method provided in this application embodiment can be applied to, for example... Figure 1The application environment is shown. Terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed in the cloud or on another server. When a user on terminal 102 wants to identify an object, they can submit an image containing the object to be identified to server 104 through terminal 102. Server 104 then performs image recognition processing on the image to be identified. When identifying an object, server 104 performs feature encoding on the image to be identified, obtaining a patch-encoded representation of the image blocks. Based on a multi-head self-attention mechanism, it extracts pixel correlation features from the image to be identified using the patch-encoded representation, obtaining an image attention weight representation and an image feature representation. It then performs feature indexing on the image attention weight representation to obtain an attention weight index representation. Based on the attention weight index representation, it performs feature indexing on the image feature representation to obtain an image feature index representation. Finally, it performs image recognition based on the image feature index representation to obtain the image recognition result of the image to be identified. The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0056] Figure 2 This is a flowchart of the image recognition method provided in the embodiments of this application. Figure 8 This is a schematic diagram of the structure of the image recognition model provided in an embodiment of this application. In one embodiment, refer to... Figure 2 and Figure 8 An image recognition method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the method includes, but is not limited to, steps S201 to S204.

[0057] Step S201: Obtain the image to be recognized.

[0058] In some embodiments, when the image to be identified is uploaded by the terminal, the server receives the image from the terminal. The terminal can be a mobile terminal, which can be a mobile image acquisition device. The image to be identified is obtained by capturing a target object using the mobile image acquisition device. The image to be identified may contain specific animals, plants, landmarks, or landscapes, and the purpose of image recognition is to identify the names of the objects contained in the image.

[0059] The image to be identified comprises multiple image blocks. In some embodiments, the server divides the image to be identified into multiple image blocks. The above-mentioned image block division of the image to be identified is an operation of determining the pixel region corresponding to the token sequence when the image to be identified is input and encoded to obtain the corresponding image block encoding representation (token sequence), and does not actually segment the image to be identified.

[0060] Step S202: Perform feature encoding on the image to be recognized to obtain the patch encoding representation of the image block.

[0061] In some embodiments, feature encoding of each image patch in the image to be recognized can be achieved through a linear embedding layer of an image recognition model. Specifically, pixels of a preset size feature encoding range corresponding to the image patch are input to the linear embedding layer to perform linear mapping on the image patch and position embedding after linear mapping, outputting a patch-encoded representation of the image patch.

[0062] Step S203: Based on the multi-head self-attention mechanism, pixel correlation features are extracted from the image to be identified by using patch encoding to obtain the image attention weight representation and image feature representation of the image to be identified.

[0063] Indicatively, each image block corresponds to a tile encoding representation. When extracting features from the image feature representation of the image to be recognized, the pixel similarity between image blocks can be indicated by calculating the similarity between the tile encoding representations. Based on the pixel similarity, the correlation between pixels in the image to be recognized is determined. Based on this correlation, the features of other image blocks are fused into the current image block, so that the image feature representation corresponding to each extracted image block includes the correlation information between other image blocks and the current image block, thereby obtaining the image feature representation of the image to be recognized.

[0064] In some embodiments, based on a multi-head self-attention mechanism, pixel correlation feature extraction of the image to be recognized is performed using patch encoding representations, which can be implemented through the encoder layer of an image recognition model. Specifically, the patch encoding representations are input into the encoder layer of the image recognition model to determine the pairwise similarity between each patch encoding representation using a multi-head attention mechanism, obtaining the patch attention weight representation corresponding to each patch encoding representation. Then, the patch attention weight representations are concatenated and compressed to obtain the image attention weight representation of the image to be recognized. Finally, the image attention weight representation features of the image to be recognized are mapped to obtain the image feature representation of the image to be recognized.

[0065] In some embodiments, the encoder layer of the image recognition model has multiple encoders, which perform pixel correlation feature extraction to extract pixel correlation features multiple times from the image to be recognized. Specifically, the first encoder receives the patch encoding representation and performs pixel correlation feature extraction to obtain a first set of image attention weight representations and a first set of intermediate image feature representations. The next encoder receives the intermediate image feature representation output by the previous encoder and performs pixel correlation feature extraction to obtain the corresponding image attention weight representation and the corresponding intermediate image feature representation. The last encoder receives the intermediate image feature representation output by the penultimate encoder and performs pixel correlation feature extraction to obtain the last set of image attention weight representations and the image feature representation of the image to be recognized. The intermediate image feature representations include the patch feature representations obtained after several pixel correlation feature extractions of each image patch; that is, the image feature representation of the image to be recognized includes the patch feature representations obtained after multiple pixel correlation feature extractions of each image patch. The image attention weight representations include the patch attention weight representations of each image patch in the corresponding pixel correlation feature extraction. The patch feature representations are obtained based on the corresponding patch attention weight representations.

[0066] Step S204: Perform feature indexing on the image attention weight representation to obtain the attention weight index representation.

[0067] The attention weight index represents the patch attention weight representation with the strongest attention intensity and the patch attention weight representation with the weakest attention intensity in the image attention weight representation.

[0068] In some embodiments, feature indexing of the image attention weight representation can be implemented through the image feature indexing layer of an image recognition model. Specifically, multiple sets of image attention weight representations generated during pixel correlation feature extraction are multiplied together, and feature indexing is performed on the multiplied image attention weight representations to obtain the attention weight index representations of the patches with the strongest and weakest attention strengths among the multiplied image attention weight representations.

[0069] Step S205: Based on the attention weight index representation, perform feature indexing on the image feature representation to obtain the image feature index representation.

[0070] In some embodiments, feature indexing of the image feature representation based on the attention weight index representation can be implemented through the image feature indexing layer (BBSM) of the image recognition model. Specifically, the attention weight index representation and the image feature representation are input into the image feature indexing layer of the image recognition model. In the image feature indexing layer, feature indexing is performed on the image feature representation according to the mapping relationship between the patch attention weight representation in the attention weight index representation and the patch feature representation in the image feature representation. This allows for the indexing of the patch feature representation corresponding to the patch attention weight representation in the attention weight index representation, i.e., the patch feature representation corresponding to the patch with the strongest attention intensity and the patch feature representation corresponding to the patch with the weakest attention intensity, thus obtaining the image feature index representation.

[0071] Step S206: Perform image recognition based on image feature index representation to obtain image recognition results.

[0072] Image recognition refers to querying the specific type of an object in an image based on its feature index representation. For example, for a landmark in an image, image recognition can determine its specific name. Or, for an animal or plant in an image, image recognition can determine its specific species.

[0073] In some embodiments, image recognition based on image feature index representation can be achieved through the classification layer of an image recognition model. Specifically, the image feature index representation is input into the normalization layer (Norm) of the image recognition model and normalized to obtain a normalized image feature index representation. This normalized representation is then input into the classification layer (MLP) of the image recognition model. The extracted image feature index representation contains the semantic features of the image patches with the strongest attention intensity and the image patches with the weakest attention intensity. A pre-trained classification layer decodes the image feature index representation to identify and classify the target objects in the image to be recognized, thereby generating the image recognition result corresponding to the image to be recognized.

[0074] In summary, the image recognition method provided in this application, when recognizing target objects in an image to be recognized, performs feature indexing on the image feature representation obtained by sequentially extracting pixel correlation features from the image to be recognized through feature encoding and multi-head self-attention mechanism, based on the strongest and weakest attention weight representations of the image attention weight representations of the image to be recognized. This results in the indexed image feature representation containing both the strongest and weakest attention weight representations of the image block. Image recognition is then performed based on the image feature index representation, fully considering the feature information of both the strongest and weakest attention weight regions in the image to be recognized. The resulting image recognition result has good fine-grained image recognition accuracy.

[0075] In some embodiments, step S202 specifically includes: performing linear mapping on image blocks to obtain linear mapping vectors; and performing position embedding on the linear mapping vectors to obtain a block-coded representation.

[0076] Image patches are input into the linear embedding layer of the image recognition model to perform linear mapping on the image patches, mapping each image patch into a linear mapping vector of a preset dimension. Then, corresponding position information is embedded into the linear mapping vector to represent the position of the image patch in the image to be recognized. After position encoding, the linear mapping vector is mapped into a linear mapping vector with embedded position information, i.e., a position embedding vector. A classification header (ClassToken) of the same dimension is added to the position embedding vector. The classification header is used for image recognition to obtain the patch encoding representation.

[0077] Figure 3 This is a flowchart of the specific method for step S203 provided in the embodiments of this application. In some embodiments, see [link to relevant documentation]. Figure 3 The method includes, but is not limited to, steps S301 to S303.

[0078] Step S301: Determine the patch attention weight representation of the patch encoding representation based on the multi-head attention mechanism.

[0079] In this embodiment, the encoder in the encoder layer of the image recognition model obtains the patch attention weight representation corresponding to each patch encoding representation through a multi-head attention mechanism.

[0080] Schematic illustration: The encoder described above includes a multi-head self-attention unit and a feedforward neural network (FFN) unit. The multi-head attention unit is essentially a multi-head self-attention structure used in telecommunications, with each head employing a typical self-attention mechanism. Specifically, in each head, for each input image token, the query vector Q, keyword K, and feature vector Value of that token are calculated. Then, the correlation weights between Q and K of all token representations are calculated. Finally, the feature vector Values ​​of all token representations are weighted and summed to obtain a new feature vector value for that token representation. This mechanism, by calculating the correlation between each pair of token representations, can better extract contextual and long-range feature information. The formula for calculating the token attention weights is:

[0081] ,

[0082] ,

[0083] in, , , and The parameters are learnable network parameters, Concat() is the merging operation, and Attention() is the kernel function of the attention mechanism based on vector similarity.

[0084] The patch similarity between the patch encoding representation of the i-th image patch and the patch encoding representation of the j-th image patch is determined by the multi-head attention unit in the encoder, where i and j are positive integers. For the i-th image patch, the patch similarity with the i-th image patch is used as a weight to perform a weighted sum of the patch encoding representations of multiple image patches to obtain the patch attention weight representation corresponding to the i-th image patch.

[0085] Step S302: The attention weight representations of each patch are spliced ​​and compressed to obtain the image attention weight representation.

[0086] Indicatively, the patch attention weight representation output by the multi-head attention unit corresponds to the patch encoding representation. That is, the image to be recognized corresponds to multiple patch attention weight representations. Therefore, before being input into the feedforward neural network unit, the attention representations need to be concatenated and compressed to obtain the input of the feedforward neural network unit.

[0087] In some embodiments, multiple patch attention weight representations are concatenated to obtain a concatenated attention representation, and the concatenated attention representation is multiplied by a specified weight matrix to obtain an image attention weight representation.

[0088] Optionally, the specified weight matrix can be a system preset or a learnable parameter, that is, it can be trained during the training process of the image recognition model.

[0089] Step S303: Perform feature mapping processing on the image attention weight representation to obtain the image feature representation.

[0090] In some embodiments, the feedforward neural network unit may include one feedforward neural network, or may include multiple feedforward neural networks.

[0091] Because the positional relationships between patch encoding representations, especially the two-dimensional structural information between image pixels, are ignored during the calculation process of the multi-head attention mechanism unit, in order to solve the above problem, the feedforward neural network unit in this embodiment is set as a feedforward neural network with convolution, that is, a convolutional layer is added between the two fully connected layers of the traditional feedforward neural network.

[0092] Schematic, the aforementioned feedforward neural network includes a first fully connected layer, a convolutional layer, and a second fully connected layer. The first fully connected layer extracts features from the combined attention representation to obtain a first feature representation. The convolutional kernels corresponding to the convolutional layer extract the relative positional relationships between image blocks in the first feature representation to obtain a second feature representation. The second fully connected layer extracts features from the second feature representation to obtain an image feature representation.

[0093] Figure 4 This is a flowchart of the specific method for step S204 provided in the embodiments of this application. In some embodiments, see [link to relevant documentation]. Figure 4 The method includes, but is not limited to, steps S401 to S402.

[0094] Step S401: Multiply the image attention weight representation to obtain the attention weight multiplication representation.

[0095] Indicatively, multiple attention heads in each encoder generate corresponding patch attention weight representations, which characterize the attention weights of the corresponding image patches with corresponding category features. By multiplying the image attention weight representations, we can fit the patch attention weight representations generated by each attention head in each encoder to obtain the overall attention weights of the corresponding image patches with corresponding category features, i.e., the cumulative attention weight representation.

[0096] In some embodiments, the image attention weight representation obtained by each encoder in the encoder layer of the above image recognition model when extracting pixel correlation features of the image to be recognized is obtained. The image attention weight representation is obtained by concatenating and compressing the patch attention weight representations of each patch encoding representation, and contains the feature information of each patch attention weight representation. Specifically, the attention weight cumulative representation is obtained by multiplying the image attention weight representations obtained by each encoder layer. The calculation formula for the attention weight cumulative representation is:

[0097] ,

[0098] ,

[0099] in, The attention weights are represented by a cumulative multiplication. Let be the image attention weight representation obtained by the s-th encoder, where s is a positive integer, s∈[1,L], and L is the number of encoders. , , and , where are the attention weights of the patch obtained by the s-th encoder, and k is the number of attention heads.

[0100] Step S402: In the cumulative multiplication representation of indexed attention weights, the attention weight representations of the patches with the strongest and weakest attention strengths are obtained to obtain the attention weight index representation.

[0101] In some embodiments, the attention weight cumulative representation is indexed, and the aggregated index is used to obtain the tile attention weight representation with the strongest attention intensity, representing the set of tile attention weight representations with the strongest attention intensity for the corresponding category feature, thus obtaining the strongest attention representation. The aggregated index is used to obtain the tile attention weight representation with the weakest attention intensity, representing the set of tile attention weight representations with the weakest attention intensity for the corresponding category feature, thus obtaining the weakest attention representation, that is, the attention weight index representation is obtained.

[0102] The strongest attention representation and the weakest attention representation can be expressed as:

[0103] ,

[0104] ,

[0105] in, Representing the highest level of attention, This represents the weakest level of attention. , , and This represents the attention weight representation of the patch with the strongest attention intensity in the cumulative attention weight representation. , , and This represents the attention weight representation of the patch with the weakest attention intensity in the cumulative attention weight representation.

[0106] In some embodiments, step S205 specifically includes: indexing the corresponding patch feature representations in the image feature representation based on the attention weight index representation to obtain a first target feature representation and a second target feature representation; concatenating the first target feature representation and the second target feature representation to obtain an image feature index representation. The first target feature representation is composed of the patch feature representation corresponding to the patch attention weight representation with the strongest attention intensity, and the second target feature representation is composed of the patch feature representation corresponding to the patch attention weight representation with the weakest attention intensity.

[0107] In this embodiment, the image feature indexing layer of the image recognition model indexes the corresponding patch feature representations in the image feature representation based on the mapping relationship between the patch attention weight representation and the patch feature representation in the attention weight index representation. This yields a first target feature representation composed of the patch feature representation corresponding to the patch with the strongest attention weight representation and a second target feature representation composed of the patch feature representation corresponding to the patch with the weakest attention weight representation. The first and second target feature representations are then concatenated to balance the influence of the patch attention weight representation with the strongest and weakest attention weight representation on the image prediction result, thus obtaining the image feature index representation.

[0108] The first target feature representation and the second target feature representation can be expressed as:

[0109] ,

[0110] ,

[0111] in, The first target feature is represented as follows. This represents the second objective feature. , , and This represents the patch feature representation corresponding to the attention weight representation of the patch with the strongest attention intensity in the image feature representation. , , and This represents the patch feature representation corresponding to the attention weight representation of the patch with the weakest attention intensity in the image feature representation.

[0112] In some embodiments, concatenating the first target feature representation and the second target feature representation specifically includes: performing a weighted operation and activation operation on the first target feature representation using a first learnable parameter to obtain a first intermediate parameter; performing a weighted operation and activation operation on the second target feature representation using a second learnable parameter to obtain a second intermediate parameter; calculating a first weight parameter and a second weight parameter based on the first and second intermediate parameters; constructing the value range of the first weight parameter and the second weight parameter and determining the values ​​of the first weight parameter and the second weight parameter; and performing a weighted fitting operation on the first target feature representation and the second target feature representation using the first weight parameter and the second weight parameter to obtain an image feature index representation.

[0113] The formulas for calculating the first and second intermediate parameters are:

[0114] ,

[0115] ,

[0116] in, As the first intermediate parameter, As the second intermediate parameter, As the first learnable parameter, The second learnable parameter is c, which is a positive integer, c∈[1,k], and softplus() is the activation operation.

[0117] To stabilize the first and second intermediate parameters and ensure that the first and second weight parameters are relatively balanced, the first and second intermediate parameters are normalized within a fixed range of values. This allows the values ​​of the first and second weight parameters to fluctuate within a certain range, avoiding extreme large or small values ​​and stabilizing the gradient descent process.

[0118] The formulas for calculating the first and second weight parameters are:

[0119] ,

[0120] ,

[0121] Where a is the first weight parameter and b is the second weight parameter.

[0122] After obtaining the first weight parameter and the second weight parameter, the product between the first weight parameter and the first target feature representation and the product between the second weight parameter and the second target feature representation are calculated. The two calculated products are then concatenated to obtain the image feature index representation.

[0123] Figure 5This is a flowchart of the specific method for step S206 provided in the embodiments of this application. In some embodiments, see [link to relevant documentation]. Figure 5 The method includes, but is not limited to, steps S501 to S503.

[0124] Step S501: Normalize the image feature index representation to obtain the image feature sequence.

[0125] Step S502: Add a classification head to the image feature sequence to construct an image classification feature representation.

[0126] Step S503: Perform multi-label classification on the image classification feature representation to obtain the image recognition result.

[0127] In this embodiment, the normalization layer in the encoder layer of the image recognition model normalizes the image feature index representation to ensure that the data of the image feature index representation is uniformly distributed, thereby obtaining the image feature sequence. Then, a classification head is added to the image feature sequence to construct the image classification feature representation.

[0128] In this embodiment, the classification layer of the image recognition model performs multi-label classification on the image classification feature representation output by the canonical layer to obtain the image recognition result of the image to be recognized.

[0129] Please see Figure 6 This application also provides an image recognition device that can implement the above-described image recognition method. The device includes:

[0130] The first module 601 is used to acquire the image to be recognized; the image to be recognized includes multiple image blocks.

[0131] The second module 602 is used to perform feature encoding on the image to be recognized, so as to obtain the patch encoding representation of the image blocks;

[0132] The third module 603 is used to extract pixel correlation features from the image to be recognized based on a multi-head self-attention mechanism and through patch coding representation, so as to obtain the image attention weight representation and image feature representation of the image to be recognized.

[0133] The fourth module 604 is used to perform feature indexing on the image attention weight representation to obtain the attention weight index representation; the attention weight index representation includes the attention weight representation of the patch with the strongest attention intensity and the attention weight representation of the patch with the weakest attention intensity in the image attention weight representation;

[0134] The fifth module 605 is used to perform feature indexing on the image feature representation based on the attention weight index representation to obtain the image feature index representation;

[0135] The sixth module 606 is used for image recognition based on image feature index representation to obtain image recognition results.

[0136] The specific implementation of this image recognition device is basically the same as the specific implementation of the image recognition method described above, and will not be repeated here.

[0137] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0138] The following reference Figure 7 To describe an electronic device 700 according to such an embodiment of the present disclosure. Figure 7 The electronic device 700 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0139] like Figure 7 As shown, the electronic device 700 is presented in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, a bus 730 connecting different system components (including storage unit 720 and processing unit 710), a display unit 740, etc.

[0140] The storage unit stores program code, which can be executed by the processing unit 710, causing the processing unit 710 to perform the steps described in the above-described image recognition method section of this specification according to various exemplary embodiments of this disclosure.

[0141] Storage unit 720 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 7201 and / or cache memory 7202, and may further include a read-only memory (ROM) 7203.

[0142] The storage unit 720 may also include a program / utility 7204 having a set (at least one) program module 7205, such program module 7205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0143] Bus 730 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0144] Electronic device 700 can also communicate with one or more external devices 700' (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 700, and / or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 750. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 760. Network adapter 760 can communicate with other modules of electronic device 700 via bus 730. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0145] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image recognition method described above.

[0146] The image recognition method, apparatus, device, and storage medium provided in this application, when recognizing target objects in an image to be recognized, perform feature indexing on the image feature representation obtained by sequentially extracting pixel correlation features from the image to be recognized through feature encoding and multi-head self-attention mechanism, based on the strongest and weakest attention weight representations of the image attention weight representations of the image to be recognized. This results in the indexed image feature representation containing both the strongest and weakest attention weight representations of the image block, and image recognition is performed based on the image feature index representation. This fully considers the feature information of both the strongest and weakest attention regions in the image to be recognized, resulting in image recognition results with good fine-grained image recognition accuracy.

[0147] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the methods described above according to the embodiments of this disclosure.

[0148] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0149] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0150] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0151] Exemplary embodiments of this disclosure have been specifically shown and described above. It should be understood that this disclosure is not limited to the detailed structures, arrangements, or implementations described herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. An image recognition method, characterized in that, include: Acquire the image to be identified; the image to be identified includes multiple image blocks; The image to be identified is feature-encoded to obtain the patch-encoded representation of the image patch; Based on the multi-head self-attention mechanism, pixel correlation features are extracted from the image to be identified through the patch encoding representation to obtain the image attention weight representation and image feature representation of the image to be identified. The image attention weight representation is indexed by features to obtain the attention weight index representation; the attention weight index representation includes the set of patch attention weight representations with the strongest attention intensity and the set of patch attention weight representations with the weakest attention intensity in the image attention weight representation; Based on the attention weight index representation, feature indexing is performed on the image feature representation to obtain the image feature index representation; Image recognition is performed based on the image feature index representation to obtain the image recognition result; The step of performing feature indexing on the image feature representation based on the attention weight index representation to obtain the image feature index representation includes: Based on the attention weight index representation, the patch feature representations in the image feature representation are indexed to obtain a first target feature representation and a second target feature representation; the first target feature representation is composed of the patch feature representation corresponding to the patch attention weight representation with the strongest attention intensity, and the second target feature representation is composed of the patch feature representation corresponding to the patch attention weight representation with the weakest attention intensity. The image feature index representation is obtained by concatenating the first target feature representation and the second target feature representation.

2. The image recognition method according to claim 1, characterized in that, The step of performing feature encoding on the image to be identified to obtain the patch-coded representation of the image patch includes: Perform a linear mapping on the image patch to obtain a linear mapping vector; The linear mapping vector is embedded with positions to obtain the tile-coded representation.

3. The image recognition method according to claim 1, characterized in that, The multi-head self-attention mechanism extracts pixel correlation features from the image to be identified through the patch encoding representation, obtaining the image attention weight representation and image feature representation of the image to be identified, including: The tile attention weight representation of the tile encoding is determined based on a multi-head attention mechanism; The attention weight representations of each of the aforementioned image patches are concatenated and compressed to obtain the image attention weight representation. The image attention weight representation is subjected to feature mapping processing to obtain the image feature representation.

4. The image recognition method according to claim 1, characterized in that, The step of performing feature indexing on the image attention weight representation to obtain the attention weight index representation includes: The attention weight representation of the image is multiplied to obtain the attention weight multiplication representation; The attention weight index representation is obtained by indexing the attention weight representations of the patch with the strongest and weakest attention intensity in the attention weight cumulative representation representation.

5. The image recognition method according to claim 1, characterized in that, The step of concatenating the first target feature representation and the second target feature representation to obtain the image feature index representation includes: The first target feature representation is weighted and activated using a first learnable parameter to obtain a first intermediate parameter; the second target feature representation is weighted and activated using a second learnable parameter to obtain a second intermediate parameter. Based on the first intermediate parameter and the second intermediate parameter, calculate the first weight parameter and the second weight parameter, construct the value range of the first weight parameter and the second weight parameter, and determine the value of the first weight parameter and the second weight parameter. The first target feature representation and the second target feature representation are weighted and fitted using the first weight parameter and the second weight parameter to obtain the image feature index representation.

6. The image recognition method according to claim 1, characterized in that, The image recognition based on the image feature index representation to obtain the image recognition result includes: The image feature index representation is normalized to obtain an image feature sequence; Add a classification header to the image feature sequence to construct an image classification feature representation; The image classification feature representation is subjected to multi-label classification to obtain the image recognition result.

7. An image recognition device, characterized in that, include: The first module is used to acquire the image to be identified; the image to be identified includes multiple image blocks. The second module is used to perform feature encoding on the image to be identified to obtain the patch encoding representation of the image block; The third module is used to extract pixel correlation features from the image to be identified based on the multi-head self-attention mechanism and the patch encoding representation, so as to obtain the image attention weight representation and image feature representation of the image to be identified. The fourth module is used to perform feature indexing on the image attention weight representation to obtain an attention weight index representation; the attention weight index representation includes the set of patch attention weight representations with the strongest attention intensity and the set of patch attention weight representations with the weakest attention intensity in the image attention weight representation; The fifth module is used to perform feature indexing on the image feature representation based on the attention weight index representation to obtain the image feature index representation; The sixth module is used to perform image recognition based on the image feature index representation to obtain the image recognition result; The step of performing feature indexing on the image feature representation based on the attention weight index representation to obtain the image feature index representation includes: Based on the attention weight index representation, the patch feature representations in the image feature representation are indexed to obtain a first target feature representation and a second target feature representation; the first target feature representation is composed of the patch feature representation corresponding to the patch attention weight representation with the strongest attention intensity, and the second target feature representation is composed of the patch feature representation corresponding to the patch attention weight representation with the weakest attention intensity. The image feature index representation is obtained by concatenating the first target feature representation and the second target feature representation.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the image recognition method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the image recognition method according to any one of claims 1 to 6.

Images