Image recognition system and training method therefor, image recognition method

Abstract

The application belongs to the technical field of image recognition, and discloses an image recognition system, a training method thereof and an image recognition method. In the system, a coarse-grained scene perception module identifies the most relevant scene of an input image; a fine-grained target perception module calculates the similarity score of each feature point and each target in the most relevant scene based on the information provided by the coarse-grained scene perception module; an encoding module has a multi-branch self-attention calculation unit, the multi-branch attention calculation unit adds the salient feature points to the same self-attention branch based on the similarity score to form a multi-branch self-attention image encoding, and focuses on encoding calculation of the added salient features; and a decoding module decodes the encoded features to identify the targets in the input image. Through the above system, the calculation complexity of the encoder can be reduced, the feature extraction capability of the key targets can be improved, the interference of different categories of targets and image background regions can be reduced, and the image processing speed can be accelerated.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, and more specifically, relates to an image recognition system and its training method, as well as an image recognition method. Background Technology

[0002] Object detection is one of the most fundamental and core tasks in computer vision, with wide applications in real-world scenarios such as autonomous driving, medical imaging, and drone detection. Modern object detection methods can be mainly divided into CNN-based architectures and Transformer-based architectures. In recent years, Transformer-based end-to-end detectors (DETR) have brought significant performance improvements. This architecture follows a two-stage detection process: after extracting image features using models such as ResNet, dense self-attention image encoding is performed in the encoder, and cross-attention decoding is performed in the decoder using query vectors.

[0003] Especially in the encoding stage, the encoder encodes image features through self-attention. When processing high-resolution images or utilizing multi-scale features, the computational complexity is high. In addition, a large amount of background information in the image can lead to encoding redundancy, further increasing the encoding complexity.

[0004] Therefore, how to reduce the computational complexity of the encoding stage and improve the image processing speed is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides an image recognition system and its training method and image recognition method, the purpose of which is to reduce the computational complexity of the encoding stage and improve the image processing speed.

[0006] To achieve the above objectives, the present invention provides an image recognition system, comprising:

[0007] The feature extraction module is used to extract features from the input image and obtain the feature vectors of each feature point;

[0008] The coarse-grained scene perception module includes a scene semantic generation unit and a first similarity calculation unit. The scene semantic generation unit generates scene semantic vectors for various scenes in a preset scene set, and the first similarity calculation unit pools all feature vectors to obtain a global feature i. avg Then select i avg The scene p with the highest similarity and its scene semantic vector s′ p ;

[0009] The fine-grained target perception module includes a fusion unit, a target semantic generation unit, and a second similarity calculation unit. The fusion unit is used to embed the target text of various targets contained in scene p into vector s′. p The joint embedding representations of various targets are obtained by concatenation. The target semantic generation unit is used to map each joint embedding representation into a target semantic vector. The second similarity calculation unit is used to obtain the similarity score between each feature point and various targets by calculating the similarity of the vectors.

[0010] The Transformer encoding module is used to encode all feature vectors to obtain encoded features. The Transformer encoding module has a multi-branch self-attention calculation unit. The multi-branch attention calculation unit is used to search for feature points with similarity scores exceeding a threshold as salient feature points. For each salient feature point, a preset number of targets with the highest similarity scores are searched as candidate targets. All salient feature points corresponding to the same candidate target are added to the same self-attention branch, so that all self-attention branches perform self-attention calculations respectively.

[0011] The Transformer decoding module is used to decode the encoded features and identify the target in the input image.

[0012] Optionally, the Transformer decoding module obtains a series of prior feature points from the output of the Transformer encoding module and constructs a content query vector and a location query vector corresponding to each prior feature point. Based on the content query vector, the target category is decoded, and based on the location query vector, the target location is decoded.

[0013] For each prior feature point, its content query vector q c The trainable query vector q of the prior feature point c1 It is formed by concatenating the fused semantic vector e, which is obtained by weighted summation of a preset number of target semantic vectors that have the highest similarity score with the encoded features of the prior feature point. The weight of each target semantic vector is positively correlated with the similarity between the target semantic vector and the encoded features of the prior feature point.

[0014] Optionally, both the scene semantic generation unit and the target semantic generation unit are trainable multilayer perceptrons;

[0015] The coarse-grained scene perception module also includes a scene category text embedding layer, which is used to convert the language descriptions of various scenes into scene text embeddings and send them to the scene semantic generation unit to obtain the scene semantic vectors of various scenes.

[0016] The fine-grained target perception module also includes a target category text embedding layer, which is used to convert the language descriptions of various targets into target text embeddings and send them to the fusion unit for splicing to obtain the joint embedding representation of various targets.

[0017] Optionally, both the scene category text embedding layer and the target category text embedding layer use the text encoder in the CLIP model.

[0018] Optionally, the feature extraction module may use an image encoder from the CLIP model.

[0019] The present invention also provides a training method for an image recognition system, comprising:

[0020] Acquire labeled images as a training set, wherein the labels mark targets in the images;

[0021] The image recognition system described above is trained using the data in the training set and the loss function.

[0022] Optionally, the Transformer decoding module in the image recognition system obtains a series of prior feature points from the output of the Transformer encoding module and constructs a content query vector and a location query vector corresponding to each prior feature point. It decodes the target category based on the content query vector and the target location based on the location query vector. For each prior feature point, its content query vector q... c The trainable query vector q of the prior feature point c1 It is formed by concatenating the fused semantic vector e, which is obtained by weighted summation of a preset number of target semantic vectors with the highest similarity score to the prior feature point. The weight of each target semantic vector is proportional to the feature similarity between the target and the prior feature point.

[0023] The training method includes grouping the content query vectors of all prior feature points, and the loss function used includes inter-group contrastive learning loss and intra-group contrastive learning loss. Specifically, content query vectors of all prior feature points with identical candidate target category sets are grouped into the same query group. The target category set of the prior feature point is a preset number of targets with the highest similarity score to that prior feature point. The inter-group contrastive learning loss is used to suppress the similarity of query vectors between different query groups using average cross-entropy loss. The intra-group contrastive learning loss is an intra-class clustering loss function with minimum margin. When the similarity between any two vectors within a query group is higher than the minimum margin threshold, the intra-group contrastive learning loss increases with the increase in similarity.

[0024] The present invention also provides an image recognition method, which includes: inputting an image to be recognized into a trained image recognition system as described in any of the preceding claims, to obtain the target in the input image.

[0025] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the preceding claims.

[0026] The present invention also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the method described in any of the preceding claims.

[0027] In summary, compared with the prior art, the technical solutions conceived in this invention have the following main advantages:

[0028] 1. The image recognition system proposed in this invention comprises a coarse-grained scene perception module that identifies the scene most relevant to the input image, a fine-grained target perception module that calculates the similarity score between each feature point and each target in the most relevant scene based on the information provided by the coarse-grained scene perception module, and a Transformer encoding module that performs grouping and filtering based on the similarity score provided by the fine-grained target perception module. The higher the similarity score with the target, the more important the feature point is, and it is selected as a salient feature point. Furthermore, the correlation between salient feature points associated with the same target is also stronger. The salient feature points with strong correlation are added to the same self-attention branch to form a multi-branch self-attention image encoding. The encoding calculation is focused on the added salient features, thereby reducing the computational complexity of the encoder, improving the feature extraction capability of key targets, reducing interference from different types of targets and image background areas, and accelerating image processing speed.

[0029] 2. Optionally, the Transformer decoding module can be optimized by adding semantic prior information to its content query vector to improve the model's ability to perceive key targets and improve image recognition performance;

[0030] 3. Optionally, during training, inter-group contrastive learning loss and intra-group contrastive learning loss can be added. Based on contrastive learning, group queries are guided to detect specific categories of objects. The model's ability to perceive key targets is further enhanced by optimizing the decoder, thereby accelerating the convergence speed of the decoder. Attached Figure Description

[0031] Figure 1 This is a structural block diagram of an image recognition system according to an embodiment of the present invention;

[0032] Figure 2 This is a structural diagram of an image recognition system according to an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0034] like Figure 1 The diagram shown is a structural block diagram of an image recognition system according to an embodiment of the present invention. Figure 2 The diagram shown is a specific structural diagram of an image recognition system according to an embodiment of the present invention, which includes a feature extraction module, a coarse-grained scene perception module, a fine-grained target perception module, a Transformer encoding module, and a Transformer decoding module.

[0035] The feature extraction module is used to extract features from the input image and obtain the feature vectors of each feature point.

[0036] Specifically, this feature extraction module can be an image encoder in CLIP, using a convolutional neural network such as ResNet to extract image features I = (i1, i2, ..., i...). N ), where N is the number of image feature points.

[0037] The coarse-grained scene perception module includes a scene semantic generation unit and a first similarity calculation unit. The scene semantic generation unit generates scene semantic vectors for various scenes in a preset scene set, and the first similarity calculation unit pools all feature vectors to obtain the global feature i. avg Then select i avg The scene p with the highest similarity and its scene semantic vector s′ p .

[0038] This coarse-grained scene perception module utilizes the strong correlation between global image features and scene semantics to perform coarse-grained scene classification of images, thereby improving the model's ability to detect key targets in application scenarios.

[0039] Specifically, a set of target scenarios is first pre-defined, which includes multiple scenarios such as road traffic scenarios, sports event scenarios, and assembly / activity scenarios. Scene semantic vectors for each scenario are then generated using a scene semantic generation unit.

[0040] In one embodiment, the scene semantic generation unit can be a trainable multilayer perceptron (MLP). For example, a scene category text embedding layer can be set up to convert the language descriptions of various scenes into scene text embeddings and send them to the multilayer perceptron to obtain scene semantic vectors for various scenes. The scene category text embedding layer can directly use the text encoder in a pre-trained CLIP model; this scene category text embedding layer does not need to be retrained when training the image recognition system.

[0041] For example, there are P scene categories, and the language description of each scene is represented as a scene. text Inputting these into the text encoder in the CLIP model yields the scene text embeddings s1, s2, ..., s1 corresponding to P scene categories. P Then, the scene texts of P scene categories are embedded into the multilayer perceptron to obtain scene semantic vectors s′1, s′2, ..., s′ corresponding to the P scene categories. P The specific calculation process is as follows:

[0042] (s1,s2,…,s P = CLIP(scene) text )#(1)

[0043] (s′1,s′2,…,s′ P )=σ(W2(σ(W1(s1,s2,…,s P )+b1))+b2)#(2)

[0044] In the formula, W1 and W2 are the weight matrices of the multilayer perceptron, b1 and b2 are the bias vectors of the multilayer perceptron, and σ is the nonlinear activation function of the multilayer perceptron (such as ReLU).

[0045] The first similarity calculation unit is used to pool all feature vectors to obtain the global feature i. avg Then select i avg The scene p with the highest similarity and its scene semantic vector s′ p In practice, first, all feature vectors output by the feature extraction module are processed. Average pooling The global feature representation of the image, i, is obtained. avg Then, calculate the semantic vectors of each scene and the global feature representation i. avg Similarity, such as calculating global image features i avg With scene category semantic vector (s′1,s′2,…,s′) P The cosine similarity between i and i is used to select the one with the most similarity. avg The scene p with the highest similarity is selected as the most relevant scene to the input image and proceeds to subsequent processing.

[0046] The fine-grained target perception module includes a fusion unit, a target semantic generation unit, and a second similarity calculation unit. The fusion unit is used to embed the target text of various targets contained in scene p into the vector s′. p The joint embedding representations of various targets are obtained by concatenation. The target semantic generation unit is used to map each joint embedding representation into a target semantic vector. The second similarity calculation unit is used to obtain the similarity score between each feature point and various targets by calculating the similarity of the vectors.

[0047] Specifically, the fine-grained target semantic perception module uses the semantic information of potential key target categories in the most relevant scenes to perform fine-grained classification of image features, assisting the encoder in achieving multi-branch attention encoding.

[0048] In this process, the target types corresponding to each scenario are pre-defined. For example, target types in road traffic typically include cars, electric vehicles, pedestrians, traffic police, and streetlights. After the coarse-grained scene perception module identifies the most relevant scenario p, the fine-grained target perception module then calculates the similarity scores between each feature point and each type of target.

[0049] First, we can obtain the target text embeddings for each type of target under the most relevant scene p. Specifically, we can set up a target category text embedding layer to convert the language descriptions of each type of target into target text embeddings, which are then sent to the fusion unit for concatenation to obtain the joint embedding representation of each type of target. The target text embedding layer can directly use the text encoder in the pre-trained CLIP model. When training the image recognition system, this scene category text embedding layer does not need to be trained again.

[0050] For example, in the most relevant scenario p, there are M target categories. The linguistic descriptions of each target category are represented as objects. text Inputting this into the text encoder in the CLIP model yields the target text embeddings o1, o2, ..., o for each of the M target categories. M Then, the target texts of the M target categories are embedded into the fusion unit to obtain the joint embedding representations o′1, o′2, ..., o′ corresponding to the M target categories. M The specific calculation process is as follows:

[0051] (o1,o2,…,o M = CLIP(object) text )#(3)

[0052] (o′1,o′2,…,o′ M )=([o1;s′ p ],[o2;s′ p ],…,[o M ;s′ p])# (4)

[0053] Among them, (o′1,o′2,…,o′ M ) represents the joint embedding representation of the target category in the most relevant scenario, and M represents the number of target categories in the scenario.

[0054] In one embodiment, the target semantic generation unit can also be a trainable multilayer perceptron, which generates joint embedding representations o′1, o′2, ..., o′ corresponding to M target categories. M Inputting the data into a multilayer perceptron yields target semantic vectors e1, e2, ..., e corresponding to M target categories. M Specifically, it can be expressed as:

[0055] (e1,e2,…,e M )=σ(W4(σ(W3(o′1,o′2,…,o′ M )+b3))+b4)#(5)

[0056] In the formula, W3 and W4 are weight matrices, and b3 and b4 are bias vectors.

[0057] The second similarity calculation unit is used to obtain the similarity score between each feature point and various types of targets by calculating the similarity of vectors. In a specific embodiment, the similarity of vectors can be calculated using the following formula:

[0058]

[0059] In the formula, A nm This represents the similarity score between feature point n and target category m.

[0060] Through the above fine-grained target perception module, the similarity matrix A can be obtained. N×M By calculating the similarity matrix, the degree of association between each image feature point and each target category can be quantified.

[0061] The Transformer encoding module is used to encode all feature vectors to obtain encoded features. The Transformer encoding module has a multi-branch self-attention calculation unit. The multi-branch attention calculation unit is used to search for feature points with similarity scores exceeding a threshold as salient feature points. For each salient feature point, it searches for a preset number of targets with the highest similarity scores as candidate targets. All salient feature points corresponding to the same candidate target are added to the same self-attention branch, so that all self-attention branches perform self-attention calculations respectively.

[0062] The Transformer encoding module typically includes a feature input layer, a self-attention computation layer, a feedforward neural network (FFN), residual connections, and a normalization layer. This invention improves the self-attention computation layer in the Transformer encoding module. Specifically, the self-attention computation is guided by the similarity information output by the fine-grained target perception module. By grouping and filtering feature points, a multi-target category attention mask is generated, and multi-branch self-attention image encoding is performed. This reduces the computational complexity of the encoder, improves the feature extraction capability of key targets, and reduces interference from different target categories and image background regions.

[0063] Specifically, a score threshold θ is first set in advance, and then in the similarity matrix A N×M In this context, for each feature point n, there are M similarity scores A corresponding to M targets. n1 ~A nM If a similarity score exceeds a threshold θ, the feature point is considered a salient feature point. The K most similar targets are selected to form a candidate target set containing K candidate target categories. Each salient feature point is associated with a candidate target set. All candidate targets are organized, and all salient feature points associated with the same candidate target are grouped into a salient feature point set. Multiple candidate targets generate multiple salient feature point sets. Each set forms a corresponding self-attention mask map based on the salient feature points it contains. For example, all salient feature points associated with candidate target k form a self-attention mask matrix. If both the i-th and j-th feature points are significant feature points of the candidate target k, then the element in the i-th row and j-th column of the self-attention mask matrix is ​​1; otherwise, it is 0. Based on the self-attention mask matrix, corresponding self-attention branches are formed. That is, a self-attention branch can be constructed for each candidate target, and different self-attention branches can be constructed for different candidate targets. Each self-attention branch performs self-attention calculation independently. The self-attention calculation formula is as follows:

[0064]

[0065] In the formula, These are the attention parameters in the self-attention branch corresponding to the target category k.

[0066] The Transformer decoding module is used to decode coded features and identify targets in the input image.

[0067] In the above image recognition system, the coarse-grained scene perception module identifies the scene most relevant to the input image. The fine-grained target perception module calculates the similarity score between each feature point and each target in the most relevant scene based on the information provided by the coarse-grained scene perception module. When the Transformer encoding module performs encoding, it groups and filters based on the similarity score provided by the fine-grained target perception module. The higher the similarity score with the target, the more important the feature point is, and it is selected as a salient feature point. Furthermore, the correlation between salient feature points associated with the same target is also stronger. The salient feature points with strong correlation are added to the same self-attention branch to form a multi-branch self-attention image encoding. The encoding calculation focuses on the added salient features, thereby reducing the computational complexity of the encoder, improving the feature extraction capability of key targets, reducing interference from different types of targets and image background areas, and accelerating image processing speed.

[0068] Traditional Transformer decoding modules typically perform target queries based solely on the results from the Transformer encoding module. For example, after the Transformer encoding module outputs the encoded features of each feature point, the Transformer decoding module selects several feature points with the highest confidence as prior feature points based on the confidence of the encoded features, and obtains their content prior and location prior. For each feature point, a content query vector q is then constructed. c and location query vector q l Based on content query vector q c Decode the target category and decode the target location q based on the location query vector. l .

[0069] In one embodiment of the present invention, in order to further enhance the system's ability to perceive key targets, the Transformer decoding module is further improved. For each prior feature point, its content query vector q c The trainable query vector q of the prior feature point c1 It is formed by concatenating the fused semantic vector e, which is obtained by weighted summation of a preset number of target semantic vectors that have the highest similarity score with the encoded features of the prior feature point. The weight of each target semantic vector is positively correlated with the similarity between the target semantic vector and the encoded features of the prior feature point.

[0070] Taking a prior feature point as an example, its encoded feature is denoted as i′. Based on the similarity score calculated by the fine-grained target perception module, the K candidate target categories with the highest similarity scores to the prior feature point are selected. The similarity between the encoded feature of the prior feature point and the target semantic vector of each candidate target category is then calculated. The weight of each candidate target category is determined based on its similarity. The weighted sum of the target semantic vectors of the K candidate target categories yields the fused semantic vector e. The trainable query vector q is then... c1 The content query vector q of the prior feature point is obtained by concatenating the fused semantic vector e with the fused semantic vector e. c .

[0071] In practice, the weight of the target semantic vector for each candidate target category can be obtained by calculating the inner product of the prior feature point encoded features and the target semantic vector of each candidate target category, and then normalizing the product. Let e ​​be the target semantic vector of the j-th candidate target category. j Its weight is α j The content query vector q of the prior feature point c The calculation formula is as follows:

[0072]

[0073] q c =[q c1 ;e]#(12)

[0074] Thus, by querying the content query vector q c Introducing semantic priors can enhance the model's ability to perceive key targets.

[0075] This invention also provides a training method for an image recognition system, used to train the image recognition system described above. The training process includes:

[0076] Acquire labeled images as the training set, with labels indicating targets in the images; train the image recognition system using the data in the training set and a loss function.

[0077] Specifically, images from the training set are input into the system to be trained. The system includes a feature extraction module, a coarse-grained scene perception module, a fine-grained target perception module, a Transformer encoding module, and a Transformer decoding module. The coarse-grained scene perception module includes a scene semantic generation unit and a first similarity calculation unit. The fine-grained target perception module includes a fusion unit, a target semantic generation unit, and a second similarity calculation unit. The Transformer encoding module includes a multi-branch self-attention calculation unit.

[0078] The feature extraction module extracts features from the input image to obtain the feature vectors of each feature point;

[0079] The scene semantic generation unit generates scene semantic vectors for various scenes in a preset scene set, and the first similarity calculation unit pools all feature vectors to obtain the global feature i. avg Then select i avg The scene p with the highest similarity and its scene semantic vector s′ p ;

[0080] The target text of various targets contained in scene p is embedded into vector s′ by the fusion unit. p The joint embedding representations of various targets are concatenated. The joint embedding representations are then mapped to target semantic vectors by the target semantic generation unit. The similarity scores between each feature point and various targets are obtained by calculating the similarity of the vectors by the second similarity calculation unit.

[0081] All feature vectors are encoded by the Transformer encoding module to obtain encoded features. Feature points with similarity scores exceeding a threshold are searched by the multi-branch attention calculation unit as salient feature points. For each salient feature point, a preset number of targets with the highest similarity scores are searched as candidate targets. All salient feature points corresponding to the same candidate target are added to the same self-attention branch, so that all self-attention branches perform self-attention calculations respectively.

[0082] The Transformer decoding module decodes the encoded features to identify targets in the input image;

[0083] The system is trained by acquiring the identified targets and their corresponding labels and calculating the loss function.

[0084] The loss function typically includes one or more of the following: classification confidence loss, bounding box regression loss, and intersection-union ratio loss, used to calculate the deviation between the recognition result and the label.

[0085] In one embodiment, when the content query vector q of the Transformer decoding module c The trainable query vector q from prior feature points c1 When concatenated with the fused semantic vector e, the loss function can be further optimized. The content query vectors of all prior feature points are grouped and the loss function used is increased with inter-group contrast learning loss and intra-group contrast learning loss. Specifically, the content query vectors of all prior feature points with the same candidate target category set are divided into the same query group. The target category set of the prior feature point is the preset number of targets with the highest similarity score to the prior feature point.

[0086] Inter-group contrastive learning loss is used to suppress the similarity of query vectors between different query groups using average cross-entropy loss.

[0087] Specifically, the cosine similarity between queries is calculated, and the average cross-entropy loss is used to suppress the similarity between queries, so that the queries between different groups have greater differences, which is beneficial for query detection of different categories of targets.

[0088] Between-group contrastive learning loss inter It can be represented as:

[0089]

[0090] In the formula, Q represents the content query vector, which is divided into Q groups. This is an indicator function; it evaluates to 0 when Q1 and Q2 are in the same group and to 1 when they are in different groups. This represents the i-th content query vector in Q1. This represents the j-th content query vector in Q2.

[0091] Specifically, the intra-group contrastive learning loss uses an intra-class clustering loss function with minimum margin. This loss function increases when the similarity between any two vectors within a group is higher than the minimum margin threshold m, thereby reducing the similarity of vectors within the group and maintaining the diversity of queries within the group.

[0092] Intragroup contrastive learning loss intra It can be represented as:

[0093]

[0094] Thus, by improving the loss function, we can guide grouped queries to detect objects of specific categories based on contrastive learning, and further enhance the model's ability to perceive key targets by optimizing the decoder, thereby accelerating the convergence speed of the decoder.

[0095] The intra-group contrastive learning loss is the intra-class clustering loss function with minimum margin. When the similarity between any two vectors in a group is higher than the minimum margin threshold, the intra-group contrastive learning loss will increase with the increase of similarity.

[0096] The present invention also relates to an image recognition method, comprising: inputting an image to be recognized into a trained image recognition system described above, and obtaining the target in the input image.

[0097] Specifically, labeled images are obtained as the training set, and the labels mark the targets in the images; the image recognition system is trained using the data in the training set and the loss function.

[0098] Specifically, the image to be recognized is input into the trained system, which includes a feature extraction module, a coarse-grained scene perception module, a fine-grained target perception module, a Transformer encoding module, and a Transformer decoding module. The coarse-grained scene perception module includes a scene semantic generation unit and a first similarity calculation unit, the fine-grained target perception module includes a fusion unit, a target semantic generation unit, and a second similarity calculation unit, and the Transformer encoding module includes a multi-branch self-attention calculation unit.

[0099] The feature extraction module extracts features from the input image to obtain the feature vectors of each feature point;

[0100] The scene semantic generation unit generates scene semantic vectors for various scenes in a preset scene set, and the first similarity calculation unit pools all feature vectors to obtain the global feature i. avg Then select i avg The scene p with the highest similarity and its scene semantic vector s′ p ;

[0101] The target text of various targets contained in scene p is embedded into vector s′ by the fusion unit. p The joint embedding representations of various targets are concatenated. The joint embedding representations are then mapped to target semantic vectors by the target semantic generation unit. The similarity scores between each feature point and various targets are obtained by calculating the similarity of the vectors by the second similarity calculation unit.

[0102] All feature vectors are encoded by the Transformer encoding module to obtain encoded features. Feature points with similarity scores exceeding a threshold are searched by the multi-branch attention calculation unit as salient feature points. For each salient feature point, a preset number of targets with the highest similarity scores are searched as candidate targets. All salient feature points corresponding to the same candidate target are added to the same self-attention branch, so that all self-attention branches perform self-attention calculations respectively.

[0103] The Transformer decoding module decodes the encoded features to identify targets in the input image.

[0104] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0105] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0106] This invention also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the method described in the above embodiments of this invention.

[0107] In summary, the system described in this invention comprises a coarse-grained scene perception module that utilizes global image features to classify images at the coarse-grained scene category level; a fine-grained target perception module that extracts semantic information of candidate target categories in specific scenes for coarse image detection; and a multi-branch self-attention computation unit in the Transformer encoding module that guides the image encoder to perform multi-branch attention encoding based on semantic information at the fine-grained key target category level. Combining these three modules reduces the computational complexity of the encoder, improves the feature extraction capability of key targets, reduces interference from different target categories and image background regions, and accelerates image processing speed. Furthermore, the Transformer decoding module is optimized by adding semantic prior information to its content query vector, enhancing the model's ability to perceive key targets and improving image recognition performance. Correspondingly, during training, inter-group contrastive learning loss and intra-group contrastive learning loss are added. Contrast learning guides grouped queries to detect specific category objects, further enhancing the model's ability to perceive key targets by optimizing the decoder, thereby accelerating the decoder's convergence speed.

[0108] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. It should be noted that the terms "in one embodiment," "for example," and "again" are intended to illustrate the present invention and are not intended to limit the present invention.

[0109] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. An image recognition system, characterized in that, include: The feature extraction module is used to extract features from the input image and obtain the feature vectors of each feature point; The coarse-grained scene perception module includes a scene semantic generation unit and a first similarity calculation unit. The scene semantic generation unit is used to generate scene semantic vectors for various scenes in a preset scene set, and the first similarity calculation unit is used to pool all feature vectors to obtain global features. The next selection was with The most similar scene and its scene semantic vector ; The fine-grained target perception module includes a fusion unit, a target semantic generation unit, and a second similarity calculation unit. The fusion unit is used to integrate the scene... The target text embeddings and vectors of various types of targets included The joint embedding representations of various targets are obtained by concatenation. The target semantic generation unit is used to map each joint embedding representation into a target semantic vector. The second similarity calculation unit is used to obtain the similarity score between each feature point and various targets by calculating the similarity of the vectors. The Transformer encoding module is used to encode all feature vectors to obtain encoded features. The Transformer encoding module has a multi-branch self-attention calculation unit. The multi-branch attention calculation unit is used to search for feature points with similarity scores exceeding a threshold as salient feature points. For each salient feature point, a preset number of targets with the highest similarity scores are searched as candidate targets. All salient feature points corresponding to the same candidate target are added to the same self-attention branch, so that all self-attention branches perform self-attention calculations respectively. The Transformer decoding module is used to decode the encoded features and identify the target in the input image; The Transformer decoding module obtains a series of prior feature points from the output of the Transformer encoding module and constructs a content query vector and a location query vector corresponding to each prior feature point. It decodes the target category based on the content query vector and decodes the target location based on the location query vector. For each prior feature point, its content query vector The trainable query vector of the prior feature point and fusion semantic vector The fused semantic vector is formed by concatenation. The weighted sum of the target semantic vectors that have the highest similarity scores with the encoded features of the prior feature point is obtained by weighted summation. The weight of each target semantic vector is positively correlated with the similarity between the target semantic vector and the encoded features of the prior feature point. Both the scene semantic generation unit and the target semantic generation unit are trainable multilayer perceptrons. The coarse-grained scene perception module also includes a scene category text embedding layer, which is used to convert the language descriptions of various scenes into scene text embeddings and send them to the scene semantic generation unit to obtain the scene semantic vectors of various scenes. The fine-grained target perception module also includes a target category text embedding layer, which is used to convert the language descriptions of various targets into target text embeddings and send them to the fusion unit for splicing to obtain the joint embedding representation of various targets.

2. The image recognition system as described in claim 1, characterized in that, Both the scene category text embedding layer and the target category text embedding layer use the text encoder in the CLIP model.

3. The image recognition system as described in claim 1, characterized in that, The feature extraction module uses the image encoder in the CLIP model.

4. A training method for an image recognition system, characterized in that, include: Acquire labeled images as a training set, wherein the labels mark targets in the images; The image recognition system as described in any one of claims 1 to 3 is trained using data from the training set and a loss function. The training method includes grouping the content query vectors of all prior feature points, and the loss function used includes inter-group contrastive learning loss and intra-group contrastive learning loss. Specifically, content query vectors of all prior feature points with identical candidate target category sets are grouped into the same query group. The target category set of the prior feature point is a preset number of targets with the highest similarity score to that prior feature point. The inter-group contrastive learning loss is used to suppress the similarity of query vectors between different query groups using average cross-entropy loss. The intra-group contrastive learning loss is an intra-class clustering loss function with minimum margin. When the similarity between any two vectors within a query group is higher than the minimum margin threshold, the intra-group contrastive learning loss increases with the increase in similarity.

5. An image recognition method, characterized in that, include: The image to be recognized is input into the trained image recognition system as described in any one of claims 1 to 3 to obtain the target in the input image.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in claim 4 or 5.

7. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the steps of the method as described in claim 4 or 5.

Images