Image recognition system, training method thereof and image recognition method

Abstract

The invention belongs to the related technical field of image recognition, and discloses an image recognition system, a training method thereof and an image recognition method, in the system, a coarse-grained scene sensing module recognizes a scene most related to an input image; the fine-granularity target sensing module calculates the similarity score of each feature point and each target in the most relevant scene based on the information provided by the coarse-granularity scene sensing module; the coding module is provided with a multi-branch self-attention calculation unit, the multi-branch self-attention calculation unit adds the saliency feature points into the same self-attention branch based on the similarity score to form multi-branch self-attention image coding, and coding calculation is emphasized on the added saliency features; and the decoding module decodes the coding features and identifies the target in the input image. Through the system, the calculation complexity of the encoder can be reduced, the feature extraction capability of key targets can be improved, the interference of different types of targets and image background areas can be reduced, and the image processing speed can be accelerated.

Description

Technical Field

[0001] The present invention belongs to the technical field related to image recognition, and more specifically, relates to an image recognition system and a training method thereof, and an image recognition method. Background Art

[0002] Object detection is one of the most basic and core tasks in computer vision, and is widely used in practical scenarios such as autonomous driving, medical imaging, and drone detection. Modern object detection methods can be mainly divided into CNN-based architectures and Transformer-based architectures. In recent years, the Transformer-based end-to-end detector (DETR) has brought significant performance improvements. This architecture follows a two-stage detection step. After extracting image features through models such as Resnet, dense self-attention image encoding is performed in the encoder, and cross-attention decoding is performed using the query vector in the decoder.

[0003] Especially in the encoding stage, the encoder encodes image features through self-attention, which has high computational complexity when processing high-resolution images or utilizing multi-scale features. In addition, a large amount of background information in the image will lead to encoding redundancy, further increasing the complexity of encoding.

[0004] Therefore, how to reduce the computational complexity of the encoding stage and improve the image processing speed is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention provides an image recognition system and a training method thereof, and an image recognition method, the purpose of which is to reduce the computational complexity of the encoding stage and improve the image processing speed.

[0006] To achieve the above object, the present invention provides an image recognition system, comprising:

[0007] The feature extraction module is used to extract the features of the input image and obtain the feature vector of each feature point;

[0008] The coarse-grained scene perception module includes a scene semantics generation unit and a first similarity calculation unit, wherein the scene semantics generation unit is used to generate scene semantic vectors of various scenes in a preset scene set, and the first similarity calculation unit is used to pool all feature vectors to obtain a global feature i avg Then select avg The scene p with the highest similarity and its scene semantic vector s′ p ;

[0009] The fine-grained target perception module includes a fusion unit, a target semantic generation unit and a second similarity calculation unit. The fusion unit is used to embed the target text of each type of target contained in the scene p into the vector s′ p The joint embedding representation of each type of target is obtained by splicing, the target semantic generation unit is used to map each of the joint embedding representations into a target semantic vector, and the second similarity calculation unit is used to obtain a similarity score between each feature point and each type of target by calculating the similarity of the vector;

[0010] A Transformer encoding module is used to encode all feature vectors to obtain encoding features. The Transformer encoding module has a multi-branch self-attention calculation unit. The multi-branch attention calculation unit is used to search for feature points with similarity scores exceeding a threshold as significant feature points. For each significant feature point, a preset number of targets with the highest similarity scores are searched as candidate targets. All significant feature points corresponding to the same candidate target are added to the same self-attention branch, so that all self-attention branches perform self-attention calculations respectively.

[0011] The Transformer decoding module is used to decode the encoded features and identify the target in the input image.

[0012] Optionally, the Transformer decoding module obtains a series of prior feature points from the output result of the Transformer encoding module and constructs a content query vector and a position query vector corresponding to each prior feature point, decodes the target category based on the content query vector, and decodes the target position based on the position query vector;

[0013] For each prior feature point, its content query vector q c The trainable query vector q of the prior feature point c1 The fused semantic vector e is obtained by weighted summing up a preset number of target semantic vectors with the highest similarity scores to the encoding features of the prior feature point, and the weight of each target semantic vector is positively correlated with the similarity between the target semantic vector and the encoding features of the prior feature point.

[0014] Optionally, the scene semantics generation unit and the target semantics generation unit are both trainable multi-layer perceptrons;

[0015] The coarse-grained scene perception module also includes a scene category text embedding layer, which is used to convert the language description of each scene into a scene text embedding and then send it to the scene semantic generation unit to obtain the scene semantic vector of each scene;

[0016] The fine-grained target perception module also includes a target category text embedding layer, which is used to convert the language description of each type of target into a target text embedding and then send it to the fusion unit for splicing to obtain the joint embedding representation of each type of target.

[0017] Optionally, the scene category text embedding layer and the target category text embedding layer both use the text encoder in the CLIP model.

[0018] Optionally, the feature extraction module selects the image encoder in the CLIP model.

[0019] The present invention also provides a training method for an image recognition system, which comprises:

[0020] Obtaining labeled images as a training set, wherein the labels mark objects in the images;

[0021] The image recognition system described in any of the above items is trained using the data in the training set and the loss function.

[0022] Optionally, the Transformer decoding module in the image recognition system obtains a series of prior feature points from the output result of the Transformer encoding module and constructs a content query vector and a position query vector corresponding to each prior feature point, decodes the target category based on the content query vector, and decodes the target position based on the position query vector; for each prior feature point, its content query vector q c The trainable query vector q of the prior feature point c1 The fused semantic vector e is obtained by weighted summing of a preset number of target semantic vectors with the highest similarity scores to the prior feature point, and the weight of each target semantic vector is proportional to the feature similarity between the target and the prior feature point;

[0023] The training method includes grouping the content query vectors of all prior feature points and the loss function used includes inter-group contrast learning loss and intra-group contrast learning loss, wherein the content query vectors of all prior feature points with exactly the same candidate target category set are divided into the same query group, and the target category set of the prior feature point is a preset number of targets with the highest similarity score with the prior feature point; the inter-group contrast learning loss is used to suppress the similarity of query vectors between different query groups by using average cross entropy loss; the intra-group contrast learning loss is an intra-class clustering loss function with a minimum interval, and when the similarity of any two vectors in the query group is higher than the minimum interval threshold, the intra-group contrast learning loss will increase with the increase of similarity.

[0024] The present invention also provides an image recognition method, which comprises: inputting an image to be recognized into a trained image recognition system as described in any one of the above items to obtain a target in the input image.

[0025] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0026] The present invention also provides a computer program product, comprising a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the steps of any of the above methods are implemented.

[0027] In general, compared with the prior art, the above technical solutions conceived by the present invention mainly have the following beneficial effects:

[0028] 1. In the image recognition system proposed by the present invention, the coarse-grained scene perception module identifies the scene most relevant to the input image, and the fine-grained target perception module calculates the similarity score between each feature point and each target in the most relevant scene based on the information provided by the coarse-grained scene perception module. When the Transformer encoding module performs encoding, grouping and screening are performed based on the similarity score provided by the fine-grained target perception module. The higher the similarity score with the target, the more important the feature point is, and it is selected as a significant feature point. The stronger the correlation between significant feature points associated with the same target is, the more significant feature points with strong correlation are added to the same self-attention branch to form a multi-branch self-attention image encoding, focusing on encoding and calculating the added significant features, thereby reducing the computational complexity of the encoder, improving the feature extraction capability of key targets, reducing the interference between different categories of targets and image background areas, and accelerating image processing speed;

[0029] 2. Optionally, optimize the Transformer decoding module and add semantic prior information to its content query vector to improve the model's ability to perceive key targets and improve image recognition results;

[0030] 3. Optionally, during training, inter-group contrastive learning loss and intra-group contrastive learning loss are added, and group query is guided to detect specific category objects based on contrastive learning. The model's perception of key targets is further enhanced by optimizing the decoder, thereby accelerating the convergence of the decoder. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a structural block diagram of an image recognition system in one embodiment of the present invention;

[0032] Figure 2 It is a specific structural diagram of an image recognition system in one embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0034] like Figure 1 FIG. 1 is a block diagram of an image recognition system according to an embodiment of the present invention. Figure 2 The figure shows a specific structural diagram of an image recognition system in one embodiment of the present invention, which includes a feature extraction module, a coarse-grained scene perception module, a fine-grained target perception module, a Transformer encoding module and a Transformer decoding module.

[0035] The feature extraction module is used to extract the features of the input image and obtain the feature vector of each feature point.

[0036] Specifically, the feature extraction module can be an image encoder in CLIP, using a convolutional neural network such as ResNet to extract image features I = (i1, i2, ..., i N ), where N is the number of image feature points.

[0037] The coarse-grained scene perception module includes a scene semantics generation unit and a first similarity calculation unit. The scene semantics generation unit is used to generate scene semantic vectors of various scenes in the preset scene set. The first similarity calculation unit is used to pool all feature vectors to obtain a global feature i avg Then select avg The scene p with the highest similarity and its scene semantic vector s′ p .

[0038] The coarse-grained scene perception module uses the strong correlation between the global features of the image and the scene semantics to perform coarse-grained scene classification on the image, thereby improving the model's ability to detect key targets in application scenarios.

[0039] Specifically, a scene set is pre-set, and the scene set includes multiple types of scenes, such as road traffic scenes, sports event scenes, assembly scenes, etc. The scene semantics generation unit is used to generate scene semantic vectors of various scenes.

[0040] In one embodiment, the scene semantics generation unit may be a trainable multi-layer perceptron (MLP). For example, a scene category text embedding layer may be provided to convert the language description of each scene into a scene text embedding and then send it to the multi-layer perceptron to obtain the scene semantic vectors of each scene. The scene category text embedding layer may directly select the text encoder in the trained CLIP model. When training the image recognition system, the scene category text embedding layer may not need to be trained again.

[0041] For example, there are P scene categories in total, and the language description of each scene is represented as scene text , and input it into the text encoder in the CLIP model to obtain the scene text embeddings s1,s2,…,s corresponding to P scene categories P , and then embed the scene texts of P scene categories into the multi-layer perceptron to obtain the scene semantic vectors s′1, s′2, …, s′ corresponding to the P scene categories P , the specific calculation process is as follows:

[0042] (s1,s2,…,s P )=CLIP(scene text )#(1)

[0043] (s′1,s′2,…,s′ P )=σ(W2(σ(W1(s1,s2,…,s P )+b1))+b2)#(2)

[0044] Where W1 and W2 are the weight matrices of the multilayer perceptron, b1 and b2 are the bias vectors of the multilayer perceptron, and σ is the nonlinear activation function of the multilayer perceptron (such as ReLU).

[0045] The first similarity calculation unit is used to pool all feature vectors to obtain the global feature i avg Then select avg The scene p with the highest similarity and its scene semantic vector s′ p In the specific operation, all feature vectors output by the feature extraction module are first Average Pooling , get the image global feature representation i avg Then calculate the semantic vector of each scene and the global feature representation i avg The similarity of the image is calculated by avg and the scene category semantic vector (s′1,s′2,…,s′ P ) and select the cosine similarity between avg The scene p with the highest similarity enters the subsequent processing as the scene most relevant to the input image.

[0046] The fine-grained target perception module includes a fusion unit, a target semantic generation unit, and a second similarity calculation unit. The fusion unit is used to embed the target text of each target contained in the scene p into the vector s′. p The joint embedding representation of each type of target is obtained by splicing, the target semantic generation unit is used to map each joint embedding representation into a target semantic vector, and the second similarity calculation unit is used to obtain the similarity score between each feature point and each type of target by calculating the similarity of the vector.

[0047] Specifically, the fine-grained target semantic perception module utilizes the semantic information of potential key target categories in the most relevant scenes to perform fine-grained classification of image features and assists the encoder in realizing multi-branch attention encoding.

[0048] The target type corresponding to each scene is set in advance. For example, the target types under road traffic usually include cars, electric vehicles, pedestrians, traffic police, street lights, etc. After the coarse-grained scene perception module is used to identify the most relevant scene p, the fine-grained target perception module is used to calculate the similarity score between each feature point and each type of target.

[0049] You can first obtain the various types of targets under the most relevant scene p and get the target text embeddings of each type of target. Specifically, you can set a target category text embedding layer to convert the language descriptions of each type of target into target text embeddings and then send them to the fusion unit for splicing to obtain the joint embedding representation of each type of target. Among them, the target text embedding layer can directly use the text encoder in the trained CLIP model. When training the image recognition system, the scene category text embedding layer does not need to be trained again.

[0050] For example, there are M target categories under the most relevant scene p, and the language description of each target is represented as object text , and input it into the text encoder in the CLIP model to obtain the target text embeddings o1, o2, …, o corresponding to the M target categories. M , and then embed the target texts of the M target categories into the input fusion unit to obtain the joint embedding representation o′1, o′2,…, o′ corresponding to the M target categories M , the specific calculation process is as follows:

[0051] (o1,o2,…,o M )=CLIP(object text )#(3)

[0052] (o′1,o′2,…,o′ M )=([o1;s′ p ],[o2;s′ p ],…,[o M ;s′ p])# (4)

[0053] Among them, (o′1,o′2,…,o′ M ) represents the joint embedding representation of the target category in the most relevant scene, and M represents the number of target categories in this scene.

[0054] In one embodiment, the target semantic generation unit can also be a trainable multi-layer perceptron, which embeds the joint representations o′1, o′2, …, o′ corresponding to the M target categories. M Input the multi-layer perceptron to obtain the target semantic vectors e1, e2, …, e corresponding to M target categories. M , which can be specifically expressed as:

[0055] (e1,e2,…,e M )=σ(W4(σ(W3(o′1,o′2,…,o′ M )+b3))+b4)#(5)

[0056] Where W3 and W4 are weight matrices, and b3 and b4 are bias vectors.

[0057] The second similarity calculation unit is used to obtain the similarity score between each feature point and each type of target by calculating the similarity of the vector. In a specific embodiment, the similarity of the vector can be calculated by the following formula:

[0058]

[0059] In the formula, A nm Represents the similarity score between feature point n and target category m.

[0060] Through the above fine-grained target perception module, we can finally get the similarity matrix A N×M By calculating the similarity matrix, the degree of association between each image feature point and each target category can be quantified.

[0061] The Transformer encoding module is used to encode all feature vectors to obtain encoded features. The Transformer encoding module has a multi-branch self-attention calculation unit. The multi-branch attention calculation unit is used to search for feature points with similarity scores exceeding a threshold as significant feature points. For each significant feature point, a preset number of targets with the highest similarity scores are searched as candidate targets. All significant feature points corresponding to the same candidate target are added to the same self-attention branch, so that all self-attention branches perform self-attention calculations separately.

[0062] The Transformer encoding module usually includes a feature input layer, a self-attention calculation layer, a feedforward neural network FFN, a residual connection and a normalization layer. The present invention improves the self-attention calculation layer in the Transformer encoding module, that is, the self-attention calculation is guided by the similarity information output by the fine-grained target perception module, and multi-target category attention masks are generated by grouping and screening feature points, and multi-branch self-attention image encoding is performed, thereby reducing the computational complexity of the encoder, improving the feature extraction capability of key targets, and reducing interference between targets of different categories and image background areas.

[0063] Specifically, we first set the score threshold θ in advance, and then N×M For each feature point n, there are M similarity scores A corresponding to M targets. n1 ~A nM As long as there is a similarity score greater than the score threshold θ, the feature point is taken as a significant feature point, and the K targets with the highest similarity are selected to form a candidate target set containing K candidate target categories. Each significant feature point is associated with a candidate target set. All candidate targets are sorted out, and all significant feature points associated with the same candidate target are formed into a significant feature point set. Multiple candidate targets generate multiple significant feature point sets. Each set forms a corresponding self-attention mask map according to the significant feature points it contains, such as all significant feature points associated with candidate target k form a self-attention mask matrix map If the i-th feature point and the j-th feature point are both salient feature points of the candidate target k, the element in the i-th row and j-th column of the self-attention mask matrix is ​​1, otherwise it is 0. The corresponding self-attention branch is formed based on the self-attention mask matrix, that is, a self-attention branch can be constructed based on each candidate target, and different self-attention branches can be constructed based on different candidate targets. Each self-attention branch performs self-attention calculation independently. The self-attention calculation formula is as follows:

[0064]

[0065] In the formula, is the attention parameter in the self-attention branch corresponding to the target category k.

[0066] The Transformer decoding module is used to decode the encoded features and identify the targets in the input image.

[0067] In the above image recognition system, the coarse-grained scene perception module identifies the scene most relevant to the input image, and the fine-grained target perception module calculates the similarity score between each feature point and each target in the most relevant scene based on the information provided by the coarse-grained scene perception module. When the Transformer encoding module performs encoding, it performs grouping and screening based on the similarity score provided by the fine-grained target perception module. The higher the similarity score with the target, the more important the feature point is, and it is selected as a significant feature point. The stronger the correlation between significant feature points associated with the same target, the more strongly correlated significant feature points are added to the same self-attention branch to form a multi-branch self-attention image encoding, focusing on the encoding calculation of the added significant features, thereby reducing the computational complexity of the encoder, improving the feature extraction capability of key targets, reducing interference between different categories of targets and image background areas, and speeding up image processing.

[0068] The traditional Transformer decoding module usually only performs target queries based on the results of the Transformer encoding module. For example, after the Transformer encoding module outputs the encoding features of each feature point, the Transformer decoding module selects several feature points with the highest confidence as prior feature points based on the confidence of the encoding features, obtains their content prior and position prior, and constructs a content query vector q for each feature point. c and the position query vector q l , based on the content query vector q c Decode the target category and decode the target location q based on the location query vector l .

[0069] In one embodiment of the present invention, in order to further enhance the system's ability to perceive key targets, the Transformer decoding module is further improved. For each prior feature point, its content query vector q c The trainable query vector q of the prior feature point c1 The fused semantic vector e is the weighted sum of a preset number of target semantic vectors with the highest similarity scores to the encoding features of the prior feature point, and the weight of each target semantic vector is positively correlated to the similarity between the target semantic vector and the encoding features of the prior feature point.

[0070] Taking a priori feature point as an example, its encoding feature is recorded as i′. Based on the similarity score calculated by the fine-grained target perception module, the K candidate target categories with the highest similarity score with the priori feature point are selected, and the similarity between the encoding feature of the priori feature point and the target semantic vector of each candidate target category is continued to be calculated. The weight of the candidate target category is determined based on the similarity of each candidate target category, and the target semantic vectors of the K candidate target categories are weighted and summed to obtain the fused semantic vector e. The trainable query vector q c1 It is concatenated with the fused semantic vector e to obtain the content query vector q of the prior feature point c .

[0071] In specific operations, the weight of the target semantic vector of each candidate target category can be obtained by calculating the inner product of the prior feature point encoding feature and the target semantic vector of each candidate target category and normalizing it. The target semantic vector of the jth candidate target category is denoted as e j , whose weight is α j , the content query vector q of the prior feature point c The calculation formula is as follows:

[0072]

[0073] q c =[q c1 ;e]#(12)

[0074] Thus, by using the content query vector q c Introducing semantic priors can enhance the model's perception of key targets.

[0075] The present invention also provides a training method for an image recognition system, which is used to train the image recognition system introduced above, and the training process includes:

[0076] Obtain labeled images as training sets, where the labels mark the targets in the images; use the data in the training set and the loss function to train the image recognition system.

[0077] Specifically, the images in the training set are input into the system to be trained, and the system includes a feature extraction module, a coarse-grained scene perception module, a fine-grained target perception module, a Transformer encoding module and a Transformer decoding module; the coarse-grained scene perception module includes a scene semantic generation unit and a first similarity calculation unit, the fine-grained target perception module includes a fusion unit, a target semantic generation unit and a second similarity calculation unit, and the Transformer encoding module includes a multi-branch self-attention calculation unit;

[0078] The features of the input image are extracted through the feature extraction module to obtain the feature vector of each feature point;

[0079] The scene semantic generation unit generates scene semantic vectors of various scenes in the preset scene set, and the first similarity calculation unit pools all feature vectors to obtain the global feature i avg Then select avg The scene p with the highest similarity and its scene semantic vector s′ p ;

[0080] The target text of each target contained in the scene p is embedded into the vector s′ through the fusion unit p The joint embedding representations of various targets are obtained by concatenation, and each joint embedding representation is mapped into a target semantic vector by a target semantic generation unit. The similarity of each feature point and each target is obtained by calculating the similarity of the vector by a second similarity calculation unit;

[0081] All feature vectors are encoded through the Transformer encoding module to obtain encoded features, wherein feature points with similarity scores exceeding a threshold are searched as significant feature points through a multi-branch attention calculation unit. For each significant feature point, a preset number of targets with the highest similarity scores are searched as candidate targets. All significant feature points corresponding to the same candidate target are added to the same self-attention branch, so that all self-attention branches perform self-attention calculations respectively.

[0082] The encoded features are decoded through the Transformer decoding module to identify the target in the input image;

[0083] Obtain the identified targets and corresponding labels and calculate the loss function to train the system.

[0084] Among them, the loss function generally includes one or more of classification confidence loss, prediction box regression loss and intersection-over-union loss, which are used to calculate the deviation between the recognition result and the label.

[0085] In one embodiment, when the content query vector q of the Transformer decoding module c The trainable query vector q from the prior feature points c1 When the content query vectors of all prior feature points are grouped and concatenated with the fused semantic vector e, the loss function can be further optimized, the content query vectors of all prior feature points are grouped, and the loss function used adds inter-group contrast learning loss and intra-group contrast learning loss, wherein the content query vectors of all prior feature points with exactly the same set of candidate target categories are divided into the same query group, and the target category set of the prior feature points is a preset number of targets with the highest similarity scores with the prior feature points.

[0086] The inter-group contrastive learning loss is used to suppress the similarity of query vectors between different query groups using average cross entropy loss.

[0087] Specifically, the cosine similarity of queries between groups is calculated, and the average cross entropy loss is used to suppress the similarity of queries between groups, so that the queries between different groups have greater differences, which is conducive to querying and detecting targets of different categories.

[0088] Inter-group contrast learning loss Loss inter It can be expressed as:

[0089]

[0090] In the formula, Q represents the content query vectors divided into Q groups. is an indicator function, which is 0 when Q1 and Q2 are in the same group and 1 when they are in different groups. represents the i-th content query vector in Q1, represents the j-th content query vector in Q2.

[0091] Specifically, the intra-group contrastive learning loss uses an intra-class clustering loss function with a minimum margin, which increases when the similarity between any two vectors in the group is higher than the minimum margin threshold m, so that the model reduces the similarity of the vectors in the group and maintains the diversity of queries in the group.

[0092] Intra-group contrast learning loss Loss intra It can be expressed as:

[0093]

[0094] In this way, by improving the loss function, we can guide group queries to detect objects of specific categories based on contrastive learning, and further enhance the model's perception of key targets by optimizing the decoder, thereby accelerating the convergence of the decoder.

[0095] The intra-group contrastive learning loss is an intra-class clustering loss function with a minimum margin. When the similarity between any two vectors in the group is higher than the minimum margin threshold, the intra-group contrastive learning loss will increase with the increase of similarity.

[0096] The present invention also relates to an image recognition method, comprising: inputting an image to be recognized into the trained image recognition system introduced above to obtain a target in the input image.

[0097] Specifically, labeled images are obtained as training sets, and the labels mark the targets in the images; the image recognition system is trained using the data in the training set and the loss function.

[0098] Specifically, the image to be recognized is input into the trained system, and the system includes a feature extraction module, a coarse-grained scene perception module, a fine-grained target perception module, a Transformer encoding module and a Transformer decoding module; the coarse-grained scene perception module includes a scene semantic generation unit and a first similarity calculation unit, the fine-grained target perception module includes a fusion unit, a target semantic generation unit and a second similarity calculation unit, and the Transformer encoding module includes a multi-branch self-attention calculation unit;

[0099] The features of the input image are extracted through the feature extraction module to obtain the feature vector of each feature point;

[0100] The scene semantic generation unit generates scene semantic vectors of various scenes in the preset scene set, and the first similarity calculation unit pools all feature vectors to obtain the global feature i avg Then select avg The scene p with the highest similarity and its scene semantic vector s′ p ;

[0101] The target text of each target contained in the scene p is embedded into the vector s′ through the fusion unit p The joint embedding representations of various targets are obtained by concatenation, and each joint embedding representation is mapped into a target semantic vector by a target semantic generation unit. The similarity of each feature point and each target is obtained by calculating the similarity of the vector by a second similarity calculation unit;

[0102] All feature vectors are encoded through the Transformer encoding module to obtain encoded features, wherein feature points with similarity scores exceeding a threshold are searched as significant feature points through a multi-branch attention calculation unit. For each significant feature point, a preset number of targets with the highest similarity scores are searched as candidate targets. All significant feature points corresponding to the same candidate target are added to the same self-attention branch, so that all self-attention branches perform self-attention calculations respectively.

[0103] The encoded features are decoded through the Transformer decoding module to identify the objects in the input image.

[0104] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above method when executed by a processor.

[0105] Specifically, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, an internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0106] The embodiment of the present invention further provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of the method of the above embodiment of the present invention.

[0107] In general, the system mentioned in the present invention, its coarse-grained scene perception module uses the global features of the image to perform image classification on the coarse-grained scene category; its fine-grained target perception module extracts the semantic information of the candidate target category in a specific scene and performs a rough detection on the image; the multi-branch self-attention calculation unit in its Transformer encoding module guides the image encoder to perform multi-branch attention encoding based on semantic information on the fine-grained key target category; combining the above three modules, it is possible to reduce the computational complexity of the encoder, improve the feature extraction capability of the key target, reduce the interference of different categories of targets and image background areas, and speed up the image processing speed. On this basis, the Transformer decoding module is further optimized, and semantic prior information is added to its content query vector to improve the model's ability to perceive key targets and improve the image recognition effect. Accordingly, during training, the inter-group contrast learning loss and the intra-group contrast learning loss are increased, and the group query is guided to detect specific category objects based on contrast learning, and the model's perception of key targets is further enhanced by optimizing the decoder, thereby accelerating the convergence speed of the decoder.

[0108] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. It should be noted that "in one embodiment", "for example", "for example", etc. of the present invention are intended to illustrate the present invention, rather than to limit the present invention.

[0109] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention.

Claims

1. An image recognition system, characterized in that: include: The feature extraction module is used to extract the features of the input image and obtain the feature vector of each feature point; The coarse-grained scene perception module includes a scene semantics generation unit and a first similarity calculation unit, wherein the scene semantics generation unit is used to generate scene semantic vectors of various scenes in a preset scene set, and the first similarity calculation unit is used to pool all feature vectors to obtain a global feature i avg Then select avg The scene p with the highest similarity and its scene semantic vector s ′ p ; The fine-grained target perception module includes a fusion unit, a target semantic generation unit and a second similarity calculation unit. The fusion unit is used to embed the target text of each type of target contained in the scene p into the vector s. ′ p The joint embedding representation of each type of target is obtained by splicing, the target semantic generation unit is used to map each of the joint embedding representations into a target semantic vector, and the second similarity calculation unit is used to obtain a similarity score between each feature point and each type of target by calculating the similarity of the vector; A Transformer encoding module is used to encode all feature vectors to obtain encoding features. The Transformer encoding module has a multi-branch self-attention calculation unit. The multi-branch attention calculation unit is used to search for feature points with similarity scores exceeding a threshold as significant feature points. For each significant feature point, a preset number of targets with the highest similarity scores are searched as candidate targets. All significant feature points corresponding to the same candidate target are added to the same self-attention branch, so that all self-attention branches perform self-attention calculations respectively. The Transformer decoding module is used to decode the encoded features and identify the target in the input image.

2. The image recognition system according to claim 1, characterized in that: The Transformer decoding module obtains a series of prior feature points from the output result of the Transformer encoding module and constructs a content query vector and a position query vector corresponding to each prior feature point, decodes the target category based on the content query vector, and decodes the target position based on the position query vector; For each prior feature point, its content query vector q c The trainable query vector q of the prior feature point c1 The fused semantic vector e is obtained by weighted summing up a preset number of target semantic vectors with the highest similarity scores to the encoding features of the prior feature point, and the weight of each target semantic vector is positively correlated with the similarity between the target semantic vector and the encoding features of the prior feature point.

3. The image recognition system according to claim 1, wherein: The scene semantics generation unit and the target semantics generation unit are both trainable multi-layer perceptrons; The coarse-grained scene perception module also includes a scene category text embedding layer, which is used to convert the language description of each scene into a scene text embedding and then send it to the scene semantic generation unit to obtain the scene semantic vector of each scene; The fine-grained target perception module also includes a target category text embedding layer, which is used to convert the language description of each type of target into a target text embedding and then send it to the fusion unit for splicing to obtain the joint embedding representation of each type of target.

4. The image recognition system according to claim 3, characterized in that: The scene category text embedding layer and the target category text embedding layer both use the text encoder in the CLIP model.

5. The image recognition system according to claim 1, characterized in that: The feature extraction module uses the image encoder in the CLIP model.

6. A training method for an image recognition system, characterized in that: include: Obtaining labeled images as a training set, wherein the labels mark objects in the images; The image recognition system according to any one of claims 1 to 5 is trained using the data in the training set and the loss function.

7. The training method according to claim 6, characterized in that: The Transformer decoding module in the image recognition system obtains a series of prior feature points from the output result of the Transformer encoding module and constructs a content query vector and a position query vector corresponding to each prior feature point, decodes the target category based on the content query vector, and decodes the target position based on the position query vector; For each prior feature point, its content query vector q c The trainable query vector q of the prior feature point c1 The fused semantic vector e is obtained by weighted summing of a preset number of target semantic vectors with the highest similarity scores to the prior feature point, and the weight of each target semantic vector is proportional to the feature similarity between the target and the prior feature point; The training method includes grouping the content query vectors of all prior feature points and the loss function used includes inter-group contrast learning loss and intra-group contrast learning loss, wherein the content query vectors of all prior feature points with exactly the same candidate target category set are divided into the same query group, and the target category set of the prior feature point is a preset number of targets with the highest similarity score with the prior feature point; the inter-group contrast learning loss is used to suppress the similarity of query vectors between different query groups by using average cross entropy loss; the intra-group contrast learning loss is an intra-class clustering loss function with a minimum interval, and when the similarity of any two vectors in the query group is higher than the minimum interval threshold, the intra-group contrast learning loss will increase with the increase of similarity.

8. An image recognition method, characterized in that: include: The image to be recognized is input into the trained image recognition system according to any one of claims 1 to 5 to obtain the target in the input image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 6 to 8 are implemented.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 6 to 8 are implemented.

Images