Image recognition method and device, computer equipment and readable storage medium

Abstract

The invention relates to an image recognition method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring image data of a target object, and performing feature extraction on the image data based on a visual self-attention model to obtain an original feature map of the target object; key semantic features and key position features of the target object in the original feature map are extracted based on the task prompt template, and classified lexical elements of the target object are obtained; and performing identification processing on the classification lexical elements according to an image identification model to obtain an action identification result of the target object. By adopting the method, the accuracy of the image recognition method can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image recognition method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] In the field of image processing technology, image action recognition is an important component. By using image recognition methods to process image data, the behavior and posture of the target object can be identified to obtain action recognition results. The action recognition results are widely used in smart transportation, smart office, industrial generation and other fields to provide early warnings for illegal and dangerous actions.

[0003] In the current image recognition methods, the key point data of the target object in the image data is extracted through a posture estimation model represented by ViT Pose (vision transformer for generic body pose estimation, a general posture estimation model), and the skeletal key point position features of the target object are obtained. Then, based on the classification model (for example, KNN (K-Nearest-Neighbors, K nearest neighbor algorithm) model or ANN (Artificial Neural Network, artificial neural network) model), the action recognition is performed on the skeletal key point position features to obtain the action recognition result.

[0004] However, current image recognition methods only determine the action recognition results of the target object based on the position features of skeletal key points. The influencing factors are single and there is often a problem of blurred boundaries in subsequent posture classification, which makes the image recognition method less accurate. Summary of the invention

[0005] Based on this, it is necessary to provide an image recognition method, apparatus, computer device, computer-readable storage medium and computer program product to address the above technical problems.

[0006] In a first aspect, the present application provides an image recognition method, comprising:

[0007] Acquire image data of a target object, and perform feature extraction on the image data based on a visual self-attention model to obtain an original feature map of the target object;

[0008] Extracting key semantic features and key position features of the target object in the original feature graph based on the task prompt template to obtain the classification word element of the target object;

[0009] The classification word is recognized and processed according to the image recognition model to obtain the action recognition result of the target object.

[0010] In one embodiment, extracting key semantic features and key position features of the target object in the original feature graph based on the task prompt template to obtain the classification word element of the target object includes:

[0011] The task prompt template and the original feature map are weighted to obtain key semantic features and key position features in the original feature map; the task prompt template is a description framework corresponding to the format of the original feature map; the key semantic features and the key position features represent the description answer corresponding to the description framework;

[0012] The key semantic features and key position features are arranged to obtain classification word elements of the target object; the classification word elements represent a complete target description composed of the description framework and the description answer.

[0013] In one embodiment, before extracting the key semantic features and key position features of the target object in the original feature map based on the task prompt template to obtain the classification word element of the target object, the method further includes:

[0014] Obtaining a target vector and initializing an initial feature matrix; the target vector represents a description framework of the image data; the target vector is obtained by training the original vector based on a sample image set corresponding to the image data;

[0015] Constructing a feature matrix based on the target vector and the initial feature matrix;

[0016] The feature matrix is ​​transformed according to an inverse Fourier algorithm, and a task prompt template is determined based on the transformed feature matrix.

[0017] In one embodiment, constructing a feature matrix based on the target vector and the initial feature matrix includes:

[0018] Generate a long natural sequence corresponding to the initial feature matrix, and randomly sort the long natural sequence to obtain a target long natural sequence;

[0019] Determining a position sequence based on the dimension of the target vector and the target long natural sequence;

[0020] Based on the position sequence, the target elements in the target vector are added one by one to the initial feature matrix to obtain a feature matrix.

[0021] In one embodiment, before extracting the key semantic features and key position features of the target object in the original feature map based on the task prompt template to obtain the classification word element of the target object, the method further includes:

[0022] Acquire a sample image data set, and perform feature extraction on the sample image data set in batches according to the visual self-attention model to obtain a sample original feature atlas;

[0023] Initialize the original vector based on the Kaiming uniform algorithm, and initialize the initial feature matrix according to the size of the sample original feature map in the sample original feature map set;

[0024] An initial task prompt template is constructed according to the original vector and the initial feature matrix, and the initial task prompt template and a preset initial image recognition model are trained based on the sample original feature atlas to obtain a task prompt template and an image recognition model.

[0025] In one embodiment, the image recognition model is a multi-layer perceptron model, and the recognition processing of the classification word element according to the image recognition model to obtain the action recognition result of the target object includes:

[0026] Inputting the classification word into the multi-layer perceptron model, and performing recognition processing on the complete target description corresponding to the classification word by the multi-layer perceptron model to obtain an initial output result;

[0027] Performing mapping processing on the initial output result based on a normalization algorithm to obtain an output result;

[0028] The action recognition result of the target object is determined according to the output result.

[0029] In a second aspect, the present application also provides an image recognition device, comprising:

[0030] A first extraction module is used to obtain image data of a target object, and perform feature extraction on the image data based on a visual self-attention model to obtain an original feature map of the target object;

[0031] A second extraction module is used to extract key semantic features and key position features of the target object in the original feature map based on the task prompt template to obtain the classification word element of the target object;

[0032] The processing module is used to perform recognition processing on the classification word element according to the image recognition model to obtain the action recognition result of the target object.

[0033] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0034] Acquire image data of a target object, and perform feature extraction on the image data based on a visual self-attention model to obtain an original feature map of the target object;

[0035] Extracting key semantic features and key position features of the target object in the original feature graph based on the task prompt template to obtain the classification word element of the target object;

[0036] The classification word is recognized and processed according to the image recognition model to obtain the action recognition result of the target object.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0038] Acquire image data of a target object, and perform feature extraction on the image data based on a visual self-attention model to obtain an original feature map of the target object;

[0039] Extracting key semantic features and key position features of the target object in the original feature graph based on the task prompt template to obtain the classification word element of the target object;

[0040] The classification word is recognized and processed according to the image recognition model to obtain the action recognition result of the target object.

[0041] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0042] Acquire image data of a target object, and perform feature extraction on the image data based on a visual self-attention model to obtain an original feature map of the target object;

[0043] Extracting key semantic features and key position features of the target object in the original feature graph based on the task prompt template to obtain the classification word element of the target object;

[0044] The classification word is recognized and processed according to the image recognition model to obtain the action recognition result of the target object.

[0045] The above-mentioned image recognition method, apparatus, computer equipment, computer-readable storage medium and computer program product obtain image data of the target object, and perform feature extraction on the image data based on the visual self-attention model to obtain the original feature map of the target object; extract the key semantic features and key position features of the target object in the original feature map based on the task prompt template to obtain the classification word element of the target object; perform recognition processing on the classification word element according to the image recognition model to obtain the action recognition result of the target object. With this method, the key semantic features and key position features of the target object are extracted through the visual self-attention model and the task prompt template to obtain the classification word element, and the action recognition result is determined by the key semantic features and key position features in the classification word element, which integrates the influencing factors of the two, improves the accuracy of the action recognition result, and thus improves the accuracy of the image recognition method. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 is a flowchart of an image recognition method in one embodiment;

[0048] Figure 2 A schematic diagram of a process for determining a classification word unit in one embodiment;

[0049] Figure 3 A schematic diagram of a process of constructing a task prompt template based on a target vector in one embodiment;

[0050] Figure 4 A schematic diagram of a process for determining a feature matrix in one embodiment;

[0051] Figure 5 A schematic diagram of a process for training an initial task prompt template and an initial image recognition model in one embodiment;

[0052] Figure 6 A schematic diagram of a process for determining an original feature map in one embodiment;

[0053] Figure 7 is initial image data of a target object in an exemplary embodiment;

[0054] Figure 8 A schematic diagram of a process for determining an action recognition result in one embodiment;

[0055] Fig. 9 A schematic diagram of a model of an image recognition method in an embodiment;

[0056] Fig.10 A schematic diagram of a process for determining a classification word unit in an exemplary embodiment;

[0057] Fig.11 is a structural block diagram of an image recognition device in one embodiment;

[0058] Fig.12 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] Action recognition, as a classic task in computer vision, aims to accurately identify the behavior or action of the target person in an image or video. Action recognition tasks are widely used in smart transportation, smart office, industrial production and other fields to provide early warnings for illegal and dangerous actions (such as smoking recognition, phone call recognition, and falling down recognition) to eliminate safety hazards.

[0061] Traditional image recognition methods are mainly divided into three types: (posture) recognition based on skeletal key points, image classification recognition, and video classification recognition. Recognition based on skeletal key points first requires identifying the skeletal joints of the target object in the image data, and then completing the posture classification based on the skeletal joint connection information. Image classification recognition directly uses deep neural networks to learn action features and complete classification judgments. Video classification recognition is additionally aimed at complex requirements such as multi-person scenes and incorporating time series information, such as fight recognition.

[0062] Among them, recognition based on skeleton key points only focuses on the changes in the target object's posture, and often ignores important information in the image, such as the pen in writing recognition, the mobile phone in mobile phone recognition, and the relative position of the target object and other objects in such images, resulting in limited accuracy in behavioral tasks. The image classification and recognition method alone needs to face the redundant feature interference of clothing, facial features, etc., and lacks the accurate capture of position association.

[0063] In addition, before applying the posture estimation model and classification model, it is necessary to supervisely train the basic visual model and classification model according to the sample image data set corresponding to the image data, so that the posture estimation model and classification model meet the preset training completion conditions. In actual scenarios, the update of sample data requires continuous iteration of the parameters of the basic visual model and classification model. Since the number of parameters of the basic visual model and classification model that need to be adjusted in each round of iteration is large, a large amount of resources (for example, computing resources, storage resources) will be consumed during the training process of the posture estimation model and classification model. That is, using the pre-trained basic visual model to fine-tune the full model for a specific task will inevitably cause a waste of the basic visual model's capabilities and a loss of computing and storage resources.

[0064] In summary, the current image recognition method only determines the action recognition result of the target object based on the position feature, which has a single influencing factor, resulting in low accuracy of the current image recognition method. In addition, the task of outputting the skeleton key points alone causes a lot of information loss and resource challenges.

[0065] Based on the above-mentioned prior art, the present application provides an image recognition method, which extracts the key semantic features and key position features of the target object through a visual self-attention model and a task prompt template to obtain a classification word, and determines the action recognition result through the key semantic features and key position features in the classification word, integrating the influencing factors of the two, improving the accuracy of the action recognition result, and thus improving the accuracy of the image recognition method. At the same time, by only fine-tuning the task prompt template and the image recognition model parameters, the utility of efficient parameter adjustment is achieved.

[0066] In one embodiment, Figure 1 As shown, an image recognition method is provided. The embodiment of the present application takes the method applied to a computer device as an example for explanation. The embodiment of the present application does not limit the execution device of the image recognition method, and includes the following steps 102 to 106:

[0067] Step 102, obtain image data of the target object, and extract features from the image data based on the visual self-attention model to obtain an original feature map of the target object.

[0068] In implementation, a computer device obtains image data of a target object, and then inputs the image data into a visual self-attention model, performs feature extraction on the image data through the visual self-attention model, and obtains an original feature map of the target object.

[0069] Specifically, the visual self-attention model is a pre-trained visual self-attention model, that is, a visual self-attention model that has been trained. The user inputs the image data of the target object into the computer device. The computer device obtains the image data of the target object to be identified. Then, the computer device inputs the image data into the pre-trained visual self-attention model, extracts the initial feature information of the target object through the visual self-attention model, and obtains the original feature map.

[0070] In an optional embodiment, the image data is a frame of image captured from video data. The embodiment of the present application does not limit the source of the image data.

[0071] Optionally, the target object is generally a human being, but may also be other types of organisms. The embodiments of the present application do not limit the target object.

[0072] Step 104 , extracting key semantic features and key position features of the target object in the original feature graph based on the task prompt template to obtain the classification word element of the target object.

[0073] In the implementation, the computer device obtains the target vector and constructs a task prompt template based on the target vector. Then, the computer device performs point multiplication and sum processing on the original feature map according to the task prompt template to extract the key semantic features and key position features of the target object and obtain the classification word element of the target object.

[0074] Specifically, the target vector is a description frame of the image data. The computer device obtains the target vector and initializes an initial feature matrix based on the format of the original feature map. Then, the computer device constructs a task prompt template according to the target vector and the initial feature matrix. The computer device performs a dot multiplication and a summation process on the task prompt template and the original feature map to obtain a classification word element of the target object. The classification word element fuses the description frame and the description answer corresponding to the description frame.

[0075] Step 106, classifying word units are recognized and processed according to the image recognition model to obtain the action recognition result of the target object.

[0076] In the implementation, the computer device inputs the classification word into the image recognition model, and performs recognition processing on the classification word through the image recognition model to obtain an initial input result. Then, the computer device performs mapping processing on the initial output result to obtain an output result, and determines the action recognition result of the target object based on the output result.

[0077] In the above-mentioned image recognition method, the key semantic features and key position features of the target object are extracted through the visual self-attention model and the task prompt template to obtain the classification word unit, and the action recognition result is determined by the key semantic features and key position features in the classification word unit, which integrates the influencing factors of the two and improves the accuracy of the action recognition result, thereby improving the accuracy of the image recognition method.

[0078] In an exemplary embodiment, if Figure 2 As shown, the specific processing process of step 104 includes steps 202 to 204. Among them:

[0079] Step 202: weighting the task prompt template and the original feature map to obtain key semantic features and key position features in the original feature map.

[0080] Among them, the task prompt template is a description framework corresponding to the format of the original feature map; the key semantic features and key position features represent the description answer corresponding to the description framework.

[0081] In the implementation, the computer device performs weighted processing on the original feature map based on the task prompt template to obtain key semantic features and key position features in the original feature map. The weighted processing process of the original feature map by the computer device essentially determines the key semantic features and key position features from the initial semantic features and initial position features in the original feature map based on the description framework in the task prompt template, thereby obtaining a description answer corresponding to the description framework.

[0082] Step 204 , perform feature arrangement on key semantic features and key position features to obtain classification word elements of the target object.

[0083] Among them, the classification word meta-representation describes the complete target description composed of the framework and the description answer.

[0084] During implementation, the computer device performs Einstein summation processing on the key semantic features and the key position features, completes the feature arrangement of the key semantic features and the key position features, and obtains the classification word element of the target object.

[0085] Specifically, the computer device performs a dot product process and a sum process on the original feature map and the task prompt template to obtain the classification word element of the target object. The dot product process and the sum process of the original feature map and the task prompt template are shown in the following formula (1):

[0086] (1)

[0087] In the above formula (1), is a classifier, and , where D is the dimension of the original feature map. R represents a set of real numbers. represents the Einstein summation, Represents the last two dimensions of the two matrices, performs dot multiplication and sum operations, X is the original feature map, and M is the task prompt template. represents the dimension of the original feature map, Represents the dimension of the task prompt template. By performing dot multiplication and summation on the original feature map through the task prompt template, the key position features and key semantic features in the original feature map can be extracted, and the key semantic features and key position features can be fused to obtain the classification word.

[0088] In an exemplary embodiment, the image data of the target object represents that a man (target object) is playing tennis. The initial position features and initial semantic features contained in the original feature map represent the positions of the target object's feet and hands, the texture features of the racket, the color and texture features of the clothing, etc. The description framework represented by the task prompt template is 1. Are the target object's hands in a clenched state? 2. Is there a tennis racket in the target object's hand? 3. Are the target object's feet separated? The computer device performs point multiplication and summation processing on the task prompt template and the original feature map to obtain a classification word. In essence, the key position features and key semantic features are determined in the original feature map based on the description framework, so as to determine the description answer according to the key position features and key semantic features, and determine the complete target description according to the description answer and the description framework. The complete target description is that the target object's feet are separated, the target object's hands are in a clenched state, and the target object has a tennis racket in his hand.

[0089] In this embodiment, the original feature map is feature extracted through the task prompt template to obtain the key position features and key semantic features of the target object, clarify the complete target description of the target object in the image data, facilitate the subsequent determination of the action recognition result based on the complete target description, and improve the action recognition efficiency and action recognition accuracy.

[0090] In an exemplary embodiment, before determining the classification word element corresponding to the target object, it is necessary to construct a task prompt template. Figure 3 As shown, before step 104 is executed, the specific processing process of the image recognition method also includes steps 302 to 306. Among them:

[0091] Step 302, obtain the target vector and initialize the initial feature matrix.

[0092] The target vector represents the description framework of the image data; the target vector represents the description framework corresponding to the target action type performed by the target object, and the target action type is the action type that needs to be identified. The target vector is obtained by training the original vector based on the sample image set corresponding to the image data.

[0093] In implementation, the computer device obtains a target vector. Then, the computer device initializes an initial feature matrix according to the format of the original feature map.

[0094] Specifically, the vector dimension, height, and width of the original feature map are D, H, and W respectively. The computer device obtains a target vector, which is a target one-dimensional vector, that is, a 1×n-dimensional vector P. The computer device generates a matrix of all zeros according to H and W, and obtains an initial feature matrix with a dimension of ×H×W.

[0095] In an exemplary embodiment, the terminal obtains a sample image set corresponding to the image data. The corresponding relationship is that the sample object in the sample image data in the sample image data set and the target object in the image data are of the same type. The sample image set contains each sample image data. The sample object and the target object in the sample image data are of the same biological type. An initial image recognition model is preset in the terminal, and the initial image recognition model is an initial multi-layer perceptron model. The terminal initializes a 1×n-dimensional vector to obtain an original vector. The terminal trains the preset initial image recognition model and the original vector based on the sample image set until the trained initial image recognition model and the original vector meet the preset training completion conditions. The terminal determines the trained initial image recognition model as the image recognition model and determines the original vector as the target vector. Among them, the binary cross-entropy loss (BCE) can be used to guide the module learning during the training process. Then, the terminal transfers the visual self-attention model and the image recognition model to the computer device, and sends the target vector to the computer device. The computer device receives the target vector and introduces the visual self-attention model and the image recognition model, so that the computer device executes the image recognition method according to the visual self-attention model, the target vector, and the image recognition model. Among them, the target vector represents the description framework of the image data. For example, if the target image represents a human playing tennis, the target vector represents the description framework when a human executes the action type of playing tennis. This description framework contains n description questions (n is the dimension of the target vector).

[0096] Optionally, n is generally set to 20, and n < H×W. H and W are the height and width of the original feature map respectively, and n is set according to the recognition requirements. If more accurate recognition is required, the larger n is, the better. If faster processing efficiency is required, the smaller n is set, the better. The embodiments of the present application do not limit n.

[0097] Optionally, the loss threshold and the accuracy threshold are determined according to the recognition requirements. The embodiments of the present application do not limit the loss threshold and the accuracy threshold, and the embodiments of the present application do not limit the training completion conditions either.

[0098] Step 304, construct a feature matrix based on the target vector and the initial feature matrix.

[0099] In implementation, the computer device generates a position sequence according to the dimension of the target vector and the dimension of the initial feature matrix, and adds the target vector to the initial feature matrix based on the position sequence to obtain the feature matrix.

[0100] Specifically, the computer device generates a target long natural sequence corresponding to the initial feature matrix based on the dimension of the initial feature matrix. Then, the computer device determines a position sequence in the target long natural sequence according to the dimension of the target vector. The computer device adds the target vector to the initial feature matrix based on the position sequence to obtain a feature matrix.

[0101] Step 306 , transform the feature matrix according to the inverse Fourier algorithm, and determine the task prompt template based on the transformed feature matrix.

[0102] In the implementation, an inverse Fourier algorithm is pre-set in the computer device. The computer device transforms the feature matrix according to the inverse Fourier algorithm to obtain a transformed feature matrix. Then, the computer device takes the real part of the transformed feature matrix to construct a task prompt template. Among them, the inverse Fourier algorithm is a standard inverse Fourier algorithm, that is, a fast inverse Fourier transform.

[0103] Specifically, the feature matrix is ​​M, and the inverse Fourier algorithm is the inverse Fourier function. The computer device inputs the feature matrix into the inverse Fourier function, performs an inverse Fourier transform on the feature matrix through the inverse Fourier function, and obtains the transformed feature matrix. Then, the computer device takes the transformed feature matrix to construct a task prompt template. The process of determining the task prompt template based on the feature matrix is ​​shown in the following formula (2):

[0104] (2)

[0105] In the above formula (2), is the feature matrix, is the task prompt template, also called the coefficient matrix. represents the real part of the element. represents the inverse Fourier function.

[0106] In this embodiment, by determining the target vector and the image recognition model, the learning parameters can be effectively reduced, the expected goals can be achieved in a short period of time, and efficient parameter adjustment can be completed. In addition, by determining different target vectors and building different task prompt templates based on different target vectors, multiple recognition tasks can be supported at the same time, and multiple recognition tasks can be plugged in and merged with each other, and the training process and reasoning process of multiple recognition tasks do not interfere with each other.

[0107] In an exemplary embodiment, Figure 4As shown, the specific processing process of step 304 includes steps 402 to 406. Among them:

[0108] Step 402: Generate a long natural sequence corresponding to the initial feature matrix, and randomly sort the long natural sequence to obtain a target long natural sequence.

[0109] In implementation, the computer device generates a long natural sequence corresponding to the initial feature matrix according to the dimension of the initial feature matrix, and then the computer device randomly sorts the long natural sequence to obtain a target long natural sequence.

[0110] Specifically, the dimension of the initial feature matrix is ​​H×W. The computer device generates an H×W long natural sequence and randomly sorts the long natural sequence to obtain a target long natural sequence. In particular, the random seed setting of the randomizer is fixed.

[0111] Step 404, determining a position sequence based on the dimension of the target vector and the target long natural sequence.

[0112] In implementation, the computer device determines a sequence of positions in the target long natural sequence according to the dimension of the target vector.

[0113] Specifically, the target vector is a 1×n dimensional vector P. The computer device takes the first n bits in the target long natural sequence to construct a position sequence S [ s 0 , s 1 , s 2 … s n-1 ] .

[0114] Step 406: based on the position sequence, add the target elements in the target vector to the initial feature matrix one by one to obtain the feature matrix.

[0115] In implementation, the computer device determines the position of the target element in the target vector in the initial feature matrix according to each position element in the position sequence, and adds the target element in the target vector to the initial feature matrix according to the position to obtain the feature matrix.

[0116] Specifically, the computer device generates a sequence of positions S[ s 0 , s 1 , s 2 … s n-1 ] , and sequentially replace the elements in the target vector P Place it into the initial feature matrix M to obtain the feature matrix. Specifically, the placement process is shown in the following formula (3):

[0117] (3)

[0118] In the above formula (3), is the i-th position element in the position sequence S. is the i-th target element among the target elements, is the position in the initial feature matrix. is the initial feature matrix The wide dimension.

[0119] In this embodiment, by generating a position sequence, the position of the target vector in the initial feature matrix is ​​clarified, and based on the position sequence, the target vector is added to the initial feature matrix to obtain the feature matrix, thereby completing the format conversion of the target vector. The obtained feature matrix contains the feature information of the target vector and conforms to the format of the original feature map.

[0120] In an exemplary embodiment, before determining the classification word element corresponding to the target object, it is necessary to construct a task prompt template. Figure 5 As shown, before step 104 is executed, the specific processing process of the image recognition method also includes steps 502 to 506. Among them:

[0121] Step 502, obtain a sample image dataset, and perform batch feature extraction on the sample image dataset according to the visual self-attention model to obtain a sample original feature atlas.

[0122] There is a corresponding relationship between the sample image data set and the image data. The corresponding relationship is that the sample object in the sample image data in the sample image data set and the target object in the image data are of the same type. The sample image set contains each sample image data. The sample object in the sample image data and the target object are of the same biological type. For example, if the target object is a man, the sample object in the sample image data is a human.

[0123] In the implementation, a pre-trained visual self-attention model is pre-set in the computer device. The computer device obtains a sample image data set. The computer device divides the sample image data set according to a preset batch number to obtain each batch of sample image data. The computer device inputs each batch of sample image data in each batch of sample image data into the visual self-attention model, and extracts features of the batch of sample image data through the visual self-attention model to obtain multiple sample original feature maps. Then. The computer device constructs a sample original feature map set according to each sample original feature map.

[0124] Specifically, the computer device obtains an initial sample image data set. The computer device annotates each initial sample image data in the initial sample image data set according to the action type of the sample object in the initial sample image data to obtain the sample image data. Specifically, if the action type of the sample object in the initial sample image data is the same as the target action type of the target object, the computer device annotates the initial sample image data as 1, and determines the initial sample image data as positive sample image data. If the action type of the sample object in the initial sample image data is not the same as the target action type of the target object, the computer device annotates the initial sample image data as 0, and determines the initial sample image data as negative sample image data. The computer device constructs a sample image data set based on each positive sample image data and each negative sample image data.

[0125] Step 504, initializing the original vector based on the Kaiming uniform algorithm, and initializing the initial feature matrix according to the size of the sample original feature map in the sample original feature map set.

[0126] In the implementation, a Kaiming uniform algorithm is pre-set in the computer device. The computer device initializes a 1×n dimensional vector based on the Kaiming uniform algorithm to obtain an original vector. The computer device initializes an initial feature matrix according to the size (dimension) of the sample original feature map in the sample original feature map. The specific process of initializing the initial feature matrix by the computer device has been described in the above step 302, and the embodiment of the present application will not be repeated here.

[0127] Step 506, constructing an initial task prompt template according to the original vector and the initial feature matrix, and training the initial task prompt template and a preset initial image recognition model based on the sample original feature atlas to obtain a task prompt template and an image recognition model.

[0128] The initial image recognition model is an initial multilayer perceptron (MLP) model. The training completion condition is that the loss value is less than a preset loss threshold.

[0129] In implementation, an initial image recognition model is pre-set in the computer device. The computer device constructs an initial task prompt template according to the original vector and the initial feature matrix. Among them, the construction process of the initial task prompt template is the same as the construction process of the feature matrix in the above step 302 or steps 402 to 406. Therefore, the construction process of the initial task prompt template is not repeated here in the embodiment of the present application. The computer device extracts features from the sample original feature map through the initial task prompt template for all original feature maps in the sample original feature map set to obtain sample classification word units. Then, the computer device performs recognition processing on the sample classification word units based on the preset initial image recognition model, calculates the recognition loss value, and determines whether the recognition loss value meets the preset training completion condition. If each sample recognition result meets the training completion condition, the computer device determines the initial task prompt template as the task prompt template, and determines the initial image recognition model as the image recognition model. If each sample recognition result does not meet the training completion condition, the computer device updates the elements in the initial task prompt template and the parameters of the initial image recognition model based on the back propagation algorithm. And for each sample original feature map in the sample original feature map set, feature extraction is performed on the sample original feature map through the initial task prompt template to obtain the sample classification word unit step until each sample recognition result meets the training completion condition.

[0130] Among them, when the initial task prompt template needs to be updated, the initial task prompt template can be used as a module in the initial image recognition model, so that the initial task prompt template and the initial image recognition model can be updated simultaneously according to the back propagation algorithm, and the consistency of the initial task prompt template and the initial image recognition model can be maintained.

[0131] Optionally, the training completion condition may be, but is not limited to, the recognition accuracy reaching an accuracy threshold or the loss value being less than a loss threshold, which is determined based on training requirements. The embodiment of the present application does not limit the training completion condition.

[0132] Optionally, the accuracy threshold and the loss threshold are determined according to training requirements, and the embodiments of the present application do not limit the accuracy threshold and the loss threshold.

[0133] In this embodiment, the initial task prompt template and the initial image recognition model are trained by a sample image data set corresponding to the image data. Compared with training two models, the basic visual model and the classification model in the prior art, the amount of training parameters is reduced, thereby reducing the amount of consumed resources, avoiding resource waste, and improving resource utilization.

[0134] In an exemplary embodiment, Figure 6 As shown, the specific processing process of step 102 includes steps 602 to 604. Among them:

[0135] Step 602: Acquire initial image data of the target object, and perform format processing on the initial image data according to a preset image format to obtain image data of the target object.

[0136] In implementation, an image format is pre-set in a computer device. The image format corresponds to the image requirements of the visual self-attention model. In addition, the image format requires that the image data contain complete image information of the target object. The computer device obtains initial image data of the target object. Then, the computer device processes the image data according to the preset image format to obtain image data of the target object.

[0137] In an exemplary embodiment, a user transmits the initial image data of the target object to a computer device through a user terminal, and the computer device acquires the initial image data of the target object. Figure 7 is the initial image data of the target object in an exemplary embodiment. The target object is a man playing tennis in the image data. The length of the target data pre-framed in the computer device is 324px and the width is 126px. The computer device performs the operation according to the preset target data (such as Figure 7 The initial image data is cropped (as shown in the box in ), and then the cropped image is converted according to a preset image format to obtain image data. The image data contains the complete image information of the target object, but other interfering image information is removed, so that the image data meets the input requirements of the visual self-attention model.

[0138] Step 604: input the image data into the visual self-attention model, extract the initial feature information of the target object through the visual self-attention model, and obtain the original feature map of the target object.

[0139] The initial feature information includes initial semantic features and initial position features.

[0140] In implementation, a computer device inputs image data into a visual self-attention model, and extracts initial feature information of a target object in the image data through the visual self-attention model, thereby obtaining an original feature map of the target object.

[0141] Specifically, the visual self-attention model is a pre-trained visual self-attention model. The computer device inputs the image data into the pre-trained visual self-attention model (vision transformer), extracts the initial feature information such as the initial position feature and the initial semantic feature of the target object in the image data through the visual self-attention model, and obtains the original feature output of the backbone network in the visual self-attention model. Figure X .in, , D, H, and W are the original features Figure X The vector dimensions, height and width.

[0142] In this embodiment, the format of the initial image data is processed so that the formatted image data meets the requirements of the visual self-attention model. In addition, the image data is feature extracted through the visual self-attention model to preliminarily obtain the initial feature information of the target object, retain the initial semantic features of the target object, avoid the loss of a large amount of information, improve the richness of information, facilitate the subsequent action recognition of the target object based on the initial semantic features and initial position features, and improve the accuracy of image recognition. In addition, the original feature map contains rich semantic features and position features, which is conducive to completing complex recognition targets in one stage.

[0143] In an exemplary embodiment, the image recognition model is a multi-layer perceptron model, such as Figure 8 As shown, the specific processing process of step 106 includes steps 802 to 806. Among them:

[0144] Step 802: input the classification word into the multi-layer perceptron model, and perform recognition processing on the complete target description corresponding to the classification word through the multi-layer perceptron model to obtain an initial output result.

[0145] In the implementation, the computer device inputs the classification word into the multi-layer perceptron (MLP) model, and the complete target description corresponding to the classification word is recognized and processed by the multi-layer perceptron model to obtain the initial input result. Among them, the multi-layer perceptron is a neural network structure with one hidden layer, the feature dimension of the hidden layer is 1 / 4 of the feature dimension D of the input layer, and the ReLU function (Linear rectification function) is used as the activation function.

[0146] Step 804, mapping the initial output result based on a normalization algorithm to obtain an output result.

[0147] Among them, the normalization algorithm is the softmax function (normalized exponential function).

[0148] In implementation, the computer device performs mapping processing on the initial output result based on a normalization algorithm to obtain an output result.

[0149] In an exemplary embodiment, the initial output result is that the ratio of the target object to the target action category and the non-action category is 8:2. The computer device inputs the initial output result to the softmax layer, and the initial output result is mapped by the softmax layer so that the sum of the ratios of the action category and the non-action category is 1, and the output result is that the ratio of the action category and the non-action category is 0.8:0.2.

[0150] Step 806: Determine the action recognition result of the target object according to the output result.

[0151] The output result includes a first probability representing the target object in the image data performing a target action type and a second probability representing the target object in the image data performing a non-target action type.

[0152] In implementation, the computer device determines whether the first probability reaches a preset probability threshold. If the first probability reaches the probability threshold, the computer device determines that the action recognition result of the target object is the target action type. If the first probability does not reach the probability threshold, the computer device determines that the action recognition result of the target object is a non-target action type.

[0153] In this embodiment, the classification word element is recognized and processed by the image recognition model to obtain the action recognition result of the target object and clarify the action type of the target object. In addition, the image recognition method of the present application can be widely applied to classification and recognition tasks in real-life scenarios, such as hat and life jacket recognition, that is, the image recognition method has a wide range of applications.

[0154] In an exemplary embodiment, Fig. 9 FIG. 1 is a schematic diagram of a model of an image recognition method in an exemplary embodiment. Fig. 9 As shown, the computer device obtains the image data of the target object, and inputs the image data into the visual self-attention model, and extracts features of the image data through the visual self-attention model to obtain the original feature map. Then, the computer device inputs the original feature map into the information aggregation module, and extracts the key semantic features and key position features in the original feature map through the task prompt template in the information aggregation module to obtain the classification word. The computer device inputs the classification word into the classification module, and recognizes and processes the classification word through the image recognition model in the classification module to obtain the action recognition result of the target object.

[0155] On the dataset of personnel images collected in real complex scenes, compared with the existing key point posture recognition and action classification recognition models, the present application can achieve more than 5% improvement in recognition accuracy in untrained images, which proves the generalization ability and effectiveness of the present application in behavior recognition methods.

[0156] In an exemplary embodiment, Fig.10 FIG. 1 is a flow chart of determining a classification word unit in an exemplary embodiment. Fig.10 As shown, the computer device extracts features from the image data of the target object through the visual self-attention model to obtain the original features of the target object. Figure X The original feature Figure XThe vector dimension, height and width are D, H and W respectively. The computer device obtains the target vector P and initializes the initial feature matrix M. The dimension of the initial feature matrix M is H×W. The computer device constructs a feature matrix based on the target vector and the initial feature matrix. Then, the computer device performs an inverse Fourier transform (ifft) on the feature matrix and constructs a task prompt template based on the real part (real) of the transformed feature matrix. The computer device performs a dot multiplication on the task prompt template and the original feature map ( Fig.10 asterisk in the middle) and summation ( Fig.10 The sum symbol in the ), get the classification word element .

[0157] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0158] Based on the same inventive concept, the embodiment of the present application also provides an image recognition device for implementing the above-mentioned image recognition method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image recognition device embodiments provided below can refer to the limitations on the image recognition method above, and will not be repeated here.

[0159] In an exemplary embodiment, Fig.11 As shown, an image recognition device 1100 is provided, comprising: a first extraction module 1101, a second extraction module 1102 and a processing module 1103, wherein:

[0160] The first extraction module 1101 is used to obtain image data of the target object and perform feature extraction on the image data based on the visual self-attention model to obtain an original feature map of the target object.

[0161] The second extraction module 1102 is used to extract key semantic features and key position features of the target object in the original feature graph based on the task prompt template to obtain the classification word element of the target object.

[0162] The processing module 1103 is used to perform recognition processing on the classification word element according to the image recognition model to obtain the action recognition result of the target object.

[0163] In an exemplary embodiment, the second extraction module 1102 includes:

[0164] The first processing submodule is used to perform weighted processing on the task prompt template and the original feature map to obtain the key semantic features and key position features in the original feature map; the task prompt template is a description framework corresponding to the format of the original feature map; the key semantic features and key position features represent the description answer corresponding to the description framework.

[0165] The first arrangement submodule is used to arrange key semantic features and key position features to obtain classification word elements of the target object; the classification word elements represent the complete target description composed of the description framework and the description answer.

[0166] In an exemplary embodiment, the image recognition model 1100 further includes:

[0167] The first acquisition module is used to acquire a target vector and initialize an initial feature matrix; the target vector represents a description framework of the image data; the target vector is obtained by training the original vector based on a sample image set corresponding to the image data.

[0168] The first building module is used to build a feature matrix based on the target vector and the initial feature matrix.

[0169] The conversion module is used to convert the feature matrix according to the inverse Fourier algorithm and determine the task prompt template based on the converted feature matrix.

[0170] In an exemplary embodiment, the first building block includes:

[0171] The first initialization submodule is used to generate a long natural sequence corresponding to the initial feature matrix, and randomly sort the long natural sequence to obtain a target long natural sequence.

[0172] The first determination submodule is used to determine the position sequence based on the dimension of the target vector and the target long natural sequence.

[0173] The first adding submodule is used to add the target elements in the target vector to the initial feature matrix one by one based on the position sequence to obtain the feature matrix.

[0174] In an exemplary embodiment, the image recognition model 1100 further includes:

[0175] The second acquisition module is used to acquire a sample image data set, and perform feature extraction on the sample image data set in batches according to the visual self-attention model to obtain a sample original feature atlas.

[0176] The second building module is used to initialize the original vector based on the Kaiming uniform algorithm and initialize the initial task prompt template according to the size of the sample original feature map in the sample original feature map set.

[0177] The third construction module is used to construct an initial task prompt template according to the original vector and the initial feature matrix, and train the initial task prompt template and a preset initial image recognition model based on the sample original feature atlas to obtain a task prompt template and an image recognition model.

[0178] In an exemplary embodiment, the first extraction module 1101 includes:

[0179] The second processing submodule is used to obtain initial image data of the target object, and perform format processing on the initial image data according to a preset image format to obtain image data of the target object.

[0180] The first extraction submodule is used to input the image data into the visual self-attention model, extract the initial feature information of the target object through the visual self-attention model, and obtain the original feature map of the target object; the initial feature information includes initial semantic features and initial position features.

[0181] In an exemplary embodiment, the image recognition model is a multi-layer perceptron model, and the processing module 1103 includes:

[0182] The first recognition submodule is used to input the classification word unit into the multi-layer perceptron model, and perform recognition processing on the complete target description corresponding to the classification word unit through the multi-layer perceptron model to obtain an initial output result.

[0183] The third processing submodule is used to perform mapping processing on the initial output result based on a normalization algorithm to obtain an output result.

[0184] The second determination submodule is used to determine the action recognition result of the target object according to the output result.

[0185] Each module in the above-mentioned image recognition device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0186] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.12As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, an image recognition method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0187] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0188] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0189] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0190] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0191] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0192] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0193] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image recognition method, characterized in that: The method comprises: Acquire image data of a target object, and perform feature extraction on the image data based on a visual self-attention model to obtain an original feature map of the target object; Extracting key semantic features and key position features of the target object in the original feature graph based on the task prompt template to obtain the classification word element of the target object; The classification word is recognized and processed according to the image recognition model to obtain the action recognition result of the target object.

2. The method according to claim 1, characterized in that The step of extracting key semantic features and key position features of the target object in the original feature graph based on the task prompt template to obtain the classification word element of the target object includes: The task prompt template and the original feature map are weighted to obtain key semantic features and key position features in the original feature map; the task prompt template is a description framework corresponding to the format of the original feature map; the key semantic features and the key position features represent the description answer corresponding to the description framework; The key semantic features and key position features are arranged to obtain classification word elements of the target object; the classification word elements represent a complete target description composed of the description framework and the description answer.

3. The method according to claim 1, characterized in that Before extracting the key semantic features and key position features of the target object in the original feature graph based on the task prompt template to obtain the classification word element of the target object, the method further includes: Obtaining a target vector and initializing an initial feature matrix; the target vector is obtained by training the original vector based on a sample image set corresponding to the image data; Constructing a feature matrix based on the target vector and the initial feature matrix; The feature matrix is ​​transformed according to an inverse Fourier algorithm, and a task prompt template is determined based on the transformed feature matrix.

4. The method according to claim 3, characterized in that The constructing a feature matrix based on the target vector and the initial feature matrix includes: Generate a long natural sequence corresponding to the initial feature matrix, and randomly sort the long natural sequence to obtain a target long natural sequence; Determining a position sequence based on the dimension of the target vector and the target long natural sequence; Based on the position sequence, the target elements in the target vector are added one by one to the initial feature matrix to obtain a feature matrix.

5. The method according to claim 1, characterized in that Before extracting the key semantic features and key position features of the target object in the original feature graph based on the task prompt template to obtain the classification word element of the target object, the method further includes: Acquire a sample image data set, and perform feature extraction on the sample image data set in batches according to the visual self-attention model to obtain a sample original feature atlas; Initialize the original vector based on the Kaiming uniform algorithm, and initialize the initial feature matrix according to the size of the sample original feature map in the sample original feature map set; An initial task prompt template is constructed according to the original vector and the initial feature matrix, and the initial task prompt template and a preset initial image recognition model are trained based on the sample original feature atlas to obtain a task prompt template and an image recognition model.

6. The method according to claim 1, characterized in that The acquiring image data of the target object and performing feature extraction on the image data based on a visual self-attention model to obtain an original feature map of the target object includes: Acquire initial image data of a target object, and perform format processing on the initial image data according to a preset image format to obtain image data of the target object; The image data is input into a visual self-attention model, and initial feature information of the target object is extracted through the visual self-attention model to obtain an original feature map of the target object; the initial feature information includes initial semantic features and initial position features.

7. The method according to claim 1, characterized in that The image recognition model is a multi-layer perceptron model, and the classification word is recognized and processed according to the image recognition model to obtain the action recognition result of the target object, including: Inputting the classification word into the multi-layer perceptron model, and performing recognition processing on the complete target description corresponding to the classification word by the multi-layer perceptron model to obtain an initial output result; Performing mapping processing on the initial output result based on a normalization algorithm to obtain an output result; The action recognition result of the target object is determined according to the output result.

8. An image recognition device, characterized in that: The device comprises: A first extraction module is used to obtain image data of a target object, and perform feature extraction on the image data based on a visual self-attention model to obtain an original feature map of the target object; A second extraction module is used to extract key semantic features and key position features of the target object in the original feature map based on the task prompt template to obtain the classification word element of the target object; The processing module is used to perform recognition processing on the classification word element according to the image recognition model to obtain the action recognition result of the target object.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images